Implement T5Gemma

[jncraton]

Support T5Gemma architecture. Here's the basic idea from the transformers T5Gemma documentation:

T5Gemma (aka encoder-decoder Gemma) was proposed in a research paper by Google. It is a family of encoder-decoder large language models, developed by adapting pretrained decoder-only models into encoder-decoder. T5Gemma includes pretrained and instruction-tuned variants. The architecture is based on transformer encoder-decoder design following T5, with improvements from Gemma 2: GQA, RoPE, GeGLU activation, RMSNorm, and interleaved local/global attention.

For reference, here is the T5Gemma PR that merged model support for these architectures into transformers.

[jncraton]

@copilot Could you confirm that this implementation matches the paper and transformers PR linked above? The converter is working properly for me, but I'm gettting the same token repeatedly in the output. Make sure every layer norm is in the correct place and make sure that rope is used in both the encoder and decoder as specified in the paper.

[Copilot]

@jncraton I've opened a new pull request, #4, to work on those changes. Once the pull request is ready, I'll request review from you.

[jncraton]

@copilot Can you check this over to see if I've perhaps done something wrong? I'm not sure I've configured the TransformerSpec correctly. It looks like BART and T5 use from_config to build the TransformerSpec. Should I be doing that?

Overall, this should work the same as the implementation in transformers, but I'm seeing the same nonsense token repeatedly in the output. I don't need any of the training oriented features, such as dropout, but otherwise everything else should look like the transformers implementation. It might also be helpful to check a sample config and the paper.

[Copilot]

@jncraton I've opened a new pull request, #6, to work on those changes. Once the pull request is ready, I'll request review from you.