T5Gemma: Encoder-Decoder LLMs via Decoder-Only Adaptation

Google DeepMind has introduced T5Gemma, a new collection of encoder-decoder large language models (LLMs) created by converting pretrained decoder-only models into an encoder-decoder architecture. This approach allows the models to leverage the strengths of the encoder-decoder design—such as higher inference efficiency and richer input representations—while utilizing the knowledge already captured in pretrained decoder-only models.

Model Adaptation: Converting Decoder-Only to Encoder-Decoder

T5Gemma is developed using a technique called model adaptation. Instead of training an encoder-decoder model from scratch, the parameters of the new model are initialized using the weights of a pretrained decoder-only model. These parameters are then further adapted through pre-training using either UL2 or PrefixLM-based objectives.

This adaptation method enables flexible architectural configurations, including "unbalanced" models. For example, a 9B encoder can be paired with a 2B decoder (T5Gemma 9B-2B). This flexibility allows developers to optimize the quality-efficiency trade-off for tasks like summarization, where understanding the input (encoder) is more critical than the complexity of the generated output (decoder).

Quality and Inference Efficiency

T5Gemma models demonstrate a superior quality-inference efficiency Pareto frontier compared to their decoder-only Gemma counterparts. This means they provide better performance for a given level of inference compute.

Key performance observations include:

  • Latency and Accuracy: In GSM8K (math reasoning) benchmarks, T5Gemma 9B-9B achieves higher accuracy than Gemma 2 9B while maintaining similar latency.
  • Efficiency Gains: The T5Gemma 9B-2B model provides a significant accuracy boost over the 2B-2B model while maintaining latency nearly identical to the much smaller Gemma 2 2B model.
  • Representation Quality: T5Gemma models nearly dominate the quality-inference efficiency frontier across several benchmarks, including SuperGLUE, which measures the quality of learned representations.

Reasoning and Instruction-Tuning Capabilities

T5Gemma shows significant improvements in reasoning and general capabilities both in its pretrained and instruction-tuned states.

Pretrained Performance

Pre-training via adaptation leads to gains in complex reasoning tasks. Compared to the original Gemma 2 9B model, T5Gemma 9B-9B scores:

  • Over 9 points higher on GSM8K (math reasoning).
  • 4 points higher on DROP (reading comprehension).

Instruction-Tuned (IT) Performance

The performance gap between T5Gemma and decoder-only models widens further after instruction tuning. For example, T5Gemma 2B-2B IT shows:

  • A MMLU score increase of nearly 12 points over Gemma 2 2B.
  • A GSM8K score increase from 58.0% to 70.7%.

Available T5Gemma Checkpoints

Google DeepMind has released a suite of T5Gemma checkpoints to the community across Hugging Face, Kaggle, and Vertex AI. The released models include:

  • Multiple Sizes: T5-sized models (Small, Base, Large, and XL), Gemma 2-based models (2B and 9B), and a model positioned between T5 Large and T5 XL.
  • Multiple Variants: Both pretrained and instruction-tuned versions are available.
  • Flexible Configurations: An unbalanced 9B-2B checkpoint is provided to explore encoder-decoder size trade-offs.
  • Training Objectives: Models trained with either PrefixLM or UL2 objectives are available to support either state-of-the-art generative performance or representation quality.

Sources