Ettin Suite: SoTA Paired Encoders and Decoders

Hugging Face has announced the Ettin Suite, the first collection of state-of-the-art (SoTA) paired encoder-only and decoder-only models ranging from 17M to 1B parameters. By using identical training data (2T tokens), model shapes, and training recipes, Ettin allows for a controlled, "apples-to-apples" comparison between bidirectional encoder architectures and causal decoder architectures.

Architectural Comparison: Encoders vs. Decoders

Ettin demonstrates that fundamental architectural differences persist regardless of the training recipe. The primary distinction remains the attention pattern: encoder models use bidirectional attention (all tokens are visible), while decoder models use causal attention (tokens only see previous tokens).

Key findings from the controlled comparison include:

  • Classification and Retrieval: Encoder models dominate these tasks. For example, a 150M encoder outperforms a 400M decoder on MNLI classification (89.2 vs 88.2).
  • Text Generation: Decoder models maintain a consistent advantage in generative tasks, a gap that widens as model size increases.
  • Scale vs. Architecture: Architecture often outweighs size for specific tasks; a 400M encoder can beat a 1B decoder on classification, while a 400M decoder beats a 1B encoder on generation.

Training Recipe and Data

The Ettin models were developed using a modernized recipe based on ModernBERT, applying modern decoder-only techniques to both architectures. All training data used for Ettin is public and reproducible.

Three-Phase Training Process

  1. Pre-training (1.7T tokens): Initial training on shorter contexts (1024 tokens) using a diverse mixture of high-quality data.
  2. Context Extension (250B tokens): Context length is increased to 8K tokens using filtered, higher-quality data to improve long-document understanding.
  3. Decay (100B tokens): Final training phase using premium sources, including textbooks and scientific papers, with a gradually reducing learning rate.

Model Sizes

Ettin provides six paired sizes to test the effects of scale:

  • 17M, 32M, 68M, 150M, 400M, and 1B parameters.

Performance Benchmarks

Encoder Results

Ettin encoder models outperform ModernBERT across all tested tasks and model sizes while utilizing entirely open training data. This provides a new range of high-performance encoders for both on-device (small sizes) and high-power (1B size) applications.

Decoder Results

Ettin decoder models match or outperform established baselines such as Llama 3.2 and SmolLM2. Performance gains are particularly notable in knowledge-intensive tasks like SciQ, attributed to the high-quality training data mixture.

Research Insights and Model Behavior

Cross-Objective Training

Ettin was used to test whether a model could be converted from one architecture's objective to another (e.g., continuing pre-training a decoder with Masked Language Modeling). The results indicate that architecture choice is fundamental:

  • Encoder-from-decoder: Generally trails native encoders in classification and retrieval.
  • Decoder-from-encoder: Performs significantly worse than native decoders, especially at larger scales.

Behavioral Analysis

Using the WinoGender benchmark, researchers found that training objectives influence model bias. Encoder models prefer gender-neutral pronouns more frequently (60%+) compared to decoders (30%+), though both architectures exhibit a male bias.

Implementation and Usage

Ettin models are available via the Hugging Face Hub. Encoders are recommended for classification and retrieval tasks, while decoders are intended for text generation.

Encoder Example

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-150m")
model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-150m")

Decoder Example

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-150m")
model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-150m")

Sources