ModernBERT Release Notes

ModernBERT is a new family of encoder-only models designed as a slot-in replacement for BERT-like models. It provides a Pareto improvement over previous encoders in both speed and accuracy, featuring a sequence length of 8,192 tokens and incorporating a significant amount of code in its training data.

Model Variants and Availability

ModernBERT is available in two sizes:

  • Base: 149 million parameters
  • Large: 395 million parameters

The models are integrated into transformers v4.48.0. For maximum efficiency, the developers recommend using ModernBERT with Flash Attention 2. Unlike earlier BERT models, ModernBERT does not use token type IDs.

Performance and Efficiency

ModernBERT outperforms previous encoder models across retrieval, natural language understanding, and code retrieval tasks.

Key Benchmarks

  • Accuracy: ModernBERT is the first base-size model to beat DeBERTaV3 on GLUE while using less than one-fifth of DeBERTa's memory.
  • Speed: It is up to 4x faster than DeBERTa for mixed-length inputs and nearly 3x faster than high-quality models like NomicBERT and GTE-en-MLM for long-context inference.
  • Context Length: With a context window of 8,192 tokens, ModernBERT's capacity is over 16x larger than most existing encoders, significantly improving performance in RAG pipelines and long-context retrieval.
  • Code Retrieval: ModernBERT is the first encoder model trained on large-scale code data, scoring over 80 on the StackOverflow-QA (SQA) dataset.

Hardware Efficiency

Tested on NVIDIA RTX 4090 GPUs, ModernBERT demonstrates superior efficiency for variable-length inputs. It avoids the heavy xformers dependency, requiring only Flash Attention. Its design allows for larger batch sizes, making it viable for deployment on smaller, cheaper GPUs or directly in browsers and phones.

Technical Architecture

ModernBERT updates the classic Transformer architecture by incorporating advances from recent Large Language Model (LLM) research, specifically drawing inspiration from Transformer++.

Architectural Improvements

  • Rotary Positional Embeddings (RoPE): Replaces old positional encoding to improve relative word understanding and enable scaling to longer sequences.
  • GeGLU Layers: Replaces standard MLP layers to improve upon the original GeLU activation function.
  • Symmetry and Stability: The architecture removes unnecessary bias terms and adds an extra normalization layer after embeddings to stabilize training.

Efficiency Mechanisms

  • Alternating Attention: To reduce computational complexity, ModernBERT uses global attention (attending to the full input) only every three layers; other layers use local attention with a sliding window of 128 tokens.
  • Unpadding and Sequence Packing: ModernBERT removes padding tokens and concatenates sequences into mini-batches to eliminate wasted compute. This implementation leverages Flash Attention's RoPE support for a 10-20% speedup over previous unpadding methods.
  • Hardware-Aware Design: The model dimensions were optimized via grid search to align with common inference GPUs (RTX 3090/4090, A10, T4, L4) while maintaining BERT-compatible embedding sizes (768 for base, 1024 for large).

Training Process

ModernBERT was trained on 2 trillion tokens from diverse English sources, including web documents, scientific articles, and code, reducing the reliance on repeated data common in older encoders.

Training Phases

  1. Initial Phase: Trained on 1.7 trillion tokens with a sequence length of 1,024.
  2. Long-Context Adaptation: Trained on 250 billion tokens with a sequence length of 8,192.
  3. Annealing: A final phase of 50 billion tokens using a sampled mix based on ProLong.

Training Optimizations

  • Masking Rate: Increased from 15% to 30%.
  • Objective: Removed the Next-Sentence Prediction (NSP) objective to reduce overhead.
  • Batch-Size Warmup: Gradually increased batch size to accelerate initial learning.
  • Weight Tiling: ModernBERT-Large was initialized by tiling the weights of ModernBERT-Base rather than using random initialization, which improved training speed and performance.

Sources