mmBERT: ModernBERT goes Multilingual

Hugging Face has announced mmBERT, a state-of-the-art massively multilingual encoder model trained on over 3 trillion tokens across more than 1,800 languages. mmBERT is the first model of its kind to improve upon XLM-R, offering significant gains in both natural language understanding (NLU) and retrieval performance while delivering 2-4x faster inference speeds.

Training Data and Progressive Language Inclusion

mmBERT was trained on a curated dataset of 3T+ tokens, utilizing a progressive language inclusion strategy to maximize the impact of low-resource language data without excessive repetition. The data sources include DCLM and Filtered DCLM for high-quality English content, FineWeb2 for broad multilingual coverage (1,800+ languages), and FineWeb2-HQ for 20 high-resource languages. Additionally, the model incorporates specialized corpora from Dolma, MegaWika v2, and ProLong, covering code, academic content, Wikipedia, and textbooks.

To optimize learning, the training data was annealed across three phases:

  • Pre-training: Focused on 60 high-resource languages.
  • Mid-training: Expanded to 110 languages.
  • Decay phase: Included all 1,833 languages.

Training Recipe and Architectural Innovations

mmBERT builds upon the ModernBERT architecture, utilizing Flash Attention 2 and unpadding techniques to achieve high throughput. It adopts the Gemma 2 tokenizer to better handle multilingual text and supports sequence lengths up to 8,192 tokens.

Model Variants

  • mmBERT-base: 110M non-embedding parameters (307M total).
  • mmBERT-small: 42M non-embedding parameters (140M total).

Novel Training Techniques

  • Inverse Mask Ratio Schedule: The masking rate was progressively reduced from 30% to 15% and finally to 5% across the three training phases to move from basic representation learning to nuanced understanding.
  • Annealed Language Learning: Data sampling temperature was adjusted from $\tau=0.7 \to 0.5 \to 0.3$, shifting bias from high-resource languages toward more uniform sampling.
  • Model Merging: TIES merging was used to combine three different variants (English-focused, 110-language, and all-language) trained during the decay phase into the final model.

Performance Benchmarks

Natural Language Understanding (NLU)

mmBERT-base substantially outperforms XLM-R base and mGTE base on the English GLUE benchmark. On the multilingual XTREME benchmark, it shows significant improvements in XNLI classification and TyDiQA question answering, though it performs similarly to previous generations on structured prediction tasks like NER and POS tagging due to tokenizer differences.

Retrieval and Code Performance

In MTEB v2 English benchmarks, mmBERT ties the capabilities of English-only models like ModernBERT. In multilingual MTEB v2 benchmarks, it shows consistent improvements over existing models. Additionally, mmBERT demonstrates strong coding performance on the CoIR code benchmark, outperforming most models except EuroBERT.

Low-Resource Language Acquisition in the Decay Phase

mmBERT demonstrates that low-resource languages can be effectively learned during a short decay phase. By introducing over 1,700 languages in the final 100B token phase, the model achieved rapid learning by leveraging its existing multilingual foundation.

In tests on Tigrinya (TiQuaD) and Faroese (FoQA), mmBERT outperformed significantly larger models, including Google Gemini 2.5 Pro and OpenAI o3 on Faroese question answering tasks. This suggests that the model can quickly adapt existing cross-lingual representations rather than learning from scratch.

Efficiency and Production Readiness

mmBERT provides 2-4x throughput improvements over previous multilingual encoders. These gains are attributed to the ModernBERT architectural heritage, specifically:

  • Flash Attention 2: Optimized attention computation for reduced memory usage.
  • Unpadding: Elimination of unnecessary padding tokens.
  • Extended Context: Efficient handling of up to 8,192 tokens, enabling the processing of longer documents at scale.

Sources