MTEB: Massive Text Embedding Benchmark

Hugging Face has released the Massive Text Embedding Benchmark (MTEB), a comprehensive framework designed to measure the performance of text embedding models across a wide variety of tasks. This benchmark provides a standardized way to evaluate how well vector representations of text encode semantic information, which is critical for downstream NLP applications such as search engines, clustering, and text classification.

MTEB Framework Capabilities

MTEB is designed to be a holistic, multilingual, and extensible evaluation tool for text embeddings. Its core characteristics include:

  • Scale: The benchmark encompasses 56 datasets spanning 8 different tasks. At launch, the MTEB leaderboard already summarized over 2,000 results.
  • Multilingual Support: MTEB includes data for up to 112 different languages, with specific benchmarking for multilingual models in Classification, Semantic Textual Similarity (STS), and Bitext Mining.
  • Extensibility: The framework is open-source, allowing the community to contribute new tasks, datasets, metrics, and leaderboard entries via its GitHub repository.

Model Performance Trade-offs

Initial benchmarking reveals a clear trade-off between computational speed, embedding size, and overall performance. Hugging Face categorizes models into three primary profiles to assist users in selection:

Maximum Speed

Models such as Glove provide high inference speed but lack context awareness, which results in lower average MTEB scores.

Balance of Speed and Performance

Models like all-mpnet-base-v2 and all-MiniLM-L6-v2 offer a middle ground, providing significantly stronger performance than basic word embeddings while maintaining reasonable speed.

Maximum Performance

Large-scale models with multi-billion parameters, including ST5-XXL, GTR-XXL, and SGPT-5.8B-msmarco, dominate the MTEB leaderboard. These models typically produce larger embeddings; for example, SGPT-5.8B-msmarco generates 4096-dimensional embeddings, which increases storage requirements.

Benchmarking and Submission Process

Developers can evaluate their own models using the mteb Python library. The process involves installing the library via pip, running the evaluation on a specific task (e.g., using Banking77Classification), and generating a results JSON file.

To share results on the public leaderboard, users can utilize an automatic script (mteb_meta.py) to generate metadata. This metadata is then added to the README.md of the model's Hugging Face Hub repository, which triggers the model's appearance on the MTEB leaderboard upon refresh.

Sources