embeddings-benchmark/mteb

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

What it solves

MTEB provides a standardized way to evaluate the performance of text embeddings and retrieval systems. It eliminates the need for researchers and developers to manually set up multiple datasets and evaluation metrics, providing a comprehensive benchmark to compare different models across various tasks.

How it works

The toolbox allows users to select a specific embedding model (either from a pre-defined list or a custom one) and run it against a set of tasks or benchmarks. It supports both a Python API and a Command Line Interface (CLI) to execute evaluations and output the results for analysis.

Who it’s for

AI researchers and developers building or fine-tuning embedding models, as well as those selecting the best embedding model for a specific retrieval or classification task.

Highlights

  • Comprehensive Benchmarking: Supports a wide range of tasks and benchmarks, including multilingual support via MMTEB.
  • Easy Integration: Simple installation via pip or uv and a straightforward API for running evaluations.
  • Citing Support: Built-in functionality to retrieve citations for the benchmarks and tasks used in evaluations.
  • Interactive Leaderboard: Maintains a public leaderboard to track and compare the performance of various embedding models.

Related

  • Dispatch
    MTEB: Massive Text Embedding Benchmark

    Hugging Face introduced MTEB, a massive and multilingual benchmark consisting of 56 datasets across 8 tasks to evaluate the performance of text embedding models.

  • Project
    FlagOpen/FlagEmbedding

    A comprehensive retrieval toolkit for search and RAG that provides state-of-the-art embedding models and rerankers for text and multimodal data.

  • Dispatch
    Hugging Face Introduces RTEB for Retrieval Embedding Evaluation

    Hugging Face has launched the beta version of the Retrieval Embedding Benchmark (RTEB), a new standard designed to measure the true generalization and retrieval accuracy of embedding models using a hybrid of open and private datasets.

  • Dispatch
    Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

    Hugging Face published a guide on training and finetuning multimodal embedding and reranker models with Sentence Transformers, demonstrating how task-specific finetuning improves performance on retrieval tasks such as Visual Document Retrieval.

  • Project
    texttron/tevatron

    A unified toolkit for training and deploying billion-scale neural retrievers across different languages and modalities using LLMs.