Hugging Face Introduces RTEB for Retrieval Embedding Evaluation

Hugging Face has introduced the beta version of the Retrieval Embedding Benchmark (RTEB), a new standard designed to reliably evaluate the retrieval accuracy of embedding models for real-world applications. RTEB addresses the "generalization gap" found in existing benchmarks by utilizing a hybrid strategy of open and private datasets to prevent models from "teaching to the test."

Addressing the Generalization Gap in Retrieval

Existing retrieval benchmarks often suffer from a discrepancy between reported zero-shot performance on public datasets and actual performance on unseen, real-world data. This is primarily driven by two factors:

  • Benchmark Overfitting: When training data overlaps with evaluation datasets, models may memorize test data rather than developing robust generalizable capabilities, leading to inflated scores.
  • Application Misalignment: Many current benchmarks rely on academic or QA-derived datasets that fail to capture the distributional biases and complexities of enterprise use cases, or they are too narrow (e.g., focusing only on code retrieval).

The RTEB Hybrid Evaluation Strategy

To ensure a fair and transparent measure of a model's ability to generalize, RTEB employs a dual-dataset approach:

  • Open Datasets: The corpus, queries, and relevance labels are fully public, allowing for transparency and reproducibility.
  • Private Datasets: These are kept private and evaluated exclusively by MTEB maintainers. This prevents data leakage and provides an unbiased measure of performance on unseen data. To maintain transparency, Hugging Face provides descriptive statistics, dataset descriptions, and sample triplets for these private sets.

A significant performance drop between open and private datasets serves as a clear signal of overfitting.

Technical Design and Domain Focus

RTEB is optimized for enterprise-grade retrieval tasks and is characterized by the following technical specifications:

Domain and Language Coverage

  • Multilingual Support: The benchmark covers 20 languages, including common languages like English and Japanese, and lower-resource languages such as Bengali and Finnish.
  • Enterprise Domains: Specific focus is placed on high-stakes domains, including law, healthcare, finance, and code.

Evaluation Metrics and Scale

  • Primary Metric: The default leaderboard metric is NDCG@10, the industry standard for measuring the quality of ranked search results.
  • Dataset Efficiency: Each dataset contains at least 1,000 documents and 50 queries to ensure statistical significance without making evaluation prohibitively expensive.

Current Limitations and Future Roadmap

Hugging Face has identified several areas for the continued evolution of RTEB:

  • QA Repurposing: Approximately 50% of current datasets are repurposed from QA sets, which may introduce high lexical overlap and favor keyword matching over semantic understanding.
  • Modality Expansion: The current version is text-only; future releases aim to incorporate text-image and other multimodal retrieval tasks.
  • Language Expansion: Efforts are underway to add more major languages, specifically Chinese and Arabic, as well as additional low-resource languages.
  • Scope: The benchmark currently focuses on realistic retrieval-first use cases rather than highly challenging synthetic datasets.

RTEB Dataset Composition

Open Datasets

Open datasets include a variety of specialized sources such as AILACasedocs and AILAStatutes (Indian legal documents), FinanceBench and FinQA (financial reports), HumanEval and MBPP (code), and ChatDoctor_HealthCareMagic (medical Q&A).

Private Datasets

Private datasets include specialized legal records from Germany, Japan, and France, as well as expert-annotated healthcare data from English and German sources and specialized code functions extracted from GitHub.

Sources