Hugging Face Community Evals

Hugging Face has launched Community Evals, a decentralized framework that allows benchmark datasets to host leaderboards and models to store their own evaluation scores. This system aims to eliminate the lack of a single source of truth for model performance by enabling the community to submit and verify results via pull requests.

Decentralized Evaluation Reporting

Hugging Face is shifting toward a decentralized model for evaluation reporting on the Hub. Instead of relying on centralized, black-box leaderboards, the system allows the community to openly report scores for benchmarks. The rollout begins with a shortlist of four benchmarks, with plans to expand to more relevant benchmarks over time.

For Benchmark Datasets

Dataset repositories can now register as benchmarks (with MMLU-Pro, GPQA, and HLE already live). These repositories automatically aggregate reported results from across the Hub to display leaderboards within the dataset card. To ensure reproducibility, benchmarks define their evaluation specifications via an eval.yaml file based on the Inspect AI format.

For Model Repositories

Evaluation scores are stored as YAML files within the .eval_results/ directory of a model's repository. These scores appear on the model card and are fed into the benchmark datasets. The system aggregates both the results provided by the model author and results submitted via open pull requests.

For the Community

Any user can submit evaluation results for any model through a pull request. These results are displayed as "community" scores without requiring the model author to merge the PR. Users can link to supporting evidence, such as research papers, model cards, third-party evaluation platforms, or inspect evaluation logs. Because the Hub is Git-based, all evaluation additions and changes are tracked via a transparent history.

Addressing the Evaluation Gap

Community Evals addresses two primary gaps in the current AI evaluation landscape:

  1. The Performance Gap: There is a documented disconnect between high benchmark scores (e.g., MMLU exceeding 91%, GSM8K over 94%, and HumanEval being "conquered") and real-world performance in tasks like web browsing, production coding, and multi-step reasoning.
  2. The Reporting Gap: Discrepancies often exist between scores reported in model cards, research papers, and various evaluation platforms, leading to a lack of a unified source of truth.

Implications for the AI Ecosystem

By exposing existing scores via Hub APIs, the community can more easily aggregate and track performance across the field to build curated dashboards and leaderboards. While Hugging Face acknowledges that this system will not solve benchmark saturation or prevent training on test sets, it increases visibility into what is being evaluated, how it is performed, and who is conducting the evaluation.

The ultimate goal is to transform the Hub into an active environment for sharing reproducible benchmarks, specifically focusing on new tasks and domains that challenge state-of-the-art models.

Sources