Hugging Face Community Evals and Every Eval Ever (EEE) Integration

Hugging Face Community Evals and Every Eval Ever (EEE) Integration

Integration of Community Evals and Every Eval Ever (EEE)

Hugging Face has enabled intercompatibility between Hugging Face Community Evals and Every Eval Ever (EEE), allowing evaluation results to be cross-posted and interpreted across both platforms. This integration links open models and leaderboards to a unified, standardized metadata store, addressing the fragmentation of AI evaluation reporting.

Solving the Evaluation Reporting Gap

AI evaluation results are currently scattered across papers, blog posts, and leaderboards, often using inconsistent formats. This lack of standardization leads to discrepancies where the same model on the same benchmark can report different scores depending on the evaluator and the settings used. For example, LLaMA 65B has been reported with MMLU scores of both 63.7 and 48.8.

To resolve this, the Every Eval Ever (EEE) project—launched in February 2026 by the EvalEval Coalition—provides a single JSON schema for evaluation results. This schema records critical metadata, including:

  • The identity of the evaluator
  • The model used
  • The method of access
  • Generation settings
  • The definition of the metric
  • Optional companion JSONL files for per-sample outputs

As of June 2026, the EEE datastore on Hugging Face contains approximately 229,000 evaluation results covering over 22,000 models and 2,200 benchmarks, derived from 31 different reporting formats.

How Community Evals and EEE Work Together

Community Evals and EEE serve complementary roles in the model evaluation ecosystem: Hugging Face provides the visibility layer where users discover models, while EEE provides the interpretability layer containing the full technical record.

Hugging Face Community Evals

Community Evals decentralizes benchmark reporting on the Hub through two mechanisms:

  1. Benchmark Registration: Benchmarks live in dataset repositories with an eval.yaml file, which generates a leaderboard of all reported scores for that benchmark across the Hub.
  2. Model Score Reporting: Scores are stored as YAML files in .eval_results/*.yaml within a model repository. These scores appear on the model card and feed into benchmark leaderboards. Results can be submitted by the model author, the community via pull requests, or independent verifiers.

The Integration Layer

When a result is sent to both platforms, the score appears on the Hugging Face model page and benchmark leaderboard, but it includes a source badge. This badge links directly back to the full EEE JSON record, providing access to the generation configuration, harness version, and reproducibility notes.

Technical Implementation and the Converter Tool

To streamline the process, Hugging Face has introduced a converter that transforms EEE records into the YAML format required by Community Evals.

Mapping and Supported Benchmarks

The converter maps specific EEE fields to Community Evals YAML fields:

  • source_data.hf_repo $\rightarrow$ dataset.id
  • evaluation_name $\rightarrow$ task_id
  • score_details.score $\rightarrow$ value
  • evaluation_timestamp $\rightarrow$ date

The tool currently supports four official benchmarks: MMLU-Pro, GPQA, HLE, and GSM8K.

Workflow and Validation

The converter does not simply reshape data; it performs a series of audits before pushing changes:

  1. Data Retrieval: It downloads the specified EEE datastore collection and verifies object hashes.
  2. Audit: It reads existing .eval_results YAMLs on the model's main branch and open PRs to check for existing scores.
  3. Conflict Detection: Results are flagged as already_present, score_conflict (if a different score exists), or missing_hf_model (if the repo does not resolve on the Hub). Valid entries are marked as ready.
  4. Human Approval: No data is pushed automatically. The tool generates local YAML previews and a review file. PRs are only opened after the user explicitly types OPEN PRS and provides a commit message.

Getting Started

Researchers and evaluators can contribute by submitting full records to the EEE datastore. To automate the submission to Hugging Face, the community eval converter tool available in the every_eval_ever GitHub repository can be used via the following command:

uv run tools/hf-community-evals/community_evals_converter.py MMLU-Pro --datastore evaleval/EEE_datastore@main

Sources