Hugging Face Leaderboard Templates: Implementing the Vectara HHEM Leaderboard

TL;DR

Hugging Face has released open-source leaderboard templates that allow developers to create lightweight, dynamic evaluation boards for LLMs. Vectara has utilized these templates to launch the Hughes Hallucination Evaluation Model (HHEM) leaderboard, which provides a standardized way to measure and compare the hallucination rates of both open-source and commercial LLMs.

The Hughes Hallucination Evaluation Model (HHEM)

The HHEM is an open-source classification model designed to assess the frequency of hallucinations—text that is nonsensical or unfaithful to source content—specifically within document summaries generated by LLMs. The model evaluates a wide range of AI systems, including commercial models like GPT-4, Google Gemini, and Anthropic Claude, as well as open-source models such as Llama 2 and Mistral 7B.

By open-sourcing HHEM, Vectara aims to democratize the evaluation of LLM hallucinations and increase community awareness regarding the varying propensities of different models to hallucinate. The HHEM is named in honor of Simon Hughes, a pioneer in hallucination research.

Implementing the HHEM Leaderboard via HF Templates

Vectara transitioned from a static GitHub repository to a dynamic Hugging Face Space by leveraging the Hugging Face leaderboard templates. This framework allows for the management of model submission requests and the automatic updating of results.

Basic Setup

For a standard leaderboard, the implementation requires:

  1. Cloning the space repository to a specific organization.
  2. Creating two associated datasets: a "requests" dataset to track user submissions for new LLM evaluations and a "results" dataset to store the final evaluation metrics.
  3. Populating the results dataset with initial data and updating the "About" and "Citations" sections.

Advanced Customizations for HHEM

Because the HHEM evaluation process is more complex than a simple result push, Vectara customized the source code across several key backend files:

  • leaderboard/src/backend/model_operations.py: Implements the SummaryGenerator (which generates summaries from a private evaluation dataset and calculates Answer Rate and Average Summary Length) and the EvaluationModel (which uses the HHEM model to calculate Factual Consistency Rate and Hallucination Rate).
  • leaderboard/src/backend/evaluate_model.py: Defines the Evaluator class, which coordinates the SummaryGenerator and EvaluationModel to produce results in JSON format.
  • leaderboard/src/backend/run_eval_suite.py: Contains the run_evaluation function, which uploads the final computed results to the results dataset for display on the leaderboard.
  • leaderboard/main_backend.py: Manages the queue of pending evaluation requests and executes the auto-evaluations, while providing an option for users to replicate existing results.

Implications for the LLM Community

The availability of these open-source templates reduces the barrier to entry for creating specialized LLM leaderboards. By providing a framework that handles the submission and update pipeline, Hugging Face enables the community to move beyond static benchmarks toward dynamic, community-driven evaluation systems that can track the performance of new models as they are released.

Sources