Hugging Face Hallucinations Leaderboard launch and initial findings

TL;DR

The Hugging Face Hallucinations Leaderboard was launched to benchmark large language models on factuality and faithfulness hallucinations across a suite of open-source datasets, providing a transparent, continuously updated ranking that helps researchers identify the most reliable models.

What the leaderboard measures

The leaderboard evaluates models on hallucination‑related benchmarks using the EleutherAI LM Evaluation Harness. All metrics are normalized to a ([0,1]) scale, where higher scores indicate fewer hallucinations. The evaluation runs on NVIDIA A100 GPUs at the Edinburgh International Data Facility and the University of Edinburgh’s internal clusters.

Benchmark suite

The leaderboard aggregates eight task families, each targeting a specific hallucination failure mode:

  • Closed‑book Open‑Domain QA – NQ Open, TriviaQA, TruthfulQA (8‑shot and 64‑shot settings). Accuracy is measured by Exact Match.
  • Summarisation – XSum and CNN/DM (2‑shot). Metrics: ROUGE‑1/2/L, factKB, and BERTScore‑Precision.
  • Reading Comprehension – RACE (2‑shot) and SQuADv2 (4‑shot). Evaluates answer selection and unanswerable‑question detection.
  • Instruction Following – MemoTrap (zero‑shot) and IFEval (zero‑shot). Tests whether models obey constraints without memorising undesired text.
  • Fact‑Checking – FEVER (16‑shot). Models predict SUPPORTS, REFUTES, or NOT ENOUGH INFO.
  • Hallucination Detection – FaithDial, True‑False, HaluEval (QA/Dialogue/Summarisation) in 8‑shot settings. Models must flag hallucinated outputs.
  • Self‑Consistency – SelfCheckGPT (238 instances). Generates six passages per instance (one deterministic, five stochastic) and uses an NLI model to flag inconsistent sentences.

Preliminary findings

Model clusters

Hierarchical clustering of the results matrix reveals three natural groups:

  1. Mistral 7B‑based models (e.g., Mistral 7B‑OpenOrca, Zephyr 7B beta, Starling‑LM 7B alpha).
  2. LLaMA 2‑based models (e.g., LLaMA 2 7B, LLaMA 2 13B, Wizard Vicuna 13B).
  3. Smaller models (e.g., BLOOM 560M, GPT‑Neo 125M, Orca Mini 3B).

Closed‑book QA

  • Mistral 7B variants outperform other models on TriviaQA (8‑shot) and TruthfulQA.
  • Falcon 7B leads on NQ Open (8‑shot). Increasing to 64‑shot improves LLaMA 2 13B to the top position (EM = 0.34).

Instruction following

  • Surprisingly, BLOOM 560M ranks among the best on MemoTrap, suggesting smaller models avoid memorising famous quotes.
  • LLaMA 2 13B Chat and Mistral 7B Instruct achieve the highest scores on the more complex IFEval tasks.

Summarisation

  • GPT‑JT 6B attains the highest ROUGE scores on CNN/DM, largely by extracting leading sentences.
  • LLaMA 2 13B underperforms, often truncating output after a single token, likely due to context‑length limits.

Reading comprehension

  • Mistral 7B and LLaMA 2 models lead on RACE.
  • For SQuADv2, mGPT excels at detecting unanswerable questions, while Starling‑LM 7B alpha is best at answering answerable ones.

Hallucination detection

  • SelfCheckGPT scores are highest for Mistral 7B OpenOrca, which frequently outputs empty answers that are trivially self‑consistent.
  • HaluEval results favor Mistral and LLaMA 2 families across QA, Dialogue, and Summarisation.

Why this matters

By providing a standardized, open‑source evaluation platform, the Hallucinations Leaderboard makes it possible to compare models on concrete hallucination metrics rather than proxy scores. This transparency helps developers select models that are less likely to spread misinformation or deviate from user intent, and it incentivises the community to improve factuality and faithfulness in future LLM releases.

Getting involved

The leaderboard is open‑ended: researchers can submit new models, propose additional tasks, or contribute computational resources via the discussion page linked on the leaderboard UI. Sample generations and detailed analytics will be expanded in the coming weeks.

Citation

If you use the leaderboard or its data, cite the accompanying arXiv paper:

@article{hallucinations-leaderboard,
  author = {Giwon Hong and Aryo Pradipta Gema and Rohit Saxena and Xiaotang Du and Ping Nie and Yu Zhao and Laura Perez‑Beltrachini and Max Ryabinin and Xuanli He and Clémentine Fourrier and Pasquale Minervini},
  title = {The Hallucinations Leaderboard - An Open Effort to Measure Hallucinations in Large Language Models},
  journal = {CoRR},
  volume = {abs/2404.05904},
  year = {2024},
  url = {https://doi.org/10.48550/arXiv.2404.05904},
  doi = {10.48550/ARXIV.2404.05904}
}

Sources