vectara/hallucination-leaderboard

What it solves

This project provides a standardized, automated way to measure how often Large Language Models (LLMs) hallucinate when summarizing text. By quantifying factual consistency, it helps developers and researchers understand the reliability of different models for tasks like Retrieval-Augmented Generation (RAG) and agentic pipelines.

How it works

The leaderboard uses Vectara's proprietary Hallucination Evaluation Model (HHEM) to grade LLM summaries. The process involves feeding a curated dataset of over 7,700 articles to various LLMs and asking them to generate concise, factual summaries using only the provided source text. The system then calculates a hallucination rate based on how well the summary aligns with the source, filtering out models that refuse to answer or provide insufficient responses.

Who it’s for

This project is for AI researchers, developers, and engineers who need to evaluate the factual reliability of LLMs before deploying them in production environments, particularly for RAG or summarization-heavy applications.

Highlights

• Automated, repeatable evaluation process that scales as new models are released.
• Focuses specifically on factual consistency in summarization rather than general knowledge.
• Uses a large, private dataset to prevent models from overfitting to the benchmark.
• Provides clear metrics including Hallucination Rate, Factual Consistency Rate, and Answer Rate.