Hugging Face Open LLM Leaderboard MMLU Evaluation Analysis

Hugging Face has identified that discrepancies in MMLU (Massive Multitask Language Understanding) scores on the Open LLM Leaderboard—specifically for the LLaMA model—stem from differences in how the benchmark is implemented across various evaluation libraries. This highlights that LLM evaluation results are highly sensitive to minute details such as prompt formatting and the method used to extract predictions.

The Impact of Implementation on MMLU Scores

Different software implementations of the same MMLU dataset can produce widely different absolute scores and alter the ranking of models. Hugging Face compared three specific implementations:

  1. The Original Implementation: The code proposed by the UC Berkeley team that developed MMLU.
  2. The HELM Implementation: The code provided in Stanford's Holistic Evaluation of Language Models (HELM).
  3. The EleutherAI LM Evaluation Harness: The library used as the backend for the Open LLM Leaderboard.

Comparative results showed that a model like LLaMA-65B scored 0.636 in the Original implementation but only 0.488 in the Harness implementation. Such a gap (approximately 30%) can mislead researchers into believing a model has been trained poorly when the difference is actually due to the evaluation methodology.

Technical Differences in Evaluation Methods

Discrepancies arise from two primary areas: the construction of the prompt and the method of calculating the score.

Prompt Variations

Even small changes in the input string affect model output. Differences observed between the three implementations include:

  • Instructions: Variations in the introductory sentence and the inclusion of the topic line.
  • Prefixes: Whether the question is preceded by a "Question:" label.
  • Choice Formatting: Whether the multiple-choice options are prepended with a "Choices" keyword.

Prediction Extraction Methods

There are three distinct ways these libraries extract answers from the model:

  • Probability Comparison of Letters (Original): The model's predicted probabilities for the single tokens "A", "B", "C", and "D" are compared. The highest probability among these four wins, even if the model would have preferred to generate a different word entirely.
  • Text Generation Comparison (HELM): The model generates a text response, which is then compared to the expected letter answer. If the model generates a word other than the expected letter (e.g., "Zygote"), it is marked incorrect.
  • Full Sequence Probability (Harness - Jan 2023): The library compares the probabilities of the full answer sequence (e.g., "C. The second pharyngeal arch"). This involves summing the logarithms of the probabilities of each token in the sequence.

Summary of Evaluation Approaches

Implementation Method of Comparison
Original Compares probabilities of the letter answers (A, B, C, D)
HELM Expects the model to generate the letter answer as text
AI Harness (Jan 2023) Compares probabilities of the full answer sequences (Letter + Text)

Conclusion and Future Updates

Evaluation results are not absolute; they are tied to specific implementations, tokenization, and prompts. To ensure fair comparison, Hugging Face emphasizes the necessity of open, standardized, and reproducible benchmarks like the EleutherAI Eval Harness and Stanford HELM.

To resolve the observed discrepancies, the EleutherAI Harness has been updated to align its MMLU evaluation more closely with the original implementation. Hugging Face is currently updating the Open LLM Leaderboard to reflect these changes using the updated EleutherAI Eval Harness v2.

Sources