Hugging Face Open LLM Leaderboard DROP Benchmark Analysis

TL;DR

Hugging Face has removed the DROP (Discrete Reasoning Over Paragraphs) benchmark from the Open LLM Leaderboard. A technical deep dive revealed that the benchmark's original implementation—reproduced in the EleutherAI Harness—contained critical flaws in normalization and generation stop-tokens that systematically penalized high-performing models and invalidated floating-point answers.

The DROP Benchmark and Initial Anomalies

DROP requires models to extract information from English-text paragraphs and perform discrete reasoning steps, such as counting or sorting, to reach a final answer. Performance is measured using custom f1 and exact match scores.

Upon integrating DROP into the Open LLM Leaderboard, Hugging Face observed a bimodal distribution of scores: while a small number of models showed a correlation between their DROP scores and their overall leaderboard average (ARC, HellaSwag, TruthfulQA, and MMLU), the vast majority of models were stuck with an f1-score below 10, regardless of their general capabilities.

Root Cause 1: Normalization Failures

Investigation revealed that the normalization process failed to correctly identify numerical answers when they were followed by whitespace characters other than a standard space (e.g., line returns).

The failure mechanism works as follows:

  1. Separation: The system splits strings on |, -, or \n.
  2. Punctuation Removal: Punctuation is stripped.
  3. Number Homogenization: Strings that can be cast to float are converted to float and then back to string (e.g., 10 becomes 10.0).

If a model generates a correct answer followed by a newline (e.g., 10\n\nPassage:), the normalization step fails to cast the entire string to a float. Consequently, the answer is treated as a non-numeric string, failing to match the gold answer (which is normalized to 10.0). This results in a failure even when the model predicted the correct value.

Root Cause 2: Stop-Token Interference

Collaborative analysis with Zeno identified two additional systemic issues caused by using a period (.) as the stop-token to end generation:

  • Floating Point Erasure: Because the stop-token is a period, any answer requiring a floating-point number (e.g., 12.25) is interrupted immediately at the decimal point, making it impossible for any model to achieve a correct floating-point result.
  • Penalty for Long-Form Answers: High-quality models that attempt to follow few-shot prompt formats often generate the answer followed by a plausible prompt for the next question. Because the generation only stops at the first period encountered in that subsequent prompt, the model generates excessive tokens, which lowers the f1-score.

Impact and Resolution

Hugging Face tested a partial fix by splitting generated answers on the first newline (\n) instead of the period. This approximation showed a much stronger correlation between DROP scores and overall model performance, confirming that the stop-token was a primary driver of the skewed results.

However, a full correction would require re-running more than 50% of the examples—including all floating-point answers and those affected by normalization—which would require an immense amount of GPU time.

Because the EleutherAI Harness implementation strictly follows the original "official DROP" code, Hugging Face concluded that the benchmark's evaluation logic itself is flawed. As a result, DROP has been removed from the Open LLM Leaderboard until a new, corrected version of the evaluation is developed. The team is currently calling for community and academic collaboration to fix the scoring and normalization logic.

Sources