Hugging Face Open LLM Leaderboard Update: Integrating Math-Verify for Improved Math Evaluation

Hugging Face has integrated the Math-Verify parser into the Open LLM Leaderboard to resolve systemic parsing errors that were underestimating model performance on mathematical tasks. This update involves a comprehensive re-evaluation of 3,751 models, leading to a significant shift in the MATH-Hard leaderboard rankings.

Why Previous Math Evaluation Was Flawed

The Open LLM Leaderboard's MATH-Hard task evaluates models on 1,324 high-difficulty (Level 5) problems from the Hendrycks MATH dataset across seven topics, including algebra, precalculus, and number theory. Previously, the evaluation pipeline relied on a strict answer format and SymPy parsing, which caused correct answers to be marked as incorrect due to three primary failure points:

1. Format Strictness

Models that failed to end their responses with the exact string "Final answer is [ANSWER]. I hope it is correct" were marked wrong, even if the mathematical result was correct. Math-Verify removes this dependency on rigid formatting.

2. Extraction and Parsing Failures

The previous system struggled to extract answers from common LaTeX borders or specific mathematical structures. Examples of failures included:

  • LaTeX Borders: Answers wrapped in \boxed{} notations (common in DeepSeek models) were not extracted.
  • Complex Structures: Matrices, intervals (e.g., (-oo, -14)), and parametric equations were often parsed incorrectly or failed entirely.
  • Invalid Symbols: Simple percentage signs (e.g., "100%") caused extraction failures.

3. Equivalence Comparison Issues

Even when extracted, the system lacked the ability to determine if two mathematically equivalent expressions were the same. It failed to recognize:

  • Numerical Equivalence: For example, 1/3 was not recognized as equal to 0.333333 due to a lack of rounding support.
  • Symbolic Equivalence: Expressions like sqrt(1/2)*7 and sqrt(0.5)*7 were marked as different.
  • Advanced Types: The system lacked support for matrix equivalence and set comparison (e.g., {1} \cup {1,4} vs {1,4}).

Impact on Model Performance and Rankings

The transition to Math-Verify resulted in an average score increase of 4.66 points across all models, with models solving an average of 61 more problems correctly.

Subject-Specific Gains

The most significant improvements occurred in algebra-related subsets:

  • Algebra: 8.27 point increase.
  • Prealgebra: 6.93 point increase.

These gains are attributed to Math-Verify's improved handling of sets and matrices, which are frequent in these subjects.

Model Family Shifts

Certain model families saw dramatic score increases because their specific output styles were previously penalized:

  • DeepSeek: Scores nearly tripled because the parser now correctly handles \boxed{} notations.
  • Qwen: Scores more than doubled, correcting a previous severe underestimation of performance.

Leaderboard Reshuffling

The MATH-Hard Top 20 rankings have been completely overhauled. Nvidia's AceMath models now dominate the leaderboard, followed closely by Qwen derivatives. While the top four overall positions on the general leaderboard remain unchanged, many other models have jumped 200 or more places in the overall rankings due to their improved math scores.

Conclusion

The adoption of Math-Verify ensures that the Open LLM Leaderboard reflects a model's actual mathematical capability rather than its ability to follow a specific string format. Hugging Face encourages researchers and developers to use Math-Verify for their own internal evaluations to achieve more reliable results.

Sources