Qwen2.5-Math-PRM and ProcessBench Release

Qwen has released two new Process Reward Models (PRMs), Qwen2.5-Math-PRM-7B and Qwen2.5-Math-PRM-72B, and a new evaluation benchmark called ProcessBench. These releases aim to improve the reliability of mathematical reasoning in Large Language Models (LLMs) by identifying and mitigating errors in intermediate reasoning steps, rather than relying solely on the final answer.

ProcessBench: A Step-Level Evaluation Benchmark

ProcessBench is designed to measure a model's ability to identify the exact location of erroneous steps in mathematical reasoning. Unlike traditional response-level evaluations, ProcessBench focuses on thestep-by-step verification of solutions.

  • Dataset Composition: The benchmark consists of 3,400 test cases, primarily focusing on competition- and Olympiad-level mathematics.
  • Annotation: Each test case includes a step-by-step solution where human experts have annotated the location of the error.
  • Task: Models are evaluated on their ability to identify the earliest step containing an error or correctly conclude that the entire solution is correct.
  • Applicability: The benchmark can be used to evaluate both PRMs and general "critic" models (where general LLMs are prompted to critique solutions step-by-step).

Qwen2.5-Math-PRM Model Capabilities

Qwen has released two PRM versions fine-tuned on Qwen2.5-Math-7B-Instruct and Qwen2.5-Math-72B-Instruct. These models are designed to provide process supervision by scoring individual reasoning steps.

Best-of-N (BoN) Performance

In Best-of-N evaluation, the PRM selects the highest-scoring response from $N$ candidates. For Qwen2.5-Math-PRM-7B, $N=8$ responses were sampled from Qwen2.5-Math-7B-Instruct across seven benchmarks (GSM8K, MATH, Minerva Math, GaoKao 2023 En, OlympiadBench, College Math, and MMLU STEM).

  • Performance Gain: Qwen2.5-Math-PRM-7B outperformed the majority voting baseline (maj@8) across all seven tasks, with an average improvement of 1.4%.
  • Comparison: Qwen2.5-Math-PRM-72B showed overall better performance than Qwen2.5-Math-RM-72B, with the most significant gains appearing in the MMLU STEM and Minerva Math tasks.

Error Identification on ProcessBench

When tested on ProcessBench, the PRMs demonstrated a high capacity for identifying erroneous reasoning steps:

  • Competitive Standing: Qwen2.5-Math-PRM-7B outperformed all other open-source models and proprietary models like GPT-4o-0806.
  • Relative Performance: While Qwen2.5-Math-PRM-7B is superior to other open-source PRMs, a performance gap remains when compared to o1-mini.
  • ORM Insights: The Outcome Reward Model (ORM) Qwen2.5-Math-RM-72B also showed significant capability in identifying step errors, surpassing some open-source PRMs.

Implications for Process Supervision

Qwen identifies that relying solely on response-level Best-of-N evaluation can lead to potential bias. By introducing ProcessBench, the lab aims to fill the gap in step-level evaluation and highlight the limitations of current data construction approaches for PRMs. The release of these models and the benchmark is intended to foster future research into scalable oversight and the trustworthiness of LLM reasoning processes.

Sources