Open R1 Update #2: OpenR1-Math-220k Dataset and Reasoning Insights

Hugging Face has released OpenR1-Math-220k, a large-scale mathematical reasoning dataset designed to replicate the distillation capabilities of DeepSeek R1. This release is part of the broader Open R1 project, which aims to open-source the training pipelines and synthetic data generation methods used to create high-performance reasoning models.

OpenR1-Math-220k Dataset

The OpenR1-Math-220k dataset provides a foundation for transferring advanced reasoning capabilities to smaller models through distillation, filling a gap left by the non-public reasoning traces used in the original DeepSeek R1 distillation.

Data Generation Pipeline

Hugging Face generated the dataset locally using 512 H100 GPUs, leveraging vLLM and SGLang to produce reasoning traces. The transition to SGLang resulted in a nearly 2x speedup, increasing throughput from 15 to 25 solutions per hour per H100, enabling the generation of 180k traces per day.

Key technical specifications of the generation process include:

  • Source Material: Based on NuminaMath 1.5, an improved version of NuminaMath-CoT.
  • Scale: 800k R1 reasoning traces were generated for 400k problems (two to four solutions per problem).
  • Prompting: DeepSeek R1 was prompted to "reason step by step, and put your final answer within \boxed{}".
  • Token Limit: A 16k token limit was set per generation, as analysis showed 25% of problems required more than 8k tokens.

Automated Filtering and Quality Control

To ensure high-quality reasoning traces, Hugging Face employed a hybrid verification system:

  1. Rule-based Verification: Using Math Verify, the team extracted final answers and compared them against ground truth data. 55% of problems had at least one correct answer.
  2. LLM-based Evaluation: For cases where ground truth answers were malformed or empty, Llama-3.3-70B-Instruct acted as a judge to determine if the model's solution was equivalent to the reference, recovering 28,000 previously rejected problems.

The final dataset consists of 220k problems with verified reasoning traces. It is available in two splits: a default split (94k problems) and an extended split (131k problems). The default split yielded the best performance after Supervised Fine-Tuning (SFT).

Performance Benchmarks

When fine-tuning Qwen2.5-Math-Instruct on the default split for 3 epochs (learning rate 5e-5, RoPE frequency increased to 300k for 32k context length), the resulting OpenR1-Qwen-7B model showed competitive performance against DeepSeek-Distill-Qwen-7B:

Model MATH-500 AIME24 AIME25
DeepSeek-Distill-Qwen-7B 91.6 43.3 40
OpenR1-Qwen-7B 90.6 36.7 40
OpenThinker-7B 89.6 30.0 33.3

Math-Verify Improvements

Hugging Face updated Math-Verify (v0.5.2) to handle complex mathematical expressions more robustly. Key improvements include:

  • Enhanced parsing for text-only answers and lists of answers.
  • Support for multiple boxed answers within a single LaTeX environment.
  • Introduction of ordered tuples to distinguish between sets and tuples based on gold answers.
  • Support for relational expressions (e.g., $1 < x < 2$) and intervals (e.g., $(1,2)$).

Community Insights on Reasoning and GRPO

Recent community developments highlight a shift in how reasoning capabilities are elicited in open models.

GRPO and Base Model Capabilities

  • Base Model Reasoning: Experiments with GRPO on the Qwen2.5-0.5B base model showed a 10-point improvement over the Instruct version on GSM8k. Some researchers suggest the "aha" moment described in DeepSeek-R1 may be a symptom of the base model's inherent capabilities rather than the RL process alone.
  • Efficiency: Unsloth has optimized GRPO to allow models up to 15B parameters to be trained with only 15GB of VRAM.
  • Non-Verifiable Domains: GRPO has been successfully applied to poetry, demonstrating its utility beyond traditionally verifiable tasks like math and code.

Data Efficiency and Latent Space Reasoning

  • Small-Scale High-Quality Data: Research into datasets like s1K (1,000 samples) and LIMO (817 samples) suggests that if a model has sufficient pre-training knowledge, a very small number of high-quality, well-structured examples can unlock advanced reasoning.
  • Latent Space Reasoning: A recent paper proposes scaling test-time compute by reasoning implicitly in latent space using recurrent language models, which is more compute-efficient than generating extensive natural language "thinking" tokens.

Chain-of-Thought (CoT) Length Control

  • Budget Forcing: This technique extends or truncates reasoning by appending "Wait" or end-of-thinking tokens, allowing for test-time scaling where increased thinking time correlates with higher accuracy.
  • Cosine Reward: A novel reward function used to incentivize shorter CoTs for correct answers and longer CoTs for incorrect ones, stabilizing RL training and preventing reward hacking (where models increase CoT length through repetition to gain rewards).

Sources