Open-R1: Reproducing DeepSeek-R1 Reasoning Pipelines
Hugging Face has launched Open-R1, a project dedicated to the fully open reproduction of the DeepSeek-R1 reasoning pipeline. By providing the necessary scripts, recipes, and curated datasets, Open-R1 enables the community to replicate the reasoning capabilities of DeepSeek-R1, specifically focusing on the transition from base models to RL-tuned reasoning models.
Project Roadmap and Current Progress
The Open-R1 project follows a three-step "plan of attack" based on the DeepSeek-R1 technical report to democratize the creation of reasoning models:
- Replicate R1-Distill models: Distill a high-quality corpus from DeepSeek-R1 to create smaller, capable reasoning models.
- Replicate the pure RL pipeline: Reproduce the R1-Zero pipeline using large-scale datasets for mathematics, reasoning, and code.
- Multi-stage training: Demonstrate the path from a base model to an RL-tuned model via multi-stage training.
As of May 26, 2025, Step 1 is completed. Hugging Face released the Mixture-of-Thoughts dataset, containing 350k verified reasoning traces distilled from R1 across math, coding, and science. This dataset was used to train OpenR1-Distill-7B, which replicates the performance of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B.
Training Methodologies: SFT and GRPO
Open-R1 supports two primary training paradigms to achieve reasoning capabilities:
Supervised Fine-Tuning (SFT)
SFT is used for distillation. By training on datasets like Mixture-of-Thoughts, developers can quickly instill reasoning traces into a base model. For example, the OpenR1-Distill-7B model achieved a score of 52.7 on AIME 2024 and 89.0 on MATH-500, closely matching or exceeding the original DeepSeek-R1-Distill-Qwen-7B.
Group Relative Policy Optimization (GRPO)
To scale training and move toward the R1-Zero approach, Open-R1 implements GRPO. The project leverages the vLLM backend via TRL to scale training across multiple nodes.
A key feature of the GRPO implementation is the code reward function, which allows the model to be rewarded based on the actual execution of generated code. This is supported through sandboxed environments:
- E2B: Cloud-based sandboxes optimized for Python.
- Morph: Cloud-based sandboxes supporting Python, JS, C++, and Rust.
- Piston: An alternative execution provider for IOI and CodeForces problems.
Evaluation and Benchmarking
Open-R1 uses lighteval to reproduce DeepSeek's reported results. To ensure accuracy, the project emphasizes the importance of sampling multiple responses per query to estimate pass@1 accuracy, particularly for high-variance benchmarks like AIME 2024 (where 64 responses per query are used).
Reproduction Results
Open-R1 successfully reproduced DeepSeek's results within 1-3 standard deviations across several key benchmarks:
| Benchmark | DeepSeek-R1-Distill-Qwen-32B (Open-R1 Eval) | DeepSeek-R1-Distill-Qwen-32B (DeepSeek Reported) |
|---|---|---|
| AIME 2024 | 69.7 | 72.6 |
| MATH-500 | 95.6 | 94.3 |
| GPQA Diamond | 63.1 | 62.1 |
| LiveCodeBench | 56.0 | 57.2 |
Data Generation and Decontamination
The project provides a full pipeline for generating synthetic reasoning data using Distilabel. Users can generate data from either small distilled R1 models (on a single H100) or the full DeepSeek-R1 model (requiring multiple nodes and vLLM dev wheels to fix CUDA graph capture issues).
To maintain benchmark integrity, Open-R1 includes a decontamination script (scripts/decontaminate.py) that uses 8-grams to identify and remove contaminated samples from training datasets.
Community Insights and Counterpoints
While the project provides a significant technical framework, community discussion on Hacker News highlights some skepticism regarding the "fully open" nature of the reproduction:
"Doesn't look like they managed to actually reproduce R1, and only stopped on Step 1 out of their 3-step plan."
Other contributors suggested looking at projects like OLMo or Nemotron for more comprehensive open training pipelines, or OpenThoughts for advanced data curation methodologies and models that outperform DeepSeek's smaller reasoning variants.