Open-R1 Update #1: Replicating DeepSeek-R1 Training and Synthetic Data
Hugging Face has launched the Open-R1 project to replicate the training pipeline and synthetic data generation of DeepSeek-R1. After one week of development, the project has successfully reproduced DeepSeek's reported results on the MATH-500 benchmark and integrated Grouped Relative Policy Optimization (GRPO) into the TRL library.
Reproducing DeepSeek-R1 Evaluation and Performance
Hugging Face has verified that it can match the evaluation scores reported by DeepSeek on the MATH-500 benchmark using the lighteval tool. The reproduced results are as follows:
| Model | MATH-500 (HF lighteval) | MATH-500 (DeepSeek Reported) |
|---|---|---|
| DeepSeek-R1-Distill-Qwen-1.5B | 81.6 | 83.9 |
| DeepSeek-R1-Distill-Qwen-7B | 91.8 | 92.8 |
| DeepSeek-R1-Distill-Qwen-14B | 94.2 | 93.9 |
| DeepSeek-R1-Distill-Qwen-32B | 95.0 | 94.3 |
| DeepSeek-R1-Distill-Llama-8B | 85.8 | 89.1 |
| DeepSeek-R1-Distill-Llama-70B | 93.4 | 94.5 |
Challenges with Response Length
A significant finding during evaluation is the extreme length of DeepSeek-R1's generations. In the OpenThoughts dataset, the average response length is 6,000 tokens, with some exceeding 20,000 tokens. This length poses a challenge for GRPO training, as generating long completions requires substantial GPU memory to store activations and gradients during the optimization step.
Training Pipeline and GRPO Integration
Grouped Relative Policy Optimization (GRPO) has been integrated into TRL version 0.14. This implementation allows for training models using one or multiple reward functions and integrates with DeepSpeed ZeRO 1/2/3 for parallelized scaling across multiple GPUs. To address the primary bottleneck of online training—generation speed—the pipeline utilizes vLLM for fast inference.
Scaling Synthetic Data Generation
To replicate the synthetic reasoning traces used to improve smaller models, Hugging Face developed a scaled inference pipeline to generate data using the large R1 model.
Infrastructure and Throughput Optimization
Initial tests using two 8xH100 nodes with vLLM showed sub-optimal throughput because the GPU KV cache filled too quickly, leading to request preemption. To resolve this, the team expanded the setup to 32 GPUs (four 8xH100 nodes), providing sufficient VRAM to handle 32 parallel requests without rescheduling.
Transition to Streaming Requests
To stabilize GPU utilization and eliminate delays caused by "stragglers" in batched inference, the team switched from batching requests to a streaming approach. By maintaining a constant number of active tasks (e.g., 500) and initiating new requests as soon as others complete, they achieved a more consistent generation rate.
Community Ecosystem and Dataset Development
Since the release of DeepSeek-R1, the open-source community has produced several projects and datasets aimed at reproducing reasoning capabilities at a smaller scale:
Notable Community Projects
- TinyZero: Demonstrates the "aha-moment" of reasoning using a 3B base model for under $30.
- Mini-R1: A tutorial providing a guide to reproducing reasoning emergence.
- HKUST Researchers: Demonstrated the emergence of reasoning in a 7B math model.
- Evolving LLM Lab: Developing a multimodal version of R1.
Key Synthetic Reasoning Datasets
- OpenThoughts-114k: 114,000 high-quality examples covering science, math, code, and puzzles.
- Bespoke-Stratos-17k: A replication of the Berkeley Sky-T1 pipeline using DeepSeek-R1 to create reasoning traces.
- Dolphin-r1: 800,000 samples combining completions from DeepSeek-R1, Gemini Flash, and Dolphin Chat.
- Magpie-Reasoning-V2-250K: 250,000 reasoning responses generated by DeepSeek-R1-Distill-Llama-70B.
Industry Context and Market Reaction
The release of DeepSeek-R1 has triggered widespread industry responses. OpenAI CEO Sam Altman noted the discovery of similar ideas to those used in o1, while Anthropic CEO Dario Amodei emphasized the need for export controls. Additionally, major providers including AWS (via BedRock and SageMaker), Dell, Together AI, and Fireworks AI have integrated DeepSeek-R1 into their platforms.
Sources
- OriginalOpen-R1: Update #1