Open-R1: A Fully Open Reproduction of DeepSeek-R1

Hugging Face has announced Open-R1, an initiative to fully reproduce the training pipeline and datasets of DeepSeek-R1. This project aims to provide the open-source community with the transparency, code, and data necessary to build high-performance reasoning models without relying on proprietary recipes.

The Architecture and Training of DeepSeek-R1

DeepSeek-R1 is a reasoning model built upon the DeepSeek-V3 base, a 671B Mixture of Experts (MoE) model. DeepSeek-V3 is noted for its cost-efficiency, having been trained for approximately $5.5M through the use of Multi-Head Latent Attention (MLA), Multi Token Prediction (MTP), and extensive hardware optimization.

DeepSeek utilized two distinct paths to develop its reasoning capabilities:

  • DeepSeek-R1-Zero: This version skipped supervised fine-tuning (SFT) entirely. It relied solely on reinforcement learning (RL) using Group Relative Policy Optimization (GRPO) and a reward system based on accuracy and structure to develop reasoning skills like self-verification and step-by-step problem solving. While powerful, its outputs often lacked readability.
  • DeepSeek-R1: To solve the readability issues of R1-Zero, this model began with a "cold start" phase using a small set of carefully crafted examples for SFT. This was followed by further RL and refinement, including the rejection of low-quality outputs based on human preference and verifiable rewards.

The Open-R1 Project Goals

While DeepSeek released the model weights for R1, the specific datasets and training code remain proprietary. The Open-R1 project is designed to fill these gaps through a three-step plan:

  1. Distillation: Replicate the R1-Distill models by distilling a high-quality reasoning dataset from DeepSeek-R1.
  2. Pure RL Pipeline: Replicate the R1-Zero pipeline by curating new, large-scale datasets specifically for mathematics, reasoning, and code.
  3. Multi-stage Training: Demonstrate a complete pipeline transitioning from a base model to SFT and then to RL.

Beyond mathematics and coding, Hugging Face intends to explore the application of these reasoning techniques in other scientific fields, such as medicine.

Community Insights and Project Status

Following the announcement, community discussions highlighted several critical points regarding the nature of "open" models and the challenges of reproduction:

  • Evaluation Metrics: Community members noted that a true reproduction requires evaluation numbers to validate performance against the original model. Project contributors clarified that the blog post is an introduction to the project, not a claim of completed reproduction.
  • Data Transparency: Some users expressed concern that releasing weights alone is "open-binary" or freeware rather than truly open source, as the full blueprint and training data are missing.
  • Replication Challenges: Contributors emphasized that without the original hyperparameters and datasets, the project relies on "best guess estimates" to attempt to reach the original performance levels.

"Historically, they [DeepSeek] have never released code or datasets of their LLM training, so I wouldn't expect this time to be different. In the meantime we have to make best guess estimates and see if we can get there ourselves."

The Open-R1 project is being developed in the open, with code hosted on GitHub and models and datasets being uploaded to the Open-R1 organization on Hugging Face.

Sources