huggingface/open-r1

Fully open reproduction of DeepSeek-R1

What it solves

Open R1 is designed to provide a fully open reproduction of the DeepSeek-R1 pipeline. It aims to fill the gaps in the original pipeline so that the community can reproduce the reasoning capabilities of R1 models, specifically focusing on distilling high-quality reasoning traces and implementing the reinforcement learning (RL) pipeline.

How it works

The project implements a multi-stage training approach based on the DeepSeek-R1 technical report. It provides scripts for:

  • Supervised Fine-Tuning (SFT): Performing simple SFT on datasets containing reasoning traces.
  • Group Relative Policy Optimization (GRPO): Training models using RL with a vLLM backend to scale across multiple nodes.
  • Synthetic Data Generation: Using Distilabel to generate synthetic data from existing models.
  • Code Execution Rewards: Integrating with sandbox providers like E2B, Morph, and Piston to provide real-time code execution rewards for training on competitive programming tasks (e.g., Codeforces, IOI).

Who it’s for

This project is for AI researchers and developers who want to reproduce DeepSeek-R1's reasoning capabilities in smaller or open-weights models, or those looking to implement GRPO and SFT distillation pipelines for reasoning models.

Highlights

  • Open Reproduction: A complete pipeline for replicating R1-Distill models and the R1-Zero RL pipeline.
  • Verifiable Rewards: Support for code interpreter rewards using sandboxed environments for high-accuracy training on math and code.
  • Scalable Training: Integration with vLLM and Hugging Face Accelerate (DeepSpeed ZeRO-2/3) for multi-node training on H100 clusters.
  • Curated Datasets: Release of reasoning datasets like Mixture-of-Thoughts and CodeForces-CoTs.

Related

  • Dispatch
    Open-R1: Reproducing DeepSeek-R1 Reasoning Pipelines

    Hugging Face's Open-R1 project provides an open-source framework and datasets to reproduce the DeepSeek-R1 reasoning pipeline through distillation and reinforcement learning.

  • Dispatch
    Open-R1: A Fully Open Reproduction of DeepSeek-R1

    Hugging Face has launched the Open-R1 project to systematically reconstruct the data and training pipeline of DeepSeek-R1, aiming to provide the open-source community with the missing datasets and code for reasoning models.

  • Dispatch
    Open-R1 Update #1: Replicating DeepSeek-R1 Training and Synthetic Data

    Hugging Face provides a first progress update on the Open-R1 project, detailing the reproduction of DeepSeek-R1 evaluation scores, the integration of GRPO into TRL, and strategies for scaling synthetic reasoning data generation.

  • Dispatch
    Open R1 Update #2: OpenR1-Math-220k Dataset and Reasoning Insights

    Hugging Face introduces OpenR1-Math-220k, a large-scale math reasoning dataset designed to reconstruct DeepSeek R1's distillation pipeline, while sharing community insights on GRPO and Chain-of-Thought length control.

  • Project
    hiyouga/EasyR1

    EasyR1 is an open‑source reinforcement‑learning‑from‑human‑feedback framework for training large language and vision‑language models. It supports many state‑of‑the‑art RL algorithms (GRPO, DAPO, GSPO, etc.), works with models like Llama 3 and Qwen 2‑VL, and scales from a single GPU to multi‑node clusters using Ray and FSDP. Features include padding‑free batches, LoRA adaptation, checkpoint merging, and integration with popular experiment trackers.