Mini-R1: Reproducing DeepSeek-R1 Reasoning via GRPO and the Countdown Game

Hugging Face has published a technical tutorial demonstrating how to reproduce the "aha moment" observed in DeepSeek-R1—where a model learns to allocate more thinking time and re-evaluate its approach without human feedback—using Group Relative Policy Optimization (GRPO) and a mathematical puzzle called the Countdown Game.

Group Relative Policy Optimization (GRPO) Explained

GRPO is a reinforcement learning (RL) algorithm designed to enhance the reasoning capabilities of Large Language Models (LLMs). Introduced in the DeepSeekMath paper, GRPO modifies traditional Proximal Policy Optimization (PPO) by removing the need for a value function model. Instead, it estimates baselines from group scores, which significantly reduces memory usage and computational overhead.

The GRPO process follows four primary steps:

  1. Sampling: The current policy generates multiple outputs for a single prompt.
  2. Reward Scoring: Each generation is scored via a reward function (either rule-based or outcome-based).
  3. Advantage Calculation: The average reward of the group serves as the baseline; the advantage of each individual solution is computed relative to this normalized group average.
  4. Policy Optimization: The policy is optimized to maximize the GRPO objective, incorporating the calculated advantages and a KL divergence term.

Technical Implementation and Setup

The Mini-R1 experiment utilized the following technical stack and configuration:

  • Base Model: Qwen/Qwen2.5-3B-Instruct. The choice of a 3B parameter model is based on observations that models generally require >1.5B parameters to effectively learn the reasoning process.
  • Dataset: Jiayi-Pan/Countdown-Tasks-3to4, consisting of puzzles with 3 to 4 numbers.
  • Hardware: 4x NVIDIA H100 80GB GPUs.
  • Software Stack: Hugging Face trl (for the GRPOTrainer), transformers, datasets, accelerate, deepspeed, and vLLM for accelerated generation.

Reward Functions

To verify correctness without human labels, the training employed two rule-based reward functions:

  • Format Reward: Ensures the model follows the structure <think> [thinking] </think><answer> [answer] </answer>.
  • Accuracy Reward: Extracts the equation from the <answer> tag and verifies if it evaluates to the target number and uses each provided number exactly once.

Distributed Training Configuration

Training was executed using DeepSpeed Zero-3 and vLLM. In a 4-GPU setup, three GPUs were dedicated to training (num_processes 3) while the final GPU was reserved for vLLM generation. A full training run of 450 steps took approximately 6 hours, with each step taking 45-60 seconds.

Training Results and Observations

Training performance was tracked via TensorBoard, with checkpoints saved every 25 steps. The model's behavior evolved through three distinct phases:

  • Step 50: The model successfully learned the required <think> and <answer> formatting.
  • Step 100: The success rate reached approximately 25%. The model began "reasoning" using natural language descriptions of its trial-and-error process.
  • Step 200: The success rate reached ~40%. The model shifted from natural language reasoning to a "programmatic execution" style, where it listed multiple combinations and reviewed results systematically.
  • Step 450: The success rate reached 50%. The model maintained the programmatic reasoning format, and performance continued to improve slowly.

Hyperparameter Tuning

Initial attempts using DeepSeekMath hyperparameters (learning rate 1e-6, beta 0.04) resulted in unstable training after 150 steps. Stability was achieved by decreasing the learning rate to 5e-7 and the beta (KL coefficient) to 0.001.

Analysis of Reasoning Shifts

The transition from word-based reasoning to programmatic execution is attributed to several potential factors:

  • Model Capacity: Qwen 2.5 3B may be too small to sustain complex natural language reasoning compared to the larger models used by DeepSeek.
  • Reward Specification: The reward functions may have inadvertently encouraged "reward hacking," where the model found a more efficient mathematical shorthand than natural language.
  • Task Specificity: Training exclusively on the Countdown Game may have naturally pushed the model toward the most effective solving method for that specific puzzle type.
  • Training Duration: The model may not have been trained long enough; the original R1 paper documented training for over 8,000 steps.

Sources