Open R1 Update #3: OlympicCoder and Code Reasoning Insights

Hugging Face has released OlympicCoder, a pair of fine-tuned 7B and 32B code models that outperform closed-source frontier models, including Claude 3.7 Sonnet, on challenging International Olympiad in Informatics (IOI) problems. This release is part of the Open R1 project's effort to reproduce the competitive programming aspects of the DeepSeek-R1 recipe through the creation of specialized datasets and the distillation of reasoning traces.

OlympicCoder and the IOI Benchmark

OlympicCoder-32B outperforms all tested open-weight models and surpasses o1-mini and DeepSeek-R1 in a 50-submission limit setting. To evaluate these capabilities, Hugging Face developed a new benchmark based on the 2024 International Olympiad in Informatics (IOI), which tests complex algorithmic problems with full test sets available under a CC-BY license.

Submission Strategy and Evaluation

To simulate real contest conditions, Hugging Face employed a round-robin submission strategy where a solution's score is only known after submission. This strategy prioritizes submissions targeting the hardest subtasks first and prefers longer generations for reasoning models. Under these strict contest conditions, no model reached the medal threshold (the 50th percentile of human contestants), though o1 came closest to the bronze level.

New Code Reasoning Datasets

To train OlympicCoder, Hugging Face released two primary datasets based on CodeForces, a popular competitive programming platform:

  • open-r1/codeforces: A dataset of over 10,000 problems, approximately 3,000 of which were not in previous datasets like DeepMind's CodeContests. About 60% of these include editorials (organizer explanations).
  • open-r1/codeforces-cots: Nearly 100,000 chain-of-thought (CoT) samples distilled from DeepSeek-R1, providing solutions in C++ and Python.

The Code Verifiability Crisis

Hugging Face identified a "verifiability crisis" in existing competitive programming datasets. Many datasets only contain short test cases (often capped at 500 characters), leading to a situation where models pass public tests but fail on the full test suite. This finding motivated the shift toward the IOI benchmark, which provides fully available and verifiable problem data.

Technical Lessons for Training Reasoning Models

Through SFT experiments using Qwen2.5 Coder Instruct as a base, Hugging Face identified five key lessons for training models on R1 reasoning traces:

  1. Sample packing hurts reasoning performance: Concatenating training samples into equal-sized chunks significantly degraded the model's ability to solve problems, likely because long reasoning traces were clipped or split across chunks.
  2. Higher learning rates are more effective: Using a learning rate of 4e-5 provided significant performance gains (nearly 10 points on LiveCodeBench) compared to the standard 2e-5 used for most Qwen SFT experiments.
  3. Editorials do not boost performance: Including official editorials in the prompt for R1 distillation did not improve the resulting model's performance; naive sampling from problem statements was slightly more effective.
  4. Prefill with <think> tokens: To consistently trigger long CoT behavior across both in-domain and out-of-domain queries, the assistant's response should be prefilled with the <think> token in the chat template.
  5. 8-bit optimizers for long context: To avoid Out-of-Memory (OOM) errors when scaling to 32B models with long contexts, combining FSDP with the paged_adamw_8bit optimizer allowed context scaling up to 22,528 tokens.

GRPO and Math Dataset Updates

GRPO Enhancements in TRL

Hugging Face has updated the Group Relative Policy Optimization (GRPO) implementation in the TRL library with several efficiency and scalability improvements:

  • Generation Reuse: Samples can now be reused multiple times (recommended $\mu$ between 2 and 4) to speed up optimization.
  • Reward Weighting: Users can now assign different weights to different reward functions (e.g., prioritizing correctness over formatting).
  • Integration: Added PEFT and vLLM integration, gradient checkpointing, and optimized selective log softmax computation.

Open R1 Math-Dataset Improvements

The OpenR1-Math-Raw dataset has been enriched with reparsed_answers (extracted via Llama-3.3-70B-Instruct) and a correctness column. Ablation studies showed that while strict verification (e.g., combining Llama verification with math_verify) significantly boosts early-stage performance, the performance gap diminishes over longer training runs where having more samples—even incorrect ones—becomes beneficial.

Future Roadmap

Hugging Face plans to focus on the following next steps:

  • Perfecting the blend of distilled datasets for general-purpose reasoners.
  • Scaling GRPO to larger models, such as Qwen2.5-Coder-32B-Instruct, to create R1-Zero variants.
  • Integrating reward signals from multiple domains and utilizing reward models for non-reasoning data.

Sources