LeapLabTHU/limit-of-RLVR

repo for paper https://arxiv.org/abs/2504.13837

What it solves

This project investigates whether Reinforcement Learning with Verifiable Rewards (RLVR) actually increases the fundamental reasoning capacity of Large Language Models (LLMs) or if it simply optimizes how the model accesses its existing knowledge. It addresses the gap in understanding whether RL-trained models can truly solve problems that their base versions cannot, or if they just become more efficient at finding correct answers through sampling.

How it works

The researchers use the pass@k metric to evaluate models. By increasing the number of attempts (k) a model is given to solve a problem, they can determine the model's "reasoning boundary." If a base model eventually finds the correct answer given enough attempts, but an RL-trained model does not, it suggests that RL has improved sampling efficiency (making the correct answer appear more often) but actually reduced the overall reasoning capacity.

Who it’s for

  • AI researchers studying the effects of reinforcement learning on LLM reasoning.
  • Developers working on reasoning-focused models (like o1 or DeepSeek-R1) who want to understand the limits of RLVR.
  • ML engineers evaluating themodel performance across different sampling strategies.

Highlights

  • Sampling Efficiency vs. Capacity: Finds that RL boosts sampling efficiency (better performance at small k) but can reduce the reasoning capacity boundary (worse performance at large k).
  • Base Model Superiority: Demonstrates that base models consistently catch up to and eventually surpass RL-trained models as the number of samples increases.
  • Algorithm Comparison: Observes that various RLVR algorithms perform similarly and are currently far from optimal.
  • vLLM Integration: Provides implementation details for ensuring response diversity using seed control and intra-run sampling in vLLM.

Related

  • Project
    opendilab/LightRFT

    LightRFT is an open‑source, distributed reinforcement‑learning framework for fine‑tuning large language and multimodal generative models. It unifies training back‑ends (DeepSpeed, FSDP) with rollout engines (SGLang, vLLM), supports a range of RL algorithms (GRPO, REINFORCE++, DAPO, etc.), and works across text, image, video, and audio modalities. The library provides ready‑to‑run examples (e.g., GSM8K with Qwen2.5), extensive docs, and Docker images, making it suitable for researchers and engineers building RL‑based alignment pipelines.

  • Project
    mll-lab-nu/VAGEN

    VAGEN is a reinforcement learning framework for training multi-turn VLM agents that improves performance by reinforcing their internal world model through explicit visual state reasoning.

  • Project
    ServiceNow/PipelineRL

    A scalable asynchronous reinforcement learning framework for LLM agents that uses in-flight weight updates to maximize GPU utilization while maintaining on-policy data freshness.

  • Project
    OpenManus/OpenManus-RL

    An open-source framework for optimizing LLM agents using reinforcement learning to enhance their reasoning, tool use, and decision-making capabilities.

  • Project
    TsinghuaC3I/MARTI

    An open-source framework for training LLM-based multi-agent systems using reinforcement learning, featuring tree search augmentation for complex reasoning and code generation.