rllm-org/rllm

Democratizing Reinforcement Learning for LLMs

What it solves

rLLM simplifies the process of training language agents using reinforcement learning (RL). It removes the friction of rewriting agent code for different environments or training backends, allowing the same agent logic to be used for both evaluation and training across various benchmarks and sandboxes.

How it works

rLLM uses a pipeline that runs an agent, collects traces, computes rewards, and updates the model. It employs a model gateway that transparently captures token IDs and logprobs from LLM calls via URL-routed sessions. This allows the framework to structure agent interactions into Episodes, Trajectories, and Steps without requiring changes to the agent's original code. The framework supports multiple training backends (such as verl, tinker, and fireworks) and can run agents in various sandboxes (like Docker, Daytona, or Modal) with snapshot and warm-pool acceleration to reduce costs.

Who it’s for

AI researchers and industry teams developing agentic programs that need a standardized way to train and evaluate them using RL methods like GRPO, REINFORCE, and RLOO.

Highlights

  • Backend Agnostic: Switch between distributed multi-GPU training (verl), single-machine training (tinker), and the Fireworks platform with a single flag.
  • Extensive Benchmark Integration: Includes over 60 integrated benchmarks covering math, code, QA, and agentic tasks (e.g., SWE-bench, AIME).
  • Flexible Harnesses: Supports 10+ CLI harnesses and allows users to wrap their own agents (e.g., LangGraph or OpenAI Agents SDK) using a simple decorator.
  • Sandbox Support: Compatible with multiple sandbox environments to ensure safe and scalable rollouts.
  • Proven Results: Used to create state-of-the-art open-source models like DeepScaleR, DeepCoder, and DeepSWE.

Related

  • Project
    microsoft/agent-lightning

    A lightweight reinforcement learning framework that allows developers to train AI agents using their existing harnesses with minimal code changes.

  • Project
    OpenManus/OpenManus-RL

    An open-source framework for optimizing LLM agents using reinforcement learning to enhance their reasoning, tool use, and decision-making capabilities.

  • Project
    OpenRLHF/OpenRLHF

    A high-performance RLHF framework that combines Ray and vLLM to scale reinforcement learning from human feedback for LLMs and VLMs.

  • Project
    InternLM/InternBootcamp

    A scalable synthetic environment framework for training and evaluating LLM agents through executable, interactive, and verifiable tasks.

  • Project
    agentscope-ai/Trinity-RFT

    A general-purpose framework for reinforcement fine-tuning of LLMs that decouples exploration, training, and data buffering to support diverse RL algorithms and agentic workflows.