alibaba/ROLL

An Efficient and User-Friendly Scaling Library for Reinforcement Learning with Large Language Models

📚 What is ROLL?

ROLL (Reinforcement Learning Optimization for Large‑Scale Learning) is an open‑source library that lets you train large language models (LLMs) with reinforcement‑learning (RL) techniques at the scale of dozens or hundreds of GPUs. It focuses on two major use‑cases:

Use‑case What it does Typical tasks
RL‑based fine‑tuning (RLVR) Runs RL‑style “reward‑model → policy” loops on LLMs for math, coding, reasoning, instruction‑following, etc. Align LLMs to human preferences, improve reasoning, generate higher‑quality answers
Agentic RL Trains LLMs that interact over multiple turns with environments (games, tool‑use, dialogue). Multi‑turn game playing, tool‑use agents, interactive assistants

The library is built to work on large‑scale GPU clusters (or even Ascend NPUs) and hides the engineering complexity of distributed training.

🚀 Core Features (in plain language)

Feature Why it matters
Ray‑based multi‑role architecture – Different parts of the RL loop (actor, learner, reward model, rollout workers) run as independent Ray actors, so you can scale from a single node to thousands of GPUs without rewriting code.
Back‑end flexibility – Plug‑in support for Megatron‑Core, vLLM, SGLang, FSDP2, and LoRA. Choose the fastest inference engine (vLLM, SGLang) and the most memory‑efficient training strategy (Megatron 5‑D parallelism, FSDP2).
Asynchronous parallel rollout – Rollouts (environment steps) and reward calculations happen in parallel, keeping GPUs busy and reducing idle time.
Rich RL algorithm library – Ready‑to‑use implementations of PPO, GRPO, Reinforce++, TOPR, RAFT++, GSPO, StarPO, GiGPO, etc., plus a configurable “strategy” system for reward clipping, normalization, advantage estimation, etc.
Agentic‑specific tools – Support for step‑wise (GiGPO) and trajectory‑wise (StarPO) training, tool‑use environments, and local debugging of multi‑turn interactions.
Hardware‑agnostic – Works on NVIDIA GPUs, AMD GPUs (Docker images provided), and Alibaba Ascend NPUs.
Observability – Built‑in logging to SwanLab, Weights & Biases, TensorBoard; per‑domain and per‑reward metrics are tracked automatically.
Checkpointing & resume – Automatic saving of model, optimizer, and Ray state so long jobs can be paused and continued.

🛠️ How to Get Started (quick outline)

  1. Install – Follow the online docs; a Docker image is available for AMD GPUs, otherwise install via pip and Ray.
  2. Pick a backend – Choose Megatron‑Core for full‑scale training, or vLLM/SGLang for fast inference during rollouts.
  3. Write a config – ROLL uses a hierarchical YAML config system. You specify the model (e.g., qwen3.5-35B), the RL algorithm (e.g., ppo), resource mapping, and any special options (LoRA, FP8 rollout, etc.).
  4. Run a pipeline – Use one of the provided scripts, e.g.:
    python run_rlvr.py --config examples/qwen2.5-7B-rlvr_megatron/rlvr_config.yaml
    Similar scripts exist for agentic, distill, and dpo pipelines.
  5. Monitor – Open the WandB/TensorBoard dashboard to watch reward curves, throughput, and GPU utilization.
  6. Resume – If the job stops, rerun the same command; ROLL will pick up from the latest checkpoint.

📖 Documentation & Community

  • Official docs – https://alibaba.github.io/ROLL/ (installation, config system, debugging guide, pipeline tutorials).
  • Paper & tech reports – Several arXiv papers (e.g., Asymmetric PPO, RollPacker, Agentic Crafting on Rock and Roll) describe the algorithms and performance gains.
  • Slack/WeChat – QR codes in the repo for community chat.
  • GitHub – Issues are actively triaged; the repo has > 200 stars and frequent releases.

🎯 Who Should Use ROLL?

  • Research labs that need to experiment with RL‑based LLM alignment or multi‑turn agentic training at scale.
  • Industry teams building LLM‑powered assistants, code‑generation tools, or game‑playing agents that require large‑scale RL fine‑tuning.
  • Engineers who want a ready‑made, Ray‑powered scaffolding so they can focus on the reward model or environment rather than distributed system plumbing.

📌 Bottom Line

ROLL is a production‑grade, highly configurable RL framework for large language models. It abstracts away the heavy lifting of distributed training, supports a wide range of RL algorithms, and works on many hardware back‑ends. If you need to align or teach an LLM through reinforcement learning—especially in multi‑turn, tool‑using, or large‑scale settings—ROLL gives you a solid, open‑source foundation.

Related

  • Project
    verl-project/verl

    A flexible and efficient RL training library for LLMs that simplifies the implementation of RLHF algorithms and scales to trillion-parameter models.

  • Project
    NVIDIA-NeMo/RL

    A scalable post-training library for multimodal models that streamlines reinforcement learning and alignment using high-performance backends like Megatron Core and vLLM.

  • Project
    OpenRLHF/OpenRLHF

    A high-performance RLHF framework that combines Ray and vLLM to scale reinforcement learning from human feedback for LLMs and VLMs.

  • Project
    rl-tools/rl-tools

    A fast, portable C++ library for deep reinforcement learning and continuous control, optimized for everything from microcontrollers to GPUs.

  • Project
    radixark/miles

    An enterprise-grade reinforcement learning framework for large-scale model post-training that integrates SGLang and Megatron-LM for high-throughput rollout and scalable training.