alibaba/ROLL

What it solves

ROLL is a scaling library designed to make reinforcement learning (RL) for Large Language Models (LLMs) more efficient and user-friendly, especially when using large-scale GPU clusters. It addresses the challenges of human preference alignment, complex reasoning, and multi-turn agentic interactions by providing a distributed architecture that optimizes both training and inference.

How it works

ROLL uses a multi-role distributed architecture powered by Ray for flexible resource allocation and task scheduling. It integrates several high-performance backends to accelerate the process:

• Inference/Generation: Uses vLLM and SGLang.
• Training: Supports FSDP2 and Megatron-LM 5D parallelism (including data, tensor, pipeline, context, and expert parallelism).
• Optimization: Implements sample-level asynchronous parallel rollouts and asynchronous training to reduce bottlenecks.
• Device Management: Features "AutoDeviceMapping" to manage how different roles are deployed across GPUs.

Who it’s for

It is intended for researchers and developers working on post-training LLMs, specifically those focusing on RLVR (Reinforcement Learning from Verifiable Rewards), agentic RL for multi-turn interactions, and distillation pipelines for both LLMs and VLMs (Vision-Language Models).

Highlights

• Comprehensive RL Algorithm Support: Out-of-the-box support for PPO, GRPO, Reinforce++, TOPR, RAFT++, and GSPO.
• Multi-Task Capabilities: Handles mathematics, coding, general reasoning, and instruction following.
• Agentic RL Support: Specialized tools for multi-turn dialogues, games, and tool use, supporting both trajectory-wise and step-wise training paradigms.
• Hardware Flexibility: Compatible with NVIDIA GPUs, AMD GPUs, and Ascend NPUs.
• Advanced Training Features: Supports LoRA training, FP8 rollout, and extreme offload/reload capabilities to maximize GPU utilization.