ServiceNow/PipelineRL

A scalable asynchronous reinforcement learning implementation with in-flight weight updates.

What it solves

PipelineRL addresses the efficiency trade-off in reinforcement learning (RL) for large language models (LLMs). Specifically, it solves the conflict between achieving high inference throughput (which requires large batches across many GPUs) and maintaining "on-policy" data freshness (ensuring the model is trained on data generated by its most recent version).

How it works

PipelineRL uses a scalable asynchronous architecture with "in-flight weight updates." Instead of stopping the entire sampling process to update the model, updated weights are broadcast to inference servers via NCCL immediately after each optimizer step. This allows the system to continue generating rollouts while updating the policy, keeping the data near on-policy without sacrificing GPU utilization.

The system is organized into six modular components:

  1. Orchestrator: Manages GPU placement and launches subprocesses.
  2. Inference Servers: vLLM-based servers that handle sampling and receive weight updates.
  3. Actor Processes: Generate raw rollouts by sampling from the LLM and collecting rewards.
  4. Preprocessor: Tokenizes sequences and computes advantages.
  5. Trainer: Performs the RL step (using a simplified GRPO algorithm) and triggers weight updates.
  6. Verifier: An optional server that checks the correctness of model outputs (e.g., for math tasks).

Who it’s for

It is designed for researchers and developers training LLM agents, particularly those focusing on reasoning tasks (like math or coding) where rewards can be verified, and who need to scale training across multiple GPUs.

Highlights

  • In-flight Weight Updates: Updates model parameters without halting the sampling pipeline.
  • Agent Agnostic: Can be adapted to any agent task by implementing load_problems and generate_rollout functions.
  • High Performance: Matches or exceeds Open-Reasoner-Zero on AIME-2024 and MATH-500 benchmarks.
  • Flexible Backend: Supports both file-system and Redis-based streaming for inter-process communication.
  • Sandbox Integration: Supports SandboxFusion for executing and verifying code in remote sandboxes.

Related

  • Dispatch
    PipelineRL: Optimizing LLM Reinforcement Learning via Inflight Weight Updates

    Hugging Face and ServiceNow Research have open-sourced PipelineRL, an experimental RL implementation that uses inflight weight updates to eliminate the trade-off between inference throughput and on-policy data collection.

  • Project
    PrimeIntellect-ai/prime-rl

    A framework for large-scale asynchronous reinforcement learning designed to train 1T+ parameter MoE models across 1000+ GPUs using FSDP2 and vLLM.

  • Project
    MoonshotAI/checkpoint-engine

    A middleware for LLM inference engines that enables rapid, in-place model weight updates across large GPU clusters, essential for reinforcement learning.

  • Dispatch
    vLLM Native RL APIs Release

    vLLM has introduced native weight syncing APIs and improved asynchronous RL support to standardize weight transfer between training and inference and eliminate deadlocks in large-scale DPEP deployments.

  • Project
    areal-project/AReaL

    A large-scale asynchronous reinforcement learning system designed for training high-performance reasoning and agentic AI models.