PrimeIntellect-ai/prime-rl

Agentic RL Training at Scale

What it solves

prime-rl addresses the challenge of scaling reinforcement learning (RL) for large-scale agentic training. It provides a framework that allows developers to train massive models (up to 1T+ parameters) across thousands of GPUs while maintaining a modular and "hackable" design for rapid experimentation.

How it works

The framework utilizes a fully asynchronous RL architecture to maximize throughput. For training and inference at scale, it integrates several high-performance technologies:

  • Training & Inference: Uses FSDP2 for training and vLLM for inference, incorporating FP8 precision, Expert Parallelism (EP), and Context Parallelism (CP).
  • Deployment: Supports multi-node deployment via Slurm, including advanced configurations like P/D disaggregation and KV offloading.
  • Pipeline: Provides an end-to-end post-training pipeline covering Supervised Fine-Tuning (SFT), RL training, and evaluations.
  • Integration: Connects natively with the Environments Hub and verifiers for agentic and software engineering (SWE) environments.

Who it’s for

It is designed for AI researchers and engineers working on frontier models, specifically those needing to perform large-scale RL post-training on MoE (Mixture-of-Experts) or dense models, and those training VLMs (Vision-Language Models).

Highlights

  • Massive Scalability: Capable of scaling to 1000+ GPUs for training 1T+ MoE models.
  • Broad Model Support: Native support for GLM, Qwen, Nemotron, and other major MoE families.
  • Multimodal Capability: Supports training for VLMs such as Qwen3.5.
  • Flexible Deployment: One-line Slurm deployment for complex frontier model setups.
  • Comprehensive Tooling: Includes built-in support for SFT, RL, and evaluation in a single framework.

Related

  • Project
    ServiceNow/PipelineRL

    A scalable asynchronous reinforcement learning framework for LLM agents that uses in-flight weight updates to maximize GPU utilization while maintaining on-policy data freshness.

  • Project
    alibaba/ROLL

    ROLL is an open‑source library for reinforcement‑learning training of large language models at scale. It provides Ray‑based distributed execution, supports many RL algorithms (PPO, GRPO, Reinforce++, etc.), works with Megatron‑Core, vLLM, SGLang, and LoRA, and offers pipelines for RL‑based fine‑tuning (RLVR) and multi‑turn agentic RL. The project includes extensive docs, hardware‑agnostic support (NVIDIA, AMD, Ascend), and built‑in observability.

  • Project
    verl-project/verl

    A flexible and efficient RL training library for LLMs that simplifies the implementation of RLHF algorithms and scales to trillion-parameter models.

  • Project
    Tencent-Hunyuan/UniRL

    UniRL is an Apache‑2.0 Python framework that applies a single reinforcement‑learning loop to many multimodal generative models (diffusion, video, LLM, VLM, and unified AR+diffusion). It offers composable trainers, multiple rollout back‑ends (vLLM‑Omni, SGLang), standard and novel RL algorithms (GRPO, Flow‑DPPO, DRPO, CPPO), and ready‑to‑run recipes for dozens of state‑of‑the‑art models. Users launch training via Hydra configs, and the system handles distributed rollout, scoring, advantage computation, and weight syncing.

  • Dispatch
    Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

    Hugging Face surveyed 16 open-source RL libraries and found that async RL training separates inference and training onto different GPU pools, uses a rollout buffer, and pushes weights asynchronously, with Ray dominating orchestration and NCCL broadcast the common weight‑sync method.