verl-project/verl

verl/HybridFlow: A Flexible and Efficient RL Post-Training Framework

What it solves

verl is a production-ready reinforcement learning (RL) training library designed specifically for large language models (LLMs). It addresses the complexity of post-training alignment, allowing developers to implement diverse RL algorithms (like PPO and GRPO) with minimal code while maintaining high throughput and scalability across large GPU clusters.

How it works

The library uses a hybrid-controller programming model (HybridFlow) that decouples computation and data dependencies. This allows it to integrate seamlessly with various LLM infrastructure components:

  • Training Backends: Supports FSDP, FSDP2, and Megatron-LM.
  • Rollout Generation: Integrates with vLLM, SGLang, and Hugging Face Transformers.
  • Resource Management: Uses flexible device mapping to place models on different GPU sets for optimal resource utilization.
  • Efficiency: Employs a 3D-HybridEngine to reduce memory redundancy and communication overhead when switching between training and generation phases.

Who it’s for

It is intended for researchers and engineers working on LLM post-training, specifically those needing to scale RLHF (Reinforcement Learning from Human Feedback) to very large models (up to 671B parameters) and those implementing advanced reasoning or multi-modal RL.

Highlights

  • Algorithm Support: Ready-to-use implementations of PPO, GRPO, GSPO, ReMax, RLOO, and more.
  • Broad Compatibility: Works with popular Hugging Face models (Qwen, Llama, Gemma, DeepSeek) and supports NVIDIA, AMD, and Ascend hardware.
  • Advanced Capabilities: Supports vision-language models (VLMs), multi-turn tool calling, and LoRA RL to save memory.
  • High Scalability: Capable of scaling to trillion-parameter models on limited hardware via expert parallelism and Megatron-bridge.

Related

  • Project
    verl-project/verl-recipe

    A collection of version-controlled recipes for the verl reinforcement learning library, providing automated environment setup and dependency management for various LLM training experiments.

  • Project
    alibaba/ROLL

    ROLL is an open‑source library for reinforcement‑learning training of large language models at scale. It provides Ray‑based distributed execution, supports many RL algorithms (PPO, GRPO, Reinforce++, etc.), works with Megatron‑Core, vLLM, SGLang, and LoRA, and offers pipelines for RL‑based fine‑tuning (RLVR) and multi‑turn agentic RL. The project includes extensive docs, hardware‑agnostic support (NVIDIA, AMD, Ascend), and built‑in observability.

  • Project
    opendilab/LightRFT

    LightRFT is an open‑source, distributed reinforcement‑learning framework for fine‑tuning large language and multimodal generative models. It unifies training back‑ends (DeepSpeed, FSDP) with rollout engines (SGLang, vLLM), supports a range of RL algorithms (GRPO, REINFORCE++, DAPO, etc.), and works across text, image, video, and audio modalities. The library provides ready‑to‑run examples (e.g., GSM8K with Qwen2.5), extensive docs, and Docker images, making it suitable for researchers and engineers building RL‑based alignment pipelines.

  • Project
    areal-project/AReaL

    AReaL is a scalable, asynchronous reinforcement learning infrastructure designed to train reasoning and agentic models by decoupling training, inference, and agent execution into microservices.

  • Project
    Tencent-Hunyuan/UniRL

    UniRL is an Apache‑2.0 Python framework that applies a single reinforcement‑learning loop to many multimodal generative models (diffusion, video, LLM, VLM, and unified AR+diffusion). It offers composable trainers, multiple rollout back‑ends (vLLM‑Omni, SGLang), standard and novel RL algorithms (GRPO, Flow‑DPPO, DRPO, CPPO), and ready‑to‑run recipes for dozens of state‑of‑the‑art models. Users launch training via Hydra configs, and the system handles distributed rollout, scoring, advantage computation, and weight syncing.