verl-project/verl

verl/HybridFlow: A Flexible and Efficient RL Post-Training Framework

What it solves

verl is a production-ready reinforcement learning (RL) training library designed specifically for large language models (LLMs). It addresses the complexity of post-training alignment, allowing developers to implement diverse RL algorithms (like PPO and GRPO) with minimal code while maintaining high throughput and scalability across large GPU clusters.

How it works

The library uses a hybrid-controller programming model (HybridFlow) that decouples computation and data dependencies. This allows it to integrate seamlessly with various LLM infrastructure components:

  • Training Backends: Supports FSDP, FSDP2, and Megatron-LM.
  • Rollout Generation: Integrates with vLLM, SGLang, and Hugging Face Transformers.
  • Resource Management: Uses flexible device mapping to place models on different GPU sets for optimal resource utilization.
  • Efficiency: Employs a 3D-HybridEngine to reduce memory redundancy and communication overhead when switching between training and generation phases.

Who it’s for

It is intended for researchers and engineers working on LLM post-training, specifically those needing to scale RLHF (Reinforcement Learning from Human Feedback) to very large models (up to 671B parameters) and those implementing advanced reasoning or multi-modal RL.

Highlights

  • Algorithm Support: Ready-to-use implementations of PPO, GRPO, GSPO, ReMax, RLOO, and more.
  • Broad Compatibility: Works with popular Hugging Face models (Qwen, Llama, Gemma, DeepSeek) and supports NVIDIA, AMD, and Ascend hardware.
  • Advanced Capabilities: Supports vision-language models (VLMs), multi-turn tool calling, and LoRA RL to save memory.
  • High Scalability: Capable of scaling to trillion-parameter models on limited hardware via expert parallelism and Megatron-bridge.

Related

  • Project
    verl-project/verl-recipe

    A collection of reproducible “recipes” (example projects) for the VeRL library, which provides reinforcement‑learning tools for large language models. Each recipe pins a specific VeRL version via a `REQUIRED_VERL.txt` file, and a helper script (`install_verl.sh`) can automatically install the matching library and run the example. The repo includes demos such as RL‑based tool use, LangGraph math agents, Single‑stream Policy Optimization, and more.

  • Project
    alibaba/ROLL

    An efficient scaling library for reinforcement learning with LLMs that optimizes training and inference across large GPU clusters using a distributed Ray-based architecture.

  • Project
    opendilab/LightRFT

    LightRFT is an open‑source, distributed reinforcement‑learning framework for fine‑tuning large language and multimodal generative models. It unifies training back‑ends (DeepSpeed, FSDP) with rollout engines (SGLang, vLLM), supports a range of RL algorithms (GRPO, REINFORCE++, DAPO, etc.), and works across text, image, video, and audio modalities. The library provides ready‑to‑run examples (e.g., GSM8K with Qwen2.5), extensive docs, and Docker images, making it suitable for researchers and engineers building RL‑based alignment pipelines.

  • Project
    areal-project/AReaL

    AReaL is an open‑source, fully asynchronous reinforcement‑learning framework for training large reasoning and agentic models. It offers modular micro‑services, supports many RL algorithms, runs on GPUs, NPUs or Ray clusters, and ships with dozens of ready‑to‑run examples (math, coding, search, vision‑language, RLHF, etc.).

  • Project
    Tencent-Hunyuan/UniRL

    UniRL is an open‑source Python framework that applies a unified reinforcement‑learning loop to many multimodal generative models (LLMs, vision‑language models, image/video diffusion, and hybrid AR‑diffusion models). It provides four entry points, Hydra‑based configs, pluggable rollout engines, and distributed training via Ray and FSDP. The repo ships standard RL algorithms plus three novel team‑proposed methods (Flow‑DPPO, DRPO, CPPO) with tutorials. Supported models include Stable Diffusion 3, FLUX.2‑Klein, Qwen‑3, HunyuanVideo, and more. Documentation, example recipes, and a WeChat community are provided. Licensed under Apache‑2.0.