verl-project/verl-omni

Multimodal RL training framework for diffusion & omni models

What it solves

VeRL-Omni provides a stable and high-performance framework for reinforcement learning (RL) post-training of multimodal generative models. It addresses the specific challenges of multimodal RL—such as complex I/O patterns, compute bottlenecks, and the need for efficient rollout generation—that differ significantly from text-only LLM training.

How it works

The framework is built on top of verl and integrates several optimizations to accelerate the RL pipeline:

  • Fast Rollouts: Uses the vLLM-Omni backend to optimize generation through rollout routing, batching, and embed caching.
  • Asynchronous Reward Serving: Supports multiple reward models (e.g., HPSv3, GenRM-OCR) and asynchronous computation to overlap the reward phase with the rollout phase.
  • Modular Backends: Offers selectable training backends like VeOmni and FSDP2 with combinable parallelism (USP/TP/DP) for distributed training.
  • Stability Mechanisms: Implements rollout correction to skip logP recomputation, ensuring faster and more stable diffusion RL pipelines.

Who it’s for

Researchers and developers working on the post-training alignment of diffusion models (image, video, audio) and omni-modality models that jointly handle text, image, audio, and video.

Highlights

  • Broad Model Support: Compatible with Qwen-Image, SD3.5, Wan2.2, LTX2.3, and Qwen3-Omni.
  • Diverse Algorithm Support: Implements various RL algorithms including FlowGRPO, DiffusionNFT, Diffusion DPO, and GSPO.
  • High Throughput: Achieves approximately 25% higher end-to-end throughput compared to diffusers-based implementations on reference setups.
  • Hardware Support: Supports both NVIDIA GPUs and Ascend NPUs.

Related

  • Dispatch
    VeRL-Omni: RL Training Framework for Diffusion and Omni-Modality Models

    vLLM has announced the pre-release of VeRL-Omni, a general reinforcement learning post-training framework designed specifically for multimodal generative models, including diffusion and omni-modality architectures.

  • Dispatch
    VeRL-Omni v0.2.0 release notes / what's new

    VeRL-Omni v0.2.0 introduces request-level batching for faster diffusion RL and a reusable omni training stack for stable multimodal autoregressive training.

  • Project
    Tencent-Hunyuan/UniRL

    UniRL is an open‑source Python framework that applies a unified reinforcement‑learning loop to many multimodal generative models (LLMs, vision‑language models, image/video diffusion, and hybrid AR‑diffusion models). It provides four entry points, Hydra‑based configs, pluggable rollout engines, and distributed training via Ray and FSDP. The repo ships standard RL algorithms plus three novel team‑proposed methods (Flow‑DPPO, DRPO, CPPO) with tutorials. Supported models include Stable Diffusion 3, FLUX.2‑Klein, Qwen‑3, HunyuanVideo, and more. Documentation, example recipes, and a WeChat community are provided. Licensed under Apache‑2.0.

  • Project
    hiyouga/EasyR1

    An efficient and scalable reinforcement learning training framework that extends support to vision language models and multiple RL algorithms like GRPO.

  • Project
    vllm-project/vllm-omni

    A framework that extends vLLM to provide high-performance serving for omni-modality models, supporting text, image, audio, video, and action outputs across various hardware backends.