verl-project/verl-omni

Multimodal RL training framework for diffusion & omni models

What it solves

VeRL-Omni addresses the unique challenges of Reinforcement Learning (RL) post-training for multimodal generative models. Unlike text-only LLMs, multimodal models have different I/O patterns, compute characteristics, and runtime bottlenecks that require specialized training frameworks to ensure stability, speed, and efficiency.

How it works

VeRL-Omni is a general RL training framework built on top of verl. It optimizes the training pipeline through several key technical approaches:

  • Fast Rollouts: It uses the vLLM-Omni backend to accelerate generation via rollout routing, batching, and embed caching.
  • Efficient Reward Serving: It supports asynchronous reward computation and HTTP scorers to overlap the reward phase with the rollout phase, reducing idle time.
  • Modular Backends: It provides selectable training backends (VeOmni and FSDP2) with combinable parallelism (USP/TP/DP) for distributed training.
  • Stability Enhancements: It implements rollout correction to skip logP recomputation, ensuring faster and more stable diffusion RL pipelines.
  • Specialized Algorithms: It integrates various RL algorithms tailored for multimodal tasks, such as FlowGRPO, Flow-DPPO, DiffusionNFT, and GSPO.

Who it’s for

This framework is designed for researchers and developers working on the post-training of:

  • Diffusion models (image, video, and audio generation).
  • Unified multimodal models (combining understanding and generation).
  • Omni-modality models (jointly handling text, image, audio, and video).

Highlights

  • High Throughput: Achieves ~25% higher end-to-end throughput compared to diffusers-based FlowGRPO implementations.
  • Broad Model Support: Compatible with models like Qwen-Image, SD3.5, Wan2.2, and Qwen3-Omni.
  • Hardware Flexibility: Supports both NVIDIA GPUs and Ascend NPUs.
  • Comprehensive Algorithm Suite: Supports a wide range of alignment techniques including DPO, GRPO variants, and specialized diffusion RL methods.