VeRL-Omni: RL Training Framework for Diffusion and Omni-Modality Models

vLLM has announced the pre-release of VeRL-Omni, a general reinforcement learning (RL) post-training framework designed for multimodal generative models. Built upon verl and vllm-omni, VeRL-Omni addresses the unique challenges of RL training for non-autoregressive and omni-modal models, such as diffusion transformers and unified understanding and generation architectures.

Technical Challenges in Multimodal RL

VeRL-Omni is designed to solve three primary technical hurdles associated with multimodal generative RL:

• Architecture Extension: The framework extends RL capabilities to diffusion transformer backbones (e.g., Qwen-Image), mixed AR-DiT architectures (e.g., Qwen-Omni), and unified models (e.g., BAGEL, HunyuanImage3.0).
• Heterogeneous Rollout Pipelines: Unlike text-based RL, multimodal rollouts are denoising trajectories in continuous latent space. These pipelines often involve multiple components, such as text encoders, Diffusion Transformers (DiT), and Variational Autoencoders (VAE).
• Workload Scheduling: Multimodal RL requires orchestrating complex workflows where reward functions are often multimodal models themselves (such as VLM judges or OCR scorers), and generation rollouts typically exhibit higher memory peaks than text generation.

Key Framework Features

VeRL-Omni provides a modular and efficient stack for multimodal RL training:

• Efficient Multimodal Rollout: By integrating vLLM-Omni, the framework utilizes high-throughput asynchronous serving for multimodal generation, employing step-wise continuous batching and embedding caching to optimize efficiency.
• Flexible Reward Engine: The system supports both rule-based and model-based rewards (e.g., VLM-as-judge). vLLM is used for efficient reward model inference, and reward computation is overlapped with rollout and training processes to reduce latency.
• Modular Training Backends: The framework includes various trainers—such as DiffusersFSDP, Megatron, and VeOmni—with built-in optimizations for diffusion and omni-modal models, supporting parallelism strategies including FSDP, USP, and TP.
• Hardware Compatibility: VeRL-Omni supports both NVIDIA GPUs and Ascend NPUs.

Supported Models and Algorithms

VeRL-Omni supports a variety of architectures and RL algorithms, with several currently released or in development:

Model	Architecture	Modality	Algorithm	Status
Qwen-Image	DiT	Text → Image	FlowGRPO, MixGRPO, GRPO-Guard	Released
BAGEL	Unified understand + gen	Text + Image	FlowGRPO	PR ready
Qwen3-Omni-Thinker	AR	Text / Image / Video / Audio	GSPO	PR ready
Wan2.2	DiT	Text → Video	DanceGRPO	WIP
SD3.5	DiT	Text → Image	DPO	WIP
HunyuanImage-3.0	Unified understand + gen	Text + Image	MixGRPO, SRPO	Planned

FlowGRPO Implementation and Performance

VeRL-Omni implements FlowGRPO, an online policy method for flow-matching models. The training workflow consists of four stages: rollout generation (collecting trajectories and images), reward model scoring, policy optimization using a CLIP-style loss, and periodic weight synchronization from the trainer to rollout workers.

Performance Benchmarks

In tests using NVIDIA H800 GPUs for LoRA fine-tuning of Qwen-Image on an OCR reward task, the framework demonstrated the following throughput:

• Colocated Training: 0.305 images/GPU/s with a time per step of 420s.
• Async Reward: 0.280 images/GPU/s with a time per step of 360s. Moving the reward model to a dedicated GPU reduces wall-clock time per step by approximately 14%.

For full-model fine-tuning of Qwen-Image (non-CFG) on 4 × NVIDIA H200 GPUs, the framework achieved 0.510 images/GPU/s at approximately 250 s/step.

Future Roadmap

VeRL-Omni is currently in pre-release. The future development roadmap includes:

• Expanded Model Support: Adding more open-source diffusion and omni-modal models for image, video, and audio generation.
• Advanced Algorithms: Integrating new RL algorithms such as DiffusionNFT.
• Fully Asynchronous RL: Developing end-to-end async pipelines across actor, rollout, and reward to improve GPU/NPU utilization.
• vLLM-Omni Co-optimization: Further accelerating rollouts through advanced parallelism, quantization, and batching.
• Optimized Trainers: Releasing more trainer engines based on Megatron-core and VeOmni.
• Hardware Expansion: Hardening the Ascend NPU path and adding more hardware backends.