verl-project/verl-omni

Multimodal RL training framework for diffusion & omni models

VeRL‑Omni

VeRL‑Omni is an open‑source reinforcement‑learning (RL) training framework that focuses on multimodal generative models such as diffusion image/video/audio generators and omni‑modality text‑image‑audio‑video models. It sits on top of the verl library and integrates tightly with the vLLM‑Omni inference backend to provide fast, stable, and scalable RL post‑training for these large models.

What it does

  • RL‑based fine‑tuning of diffusion and omni‑modality generators (e.g., Qwen‑Image, SD3.5, Qwen‑3‑Omni, BAGEL, Wan2.2, etc.).
  • Supports a variety of RL algorithms tailored for generative models: FlowGRPO, Flow‑DPPO, MixGRPO, Diffusion‑OPD, DiffusionNFT, DPO, GSPO, DAPO, and more.
  • Provides high‑throughput rollout via the vLLM‑Omni backend, with batching, embed‑caching, and routing optimisations.
  • Offers asynchronous, multi‑reward serving (HTTP scorer, HPSv3, GenRM‑OCR, UnifiedReward, etc.) to overlap rollout and reward computation.
  • Works with distributed training back‑ends (VeOmni, FSDP2) and multiple parallelism strategies (USP/TP/DP).
  • Includes stability tricks such as rollout‑correction to skip costly log‑prob recomputation and deterministic RL pipelines.
  • Ascend NPU support (A2, A3, A5) for hardware‑accelerated training.

Who might use it

  • Researchers fine‑tuning diffusion models for higher image/video/audio quality using RL.
  • Engineers building commercial multimodal generation services that need fast, stable RL pipelines.
  • Teams that want to experiment with on‑policy distillation (DiffusionOPD), multi‑teacher RL, or reward‑model alignment for omni‑modality models.

Quick start (high‑level)

  1. Install – follow the docs (pip install verl-omni or build from source).
  2. Pick a recipe – e.g., the FlowGRPO recipe for Qwen‑Image is in examples/flowgrpo_trainer/.
  3. Configure – edit the provided Hydra YAML files to set model, dataset, reward endpoints, and parallelism.
  4. Run – launch the trainer (python -m verl_omni.train ...).
  5. Monitor – logs are compatible with Weights & Biases, TensorBoard, etc.

Documentation & Resources

License

Apache 2.0 – permissive open‑source license.

Citation

If you use VeRL‑Omni in research, cite the provided BibTeX entry.

Bottom line: VeRL‑Omni is a production‑grade, extensible RL training stack for the newest generation of multimodal and omni‑modality generative models, offering speed, stability, and hardware flexibility out of the box.

Related

  • Dispatch
    VeRL-Omni: RL Training Framework for Diffusion and Omni-Modality Models

    vLLM has announced the pre-release of VeRL-Omni, a general reinforcement learning post-training framework designed specifically for multimodal generative models, including diffusion and omni-modality architectures.

  • Dispatch
    VeRL-Omni v0.2.0 release notes / what's new

    VeRL-Omni v0.2.0 introduces request-level batching for faster diffusion RL and a reusable omni training stack for stable multimodal autoregressive training.

  • Project
    ByteDance-Seed/VeOmni

    A versatile framework for scaling the pre-training and post-training of single- and multi-modal models across various hardware accelerators.

  • Project
    vllm-project/vllm-omni

    vLLM‑Omni is an open‑source extension of the vLLM inference engine that adds fast, scalable serving for multimodal models (text, image, audio, video, and action). It supports both autoregressive and diffusion‑style architectures, provides a full‑duplex realtime API, works across GPUs, NPUs, and other accelerators, and ships ready‑to‑run recipes for many popular Hugging Face models. The project is production‑grade, Apache‑2.0 licensed, and backed by documentation, a Slack community, and recent technical reports.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.