Tencent-Hunyuan/UniRL

UniRL is a Framework for Unified Multimodal Model Reinforcement Learning

UniRL – Reinforcement Learning for Unified Multimodal Models

What it is – UniRL is an open‑source Python framework that lets researchers and engineers apply a single reinforcement‑learning (RL) loop to a wide variety of multimodal generative models (text‑to‑image, text‑to‑video, vision‑language, large language models, and “unified” models that combine autoregressive and diffusion components). The loop is the classic RL post‑training cycle: generate samples, score them with a reward service, compute advantages, update the policy, and sync the new weights back to the rollout workers.

Key ideas

  • Layered, composable architecture – entry‑points are driven by Hydra configs and instantiate a trainer that wires together:
    • Rollout engines (vLLM‑Omni, SGLang, etc.)
    • RL algorithms (both standard ones like GRPO and team‑proposed ones such as Flow‑DPPO, DRPO, CPPO)
    • Model bundles (diffusion, AR, unified models)
    • Reward services (any external scorer)
    • Distributed runtime (Ray DevicePool, FSDP, LoRA/full‑weight sync)
  • Multiple training modes – diffusion RL, autoregressive (AR) RL, supervised fine‑tuning (SFT), prompt‑enhancement, unified RL, agentic RL, and async variants that separate rollout and training workers.
  • Broad model support – ready‑to‑run recipes for dozens of state‑of‑the‑art models (Stable Diffusion 3.5, FLUX.2‑Klein, Qwen‑Image, various video diffusion models, Qwen‑3 series LLMs/VLMs, HunyuanVideo, etc.). Each recipe is a YAML config that selects a model, algorithm, rollout backend, and any special flags.
  • Team‑proposed algorithms – novel RL methods introduced by the UniRL team:
    • Flow‑DPPO – divergence‑based trust‑region for diffusion/flow models.
    • DRPO – smooth advantage‑weighted quadratic regularizer for token‑level LLM RL.
    • CPPO – position‑weighted cumulative‑prefix trust‑region mask. All come with step‑by‑step tutorials and runnable examples.

How to use it

  1. Install (see INSTALL.md).
  2. Pick a recipe, e.g. diffusion/sd3/sd3_trainside.yaml.
  3. Run the entry‑point, e.g. python -m unirl.train_diffusion --config-name=diffusion/sd3/sd3_trainside.
  4. The framework launches rollout workers, generates samples, scores them, and updates the model automatically. Multi‑node and GPU‑slab configurations are supported.

Who it’s for – researchers building RL‑based fine‑tuning pipelines for generative AI, product teams that need a unified way to improve multimodal models with reward signals, and anyone wanting to experiment with the new algorithms (Flow‑DPPO, DRPO, CPPO) on top of popular diffusion or LLM back‑ends.

Documentation & community – full docs at https://unirl‑project.github.io/unirl/, a WeChat group for Chinese‑language discussion, and a roadmap that lists upcoming model/algorithm extensions.

License – Apache‑2.0.

Related

  • Project
    X-GenGroup/Flow-Factory

    Flow‑Factory is an open‑source Python framework that unifies supervised fine‑tuning and reinforcement‑learning style optimization (GRPO, DPO, DPPO, etc.) for diffusion and flow‑matching generative models across image, video, and audio‑video tasks. It supports many state‑of‑the‑art models (Stable‑Diffusion 3.5, FLUX 1/2, Qwen‑Image, Wan2.x, MiniMax H3, etc.), offers a flexible YAML‑driven training pipeline, integrates with popular experiment trackers, and provides detailed guidance for adding new models or datasets.

  • Dispatch
    VeRL-Omni: RL Training Framework for Diffusion and Omni-Modality Models

    vLLM has announced the pre-release of VeRL-Omni, a general reinforcement learning post-training framework designed specifically for multimodal generative models, including diffusion and omni-modality architectures.

  • Project
    alibaba/ROLL

    ROLL is an open‑source library for reinforcement‑learning training of large language models at scale. It provides Ray‑based distributed execution, supports many RL algorithms (PPO, GRPO, Reinforce++, etc.), works with Megatron‑Core, vLLM, SGLang, and LoRA, and offers pipelines for RL‑based fine‑tuning (RLVR) and multi‑turn agentic RL. The project includes extensive docs, hardware‑agnostic support (NVIDIA, AMD, Ascend), and built‑in observability.

  • Project
    opendilab/LightRFT

    LightRFT is an open‑source, distributed reinforcement‑learning framework for fine‑tuning large language and multimodal generative models. It unifies training back‑ends (DeepSpeed, FSDP) with rollout engines (SGLang, vLLM), supports a range of RL algorithms (GRPO, REINFORCE++, DAPO, etc.), and works across text, image, video, and audio modalities. The library provides ready‑to‑run examples (e.g., GSM8K with Qwen2.5), extensive docs, and Docker images, making it suitable for researchers and engineers building RL‑based alignment pipelines.

  • Project
    hiyouga/EasyR1

    EasyR1 is an open‑source reinforcement‑learning‑from‑human‑feedback framework for training large language and vision‑language models. It supports many state‑of‑the‑art RL algorithms (GRPO, DAPO, GSPO, etc.), works with models like Llama 3 and Qwen 2‑VL, and scales from a single GPU to multi‑node clusters using Ray and FSDP. Features include padding‑free batches, LoRA adaptation, checkpoint merging, and integration with popular experiment trackers.