Tencent-Hunyuan/UniRL

UniRL is a Framework for Unified Multimodal Model Reinforcement Learning

What it solves

UniRL provides a unified reinforcement learning (RL) post-training framework that works across different types of multimodal models. Instead of having separate pipelines for different architectures, it allows developers to use a single RL loop—generating samples, scoring them, computing advantages, and updating policies—for image diffusion, video diffusion, autoregressive (AR) vision-language models, and LLMs.

How it works

UniRL is built as a layered, composable system. It uses specific entrypoints (like train_diffusion or train_ar) that load configuration files to set up a trainer. This trainer then coordinates several pluggable components:

  • Rollout Engines: Handle the generation of samples.
  • Algorithms: Implement the RL logic (e.g., GRPO, Flow-DPPO, DRPO).
  • Model Bundles: Support a wide range of models including Stable Diffusion, Qwen, and HunyuanVideo.
  • Reward Services: Provide the scores used to guide the model's learning.
  • Distributed Runtime: Manages hardware efficiency using Ray DevicePool, FSDP, and LoRA/full-weight synchronization.

For agentic workflows, the framework extends the AR path to support multi-turn tool interactions, treating each turn as part of a sample lineage and scoring the final answer.

Who it’s for

It is designed for researchers and developers working on post-training multimodal models who need a scalable, flexible framework to apply RL algorithms to diverse model architectures (text, image, video, and omni-modality).

Highlights

  • Unified Loop: One RL post-training process applicable to multiple model families.
  • Broad Model Support: Compatible with image/video diffusion (SD3, WAN, HunyuanVideo), AR models (Qwen-VL, Qwen3), and unified models (HunyuanImage3).
  • Custom RL Algorithms: Includes team-proposed algorithms like Flow-DPPO for flow matching, DRPO for LLM RL, and CPPO for position-weighted trust regions.
  • Agentic Capabilities: Supports multi-turn tool use and service-scored trajectories.
  • Distributed Infrastructure: Integrated with Ray and FSDP for high-performance training.

Related

  • Project
    X-GenGroup/Flow-Factory

    A unified framework for applying reinforcement learning to fine-tune diffusion and flow-matching models across image, video, and audio-video tasks.

  • Project
    verl-project/verl-omni

    A general RL training framework designed for multimodal generative models, providing fast and stable post-training for diffusion and omni-modality models.

  • Project
    hiyouga/EasyR1

    An efficient, scalable reinforcement learning training framework that supports multi-modality models and various RL algorithms like GRPO.

  • Dispatch
    VeRL-Omni: RL Training Framework for Diffusion and Omni-Modality Models

    vLLM has announced the pre-release of VeRL-Omni, a general reinforcement learning post-training framework designed specifically for multimodal generative models, including diffusion and omni-modality architectures.

  • Project
    unilabsim/UniLab

    UniLab is a heterogeneous architecture for robot reinforcement learning that decouples CPU-parallel simulation from GPU-accelerated policy training via shared memory.