Tencent-Hunyuan/UniRL

What it solves

UniRL provides a unified reinforcement learning (RL) post-training framework that works across different types of multimodal models. Instead of having separate pipelines for different architectures, it allows developers to use a single RL loop—generating samples, scoring them, computing advantages, and updating policies—for image diffusion, video diffusion, autoregressive (AR) vision-language models, and LLMs.

How it works

UniRL is built as a layered, composable system. It uses specific entrypoints (like train_diffusion or train_ar) that load configuration files to set up a trainer. This trainer then coordinates several pluggable components:

• Rollout Engines: Handle the generation of samples.
• Algorithms: Implement the RL logic (e.g., GRPO, Flow-DPPO, DRPO).
• Model Bundles: Support a wide range of models including Stable Diffusion, Qwen, and HunyuanVideo.
• Reward Services: Provide the scores used to guide the model's learning.
• Distributed Runtime: Manages hardware efficiency using Ray DevicePool, FSDP, and LoRA/full-weight synchronization.

For agentic workflows, the framework extends the AR path to support multi-turn tool interactions, treating each turn as part of a sample lineage and scoring the final answer.

Who it’s for

It is designed for researchers and developers working on post-training multimodal models who need a scalable, flexible framework to apply RL algorithms to diverse model architectures (text, image, video, and omni-modality).

Highlights

• Unified Loop: One RL post-training process applicable to multiple model families.
• Broad Model Support: Compatible with image/video diffusion (SD3, WAN, HunyuanVideo), AR models (Qwen-VL, Qwen3), and unified models (HunyuanImage3).
• Custom RL Algorithms: Includes team-proposed algorithms like Flow-DPPO for flow matching, DRPO for LLM RL, and CPPO for position-weighted trust regions.
• Agentic Capabilities: Supports multi-turn tool use and service-scored trajectories.
• Distributed Infrastructure: Integrated with Ray and FSDP for high-performance training.