redai-infra/Relax

An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale

What it solves

Relax is a high-performance reinforcement learning (RL) post-training framework designed for omni-modal large language models. It addresses the challenge of scaling RL training across different modalities (text, vision, and audio) while eliminating the GPU idle time typically caused by the sequential nature of rollout (inference) and training phases.

How it works

Relax uses a service-oriented architecture built on Ray Serve, where different roles (Actor, Rollout, Critic, etc.) are deployed as independent services. It employs a system called TransferQueue to decouple training and inference, allowing them to run on separate GPU clusters in parallel. This asynchronous execution is supported by three modes:

  • Colocate (Sync): Actor and Rollout share GPUs and time-share resources.
  • Fully Async: Roles run on independent clusters with streaming data exchange and asynchronous weight synchronization via the Distributed Checkpoint Service (DCS).
  • Hybrid: Actor and Rollout use separate placement groups, but other components like the reference model run in-process on the actor's GPUs.

For the backend, it integrates Megatron-LM for training (supporting TP/PP/CP/EP parallelism) and SGLang for high-throughput inference.

Who it’s for

It is intended for AI researchers and engineers working on post-training multimodal models, specifically those needing to scale RL training for models like the Qwen3 series or GLM5, and those implementing complex agentic RL workflows.

Highlights

  • Omni-Modal Support: Unified framework for text, vision, and audio RL, supporting end-to-end training for models like Qwen3-Omni.
  • Agentic RL: Supports multi-turn interaction, loss masking, and multimodal context carry-over for closed-loop training.
  • Elastic Rollout Scaling: Ability to dynamically add or remove inference engines via HTTP REST API mid-training.
  • Rich Algorithm Suite: Out-of-the-box support for GRPO, GSPO, SAPO, CISPO, and On-Policy Distillation.
  • Production-Ready Ops: Includes a HealthManager for auto-recovery and a centralized Metrics Service for WandB, TensorBoard, and ClearML.