redai-studio/Relax

An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale

Relax – Asynchronous Omni‑Modal Reinforcement‑Learning Engine

What it is – Relax (Reinforcement Engine Leveraging Agentic X‑modality) is an open‑source framework for post‑training reinforcement learning on large multimodal language models. It lets you fine‑tune models such as Qwen‑3‑Omni, Qwen‑3‑VL, GLM‑5, etc., with a full suite of on‑policy algorithms (PPO, GRPO, M2‑PO, RLOO, REINFORCE++, …) while handling text, images, video, and audio in a single pipeline.

Why it matters – Training RL for LLMs is usually bottlenecked by the rollout (inference) stage. Relax decouples rollout from the training step using a custom TransferQueue and runs each component as an independent Ray Serve service. This makes the system:

  • Fully asynchronous – rollout, actor, reference, advantage computation, and reward models can live on separate GPU clusters, keeping every GPU busy.
  • Elastic – you can add or remove inference engines on the fly via a REST API, which is useful for cloud‑bursting or federated clusters.
  • Omni‑modal – the same code path supports pure‑text, vision‑language, and audio‑visual tasks, enabling end‑to‑end RL on models like Qwen‑3‑Omni.
  • Production‑ready – health‑checks, automatic recovery, centralized metrics (WandB / TensorBoard / ClearML), and real‑time notifications are built in.

Core components

Layer Role
Entrypoints train.py – CLI, Ray cluster connection, controller launch
Orchestration Controller, Service, Registry – schedule loops, manage placement groups
Components Ray Serve deployments: Actor, Rollout, Critic, ActorFwd, Advantages, GenRM
Engine SGLang‑based rollout engine, reward routing, data filtering
Backends Megatron‑LM (TP/PP/CP/EP) for training, SGLang for high‑throughput inference
Distributed Ray actor groups + Distributed Checkpoint Service (DCS) for NCCL/GLOO weight sync

Three execution modes are offered:

  • Colocate (Sync) – actor and rollout share GPUs (memory‑efficient, strict on‑policy).
  • Fully Async – each role runs on its own GPU cluster, communicating via TransferQueue.
  • Hybrid – rollout and actor are separate, but reference/advantage steps stay on‑process to reduce overhead while still streaming data.

Algorithms & rewards

Relax ships with a rich algorithm suite (PPO, GRPO, M2‑PO, RLOO, REINFORCE++ variants, GSPO, SAPO, CISPO, on‑policy distillation) and a plug‑and‑play reward hub (math verification, GPQA, F1, IFBench, multimodal Open‑R1, and a built‑in LLM‑as‑judge called GenRM). Adding a new reward is as simple as dropping a Python file into relax/engine/rewards/.

Getting started (Docker‑first)

# Pull the official image (includes CUDA, PyTorch, Megatron‑LM, SGLang, Ray)
docker pull ghcr.io/redai-studio/relaxrl:latest

# Run a container with GPU access and mount your workspace
docker run -it --gpus all --ipc=host --network=host \
  -v /path/to/workspace:/root ghcr.io/redai-studio/relaxrl:latest bash

# Inside the container
git clone https://github.com/redai-studio/Relax.git /root/Relax
cd /root/Relax && pip install -e .

Quick‑start recipes (one‑liner scripts)

  1. Text‑only math RL – 8‑GPU PPO on Qwen‑3‑4B using the dapo-math-17k dataset.
  2. Vision‑language – 8‑GPU GRPO on Qwen‑3‑VL‑4B with the multimodal-open-r1-8k-verified dataset.
  3. Omni‑modal (image + audio) – 16‑GPU (2‑node) GRPO on Qwen‑3‑Omni‑30B‑A3B using the AVQA-R1-6K dataset.

Each script handles dataset download (via hf download), model fetch, and launches train.py with the appropriate flags. Logs show rollout progress and training steps; checkpoints are saved in Megatron DCP format and can be converted to HuggingFace weights with the provided conversion script.

When to use Relax

  • Research on multimodal RL – you need a single code base that can train on text, images, video, and audio simultaneously.
  • Scaling RL to large models – the async architecture lets you keep dozens of GPUs busy, even across multiple clusters.
  • Production‑grade fine‑tuning – built‑in health management, metrics aggregation, and alerting make long‑running jobs more reliable.
  • Experimenting with new policies – the modular algorithm registry lets you drop in a custom loss without touching the core engine.

Where to learn more

  • Documentation site – https://redai-studio.github.io/Relax (bilingual, full guides on architecture, async training, hybrid mode, elastic rollout scaling, etc.)
  • Paper – arXiv:2604.11554 (provides the research background and performance numbers).
  • GitHub discussions – WeChat QR, Docker image, and community Q&A are linked from the README badges.

Bottom line – Relax is a production‑oriented, highly configurable RL platform that bridges the gap between massive multimodal LLMs and reinforcement‑learning fine‑tuning, offering asynchronous execution, elastic scaling, and a breadth of algorithms out of the box.

Related

  • Project
    opendilab/LightRFT

    LightRFT is an open‑source, distributed reinforcement‑learning framework for fine‑tuning large language and multimodal generative models. It unifies training back‑ends (DeepSpeed, FSDP) with rollout engines (SGLang, vLLM), supports a range of RL algorithms (GRPO, REINFORCE++, DAPO, etc.), and works across text, image, video, and audio modalities. The library provides ready‑to‑run examples (e.g., GSM8K with Qwen2.5), extensive docs, and Docker images, making it suitable for researchers and engineers building RL‑based alignment pipelines.

  • Project
    Tencent-Hunyuan/UniRL

    UniRL is an open‑source Python framework that applies a unified reinforcement‑learning loop to many multimodal generative models (LLMs, vision‑language models, image/video diffusion, and hybrid AR‑diffusion models). It provides four entry points, Hydra‑based configs, pluggable rollout engines, and distributed training via Ray and FSDP. The repo ships standard RL algorithms plus three novel team‑proposed methods (Flow‑DPPO, DRPO, CPPO) with tutorials. Supported models include Stable Diffusion 3, FLUX.2‑Klein, Qwen‑3, HunyuanVideo, and more. Documentation, example recipes, and a WeChat community are provided. Licensed under Apache‑2.0.

  • Dispatch
    Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

    Hugging Face surveyed 16 open-source RL libraries and found that async RL training separates inference and training onto different GPU pools, uses a rollout buffer, and pushes weights asynchronously, with Ray dominating orchestration and NCCL broadcast the common weight‑sync method.

  • Project
    verl-project/verl

    A flexible and efficient RL training library for LLMs that simplifies the implementation of RLHF algorithms and scales to trillion-parameter models.

  • Project
    NVIDIA-NeMo/RL

    NeMo RL is NVIDIA’s open‑source library for scalable reinforcement‑learning fine‑tuning of large language and vision‑language models. It supports a range of RL algorithms (GRPO, DAPO, GDPO, etc.), works with both PyTorch DTensor and Megatron Core backends, and offers high‑performance generation via vLLM or Megatron inference. Ready‑to‑run Docker containers, Ray‑based distributed training, and extensive YAML recipes make it suitable for quick prototyping up to multi‑node, multi‑GPU production runs.