redai-studio/Relax

An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale

Relax – Asynchronous Reinforcement‑Learning Engine for Omni‑Modal LLMs

What it is – Relax (Reinforcement Engine Leveraging Agentic X‑modality) is an open‑source framework that lets you fine‑tune large multimodal language models (text + vision + audio) with reinforcement learning at scale. It is built on Ray Serve for a service‑oriented, fault‑tolerant deployment, uses Megatron‑LM as the training backend, and SGLang for high‑throughput inference. A custom TransferQueue decouples rollout (inference) from training, enabling fully asynchronous pipelines that can run on separate GPU clusters.

Core Highlights (from the README)

  • Omni‑modal RL – One framework handles text, images, video, and audio in a single training loop (e.g., Qwen3‑Omni).
  • Six‑layer service architecture – Every component (actor, rollout, critic, etc.) is a separate Ray Serve deployment, giving native elastic scheduling and automatic recovery.
  • Three execution modes –
    1. Colocate (Sync) – actor and rollout share GPUs (memory‑efficient).
    2. Fully Async – each role runs on its own GPU cluster, communicating via TransferQueue for maximum throughput.
    3. Hybrid – rollout and actor are separate, but reference/advantage calculations stay on‑process, balancing speed and resource use.
  • Rich algorithm suite – PPO, GRPO, M2PO, RLOO, REINFORCE++ (and variants), GSPO, SAPO, CISPO, plus on‑policy distillation.
  • Elastic rollout scaling – HTTP REST API lets you add or remove inference engines mid‑training without stopping the loop, supporting both same‑cluster and cross‑cluster federation.
  • Production‑ready ops – Health manager with auto‑recovery, centralized metrics (WandB / TensorBoard / ClearML), and real‑time notifications via Apprise.
  • Model support – Designed for Megatron‑LM models such as Qwen‑3 series, GLM‑5, Kimi‑K2/K3, and custom HuggingFace checkpoints.
  • Reward hub – Built‑in reward functions for math, visual QA, multiple‑choice, and a “GenRM” LLM‑as‑judge mode; custom rewards are added by dropping a file into relax/engine/rewards/.

Typical Workflow (quick‑start style)

  1. Prepare data & model – Download a HuggingFace dataset and a Megatron‑compatible checkpoint (e.g., Qwen‑3‑4B).
  2. Launch the Docker image – The official image (ghcr.io/redai-studio/relaxrl:latest) contains CUDA, PyTorch, Megatron‑LM, SGLang, and Ray pre‑installed.
  3. Run a script – One‑line bash scripts launch a full RL job in any of the three modes. Example for text‑only PPO on 8 GPUs:
    export EXP_DIR=/root
bash scripts/training/text/run-qwen3-4B-8xgpu.sh
  4. Monitor – Logs appear in the console; metrics are streamed to the configured service (WandB/TensorBoard).
  5. Checkpoint & export – Checkpoints are saved in Megatron DCP format; a conversion script turns them into HuggingFace weights for downstream use.

Installation

# Pull the pre‑built container (includes all dependencies)
docker pull ghcr.io/redai-studio/relaxrl:latest

# Run it with GPU access and mount your workspace
docker run -it --gpus all --ipc=host --network=host \
  -v /path/to/workspace:/root ghcr.io/redai-studio/relaxrl:latest bash

# Inside the container, clone the repo and install in editable mode
git clone https://github.com/redai-studio/Relax.git /root/Relax
cd /root/Relax && pip install -e .

The README also provides a detailed installation guide for multi‑node clusters and driver requirements.

Who would use Relax?

  • Research labs building next‑generation multimodal LLMs that need RL‑based alignment (e.g., visual‑question‑answering, audio‑visual agents).
  • Industrial AI teams that want a production‑grade, fault‑tolerant RL pipeline capable of scaling across many GPU nodes.
  • Developers looking to experiment with advanced RL algorithms (M2PO, RLOO, REINFORCE++) on large MoE models without writing low‑level distributed code.

Where to learn more

  • Docs site – https://redai-studio.github.io/Relax (bilingual, full architecture, training guides, API reference).
  • Paper – arXiv link 2604.11554 (linked in the badge).
  • Discussion channels – WeChat QR badge, Docker image badge, and GitHub Discussions for community help.

Bottom line

Relax is a full‑stack, asynchronous RL platform that bridges the gap between massive multimodal LLMs and reinforcement‑learning‑based fine‑tuning. Its service‑oriented design, support for a wide algorithm set, and elastic rollout capabilities make it one of the most complete open‑source solutions for omni‑modal RL at scale.

Related

  • Project
    alibaba/ROLL

    ROLL is an open‑source library for reinforcement‑learning training of large language models at scale. It provides Ray‑based distributed execution, supports many RL algorithms (PPO, GRPO, Reinforce++, etc.), works with Megatron‑Core, vLLM, SGLang, and LoRA, and offers pipelines for RL‑based fine‑tuning (RLVR) and multi‑turn agentic RL. The project includes extensive docs, hardware‑agnostic support (NVIDIA, AMD, Ascend), and built‑in observability.

  • Project
    opendilab/LightRFT

    LightRFT is an open‑source, distributed reinforcement‑learning framework for fine‑tuning large language and multimodal generative models. It unifies training back‑ends (DeepSpeed, FSDP) with rollout engines (SGLang, vLLM), supports a range of RL algorithms (GRPO, REINFORCE++, DAPO, etc.), and works across text, image, video, and audio modalities. The library provides ready‑to‑run examples (e.g., GSM8K with Qwen2.5), extensive docs, and Docker images, making it suitable for researchers and engineers building RL‑based alignment pipelines.

  • Project
    hiyouga/EasyR1

    EasyR1 is an open‑source reinforcement‑learning‑from‑human‑feedback framework for training large language and vision‑language models. It supports many state‑of‑the‑art RL algorithms (GRPO, DAPO, GSPO, etc.), works with models like Llama 3 and Qwen 2‑VL, and scales from a single GPU to multi‑node clusters using Ray and FSDP. Features include padding‑free batches, LoRA adaptation, checkpoint merging, and integration with popular experiment trackers.

  • Project
    X-GenGroup/Flow-Factory

    Flow-Factory is a unified framework for fine-tuning and aligning diffusion and flow-matching models using advanced reinforcement learning and supervised techniques.

  • Dispatch
    Qwen 3.5‑397B‑A17B release: hybrid linear‑attention MoE model with 1 M token context and state‑of‑the‑art multimodal performance

    Qwen 3.5‑397B‑A17B is a 397 billion‑parameter multimodal model that activates only 17 billion parameters per token, delivering state‑of‑the‑art performance on language, coding, reasoning and vision tasks while being up to 19× faster than its predecessor.