areal-project/AReaL

The RL Bridge for LLM-based Agent Applications. Made Simple & Flexible.

AReaL – Large‑Scale Asynchronous Reinforcement‑Learning System

What it is AReaL is an open‑source framework for training reinforcement‑learning (RL) agents at scale. It was created by researchers at Tsinghua University and Ant Group and focuses on fully asynchronous training, which lets many workers generate experience and update model weights without waiting for each other. This design gives higher throughput and lower cost, especially when training large reasoning or agentic models.

Core strengths

Feature Why it matters
Flexibility You can plug in any black‑box agent (e.g., OpenAI Agents, CAMEL‑AI, custom APIs) simply by changing a base_url.
Scalability Asynchronous pipelines run on multiple GPUs or a Ray cluster, supporting Megatron, PyTorch FSDP, and Huawei Ascend NPU back‑ends.
Performance Reported speed‑ups of up to 2.8× over synchronous baselines and state‑of‑the‑art results on math, coding, search, and customer‑service tasks.
Modular services (AReaL 2.0) Training, inference, agent, and weight‑update are separate micro‑services, making deployment on clouds (GCP, AWS, Kubernetes via SkyPilot) straightforward.

Key algorithms (all have async and sync modes): GRPO, GSPO, PPO, DAPO, LitePPO, Dr‑GRPO, REINFORCE++, RLOO, SAPO, IcePop, KPop, M2PO, DPO, RLHF reward modeling, SFT, distillation, etc.

Typical workloads

  • Math & reasoning – GSM8K, multi‑turn math, countdown games, LoRA‑efficient training.
  • Agentic RL – General agents, online RL loops (Hermes), coding agents (SWE), search agents, tool‑integrated reasoning, OpenAI Agents, CAMEL‑AI.
  • Vision‑language – Geometry3K, CLEVR‑Count with Qwen‑VL models, NPU‑accelerated VLM training.
  • Alignment – Reward modeling for RLHF, distillation pipelines.
  • Cloud / edge deployment – SkyPilot integration, NPU support.

Getting started (from the README)

# clone and install
git clone https://github.com/areal-project/AReaL && cd AReaL
pip install uv
# install flash‑attention wheel (choose matching wheel)
uv pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.9-cp312-cp312-linux_x86_64.whl"
uv sync --extra cuda   # adds training packages + SGLang (default inference backend)
# single‑node run (GSM8K math example)
python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local

For a Ray cluster, add cluster.n_nodes, cluster.n_gpus_per_node, and a shared storage path, then set scheduler.type=ray.

Documentation & community

  • Full docs: https://areal-project.github.io/AReaL/
  • Chinese docs, WeChat group, and community meeting recordings are linked in the repo.
  • Model & data collections on Hugging Face (🤗 Models & Data).
  • Recent releases include AReaL 2.0 (micro‑service architecture) and AReaL‑lite (lightweight version).

Who should use it Researchers and engineers building large‑scale RL agents—especially those needing asynchronous pipelines, multi‑GPU/NPU scaling, or integration with existing LLM back‑ends—will find AReaL a ready‑to‑run platform with many example recipes.

Related

  • Project
    RLinf/RLinf

    RLinf is an open‑source, PyTorch‑compatible reinforcement‑learning infrastructure for large‑scale embodied and agentic AI. It ties together many simulators and real‑world robot platforms, supports a wide range of accelerators, and provides ready‑made pipelines for fine‑tuning huge vision‑language, diffusion, and MoE models with RL algorithms such as PPO, GRPO, SAC, and newer flow‑transformation methods. The library includes extensive system optimizations (e.g., FUSCO) and dozens of example recipes, making it suitable for researchers and engineers who need a unified, scalable backbone for training and deploying RL agents in both simulation and the real world.

  • Project
    ServiceNow/PipelineRL

    A scalable asynchronous reinforcement learning framework for LLM agents that uses in-flight weight updates to maximize GPU utilization while maintaining on-policy data freshness.

  • Project
    XYZ-AI-Lab/axrl

    An agentic RL post-training framework that coordinates high-throughput rollouts via SGLang and large-scale training via Megatron for multi-turn agent workflows.

  • Project
    alibaba/ROLL

    An efficient scaling library for reinforcement learning with LLMs that optimizes training and inference across large GPU clusters using a distributed Ray-based architecture.

  • Project
    OpenManus/OpenManus-RL

    An open-source framework for optimizing LLM agents using reinforcement learning to enhance their reasoning, tool use, and decision-making capabilities.