areal-project/AReaL

The RL Bridge for LLM-based Agent Applications. Made Simple & Flexible.

What it solves

AReaL is an infrastructure designed to bridge the gap between foundation model training and the development of complex, agentic AI applications. It addresses the need for efficient, scalable, and cost-effective training methods for reasoning and agent-based models by utilizing a fully asynchronous reinforcement learning (RL) paradigm.

How it works

AReaL operates as a microservice-based architecture that decouples key components of the training pipeline: training, inference, agent execution, and weight updates. By using an asynchronous RL approach, the system allows for faster training speeds compared to traditional synchronous methods. It supports a wide variety of RL algorithms (such as GRPO, PPO, and RLOO) and integrates with various agent frameworks, allowing developers to train agents by simply connecting their existing agentic runtimes to the AReaL service via a base URL.

Who it’s for

This project is intended for AI researchers and engineers who need to train large-scale reasoning models or complex AI agents. It is particularly useful for those who want to experiment with advanced RL algorithms or integrate RL training into existing agentic workflows without needing to rewrite their entire codebase.

Highlights

  • Fully Asynchronous RL: Optimized for industry-leading speed and stability in training large-scale models.
  • Microservice Architecture: Decouples training, inference, and agent services for modularity and scalability.
  • Broad Algorithm Support: Includes native implementations of state-of-the-art algorithms like GRPO, PPO, GSPO, and various token-masking techniques like KPop and IcePop.
  • Agentic Flexibility: Enables "black-box" agent training, allowing developers to plug in custom agentic runtimes with minimal configuration.
  • Hardware Versatility: Supports training across diverse hardware, including NVIDIA GPUs and Huawei Ascend NPUs.

Related

  • Project
    PrimeIntellect-ai/prime-rl

    prime-rl is a framework for large‑scale reinforcement learning, designed for asynchronous RL training at scale. It enables training models on up to 1000+ GPUs with features like FSDP2, vLLM, and support for agentic and multimodal tasks.

  • Project
    RLinf/RLinf

    A scalable reinforcement learning infrastructure designed for Embodied and Agentic AI, providing a unified backbone for training VLA models and deploying them across simulators and real-world robots.

  • Project
    ServiceNow/PipelineRL

    A scalable asynchronous reinforcement learning framework for LLM agents that uses in-flight weight updates to maximize GPU utilization while maintaining on-policy data freshness.

  • Project
    PaddlePaddle/PARL

    A flexible and high-efficiency reinforcement learning framework that supports reproducible algorithms and large-scale distributed training across CPU and GPU clusters.

  • Project
    microsoft/agent-lightning

    A lightweight agentic RL framework that allows developers to train AI agents with reinforcement learning using their existing harnesses and environments with minimal code changes.