areal-project/AReaL
The RL Bridge for LLM-based Agent Applications. Made Simple & Flexible.
What it solves
AReaL 是一个强化学习（RL）基础设施，旨在弥合训练基础模型与将其部署为代理式应用之间的鸿沟。它解决了训练大规模推理和代理模型的低效率与复杂性，使开发者能够更容易且具成本效益地完成此过程。
How it works
AReaL 采用完全异步的 RL 训练范式，以优化速度和可扩展性。在 2.0 版中，它转变为微服务架构，将系统拆分为训练、推理、代理执行和权重更新等独立服务。它支持多种 RL 算法（如 GRPO、PPO 与 DPO），并与各种训练后端（如 Megatron 与 PyTorch FSDP）集成，以在多节点和多 GPU 上处理大规模模型训练。
Who it’s for
此系统面向构建大规模推理模型、代码代理、搜索代理和视觉语言模型（VLM）的 AI 研究人员和开发者，需高性能、可扩展的 RL 训练框架。
Highlights
- Asynchronous Training: Achieves industry-leading speed and stability in RL training, significantly reducing training time compared to synchronous systems.
- Microservice Architecture: Decouples training, inference, and agent services for better modularity and and scalability.
- Broad Algorithm Support: Implements a variety of cutting-edge RL algorithms including GRPO, GSPO, PPO, and DAPO.
- Agentic RL Flexibility: Allows seamless customization for black-box agent applications by simply replacing the
base_url. - Hardware Support: Supports training on NVIDIA GPUs and Huawei Ascend NPUs.
- Lightweight Version: Offers AReaL-lite for rapid prototyping and algorithm-first development.