areal-project/AReaL
The RL Bridge for LLM-based Agent Applications. Made Simple & Flexible.
What it solves
AReaL 是一個強化學習（RL）基礎設施，旨在彌合訓練基礎模型與將其部署為代理式應用之間的鴻溝。它解決了訓練大規模推理與代理模型的低效率與複雜性，使開發者能更容易且具成本效益地完成此過程。
How it works
AReaL 採用完全非同步的 RL 訓練範式，以優化速度與可擴展性。在 2.0 版中，它轉換為微服務架構，將系統分割為訓練、推論、代理執行與權重更新等獨立服務。它支援多種 RL 演算法（如 GRPO、PPO 與 DPO），並與各種訓練後端（如 Megatron 與 PyTorch FSDP）整合，以在多節點與多 GPU 上處理大規模模型訓練。
Who it’s for
此系統設計給 AI 研究人員與開發者，用於構建大規模推理模型、程式碼代理、搜尋代理與視覺語言模型（VLM），需要高效能、可擴展的 RL 訓練框架。
Highlights
- Asynchronous Training: Achieves industry-leading speed and stability in RL training, significantly reducing training time compared to synchronous systems.
- Microservice Architecture: Decouples training, inference, and agent services for better modularity and and scalability.
- Broad Algorithm Support: Implements a variety of cutting-edge RL algorithms including GRPO, GSPO, PPO, and DAPO.
- Agentic RL Flexibility: Allows seamless customization for black-box agent applications by simply replacing the
base_url. - Hardware Support: Supports training on NVIDIA GPUs and Huawei Ascend NPUs.
- Lightweight Version: Offers AReaL-lite for rapid prototyping and algorithm-first development.