AgileRL/AgileRL

AgileRL – End‑to‑end post‑training for LLMs and classic RL

What it is – A Python library that lets you take a pre‑trained large language model (LLM) and continue training it with the whole spectrum of modern techniques: supervised fine‑tuning (SFT), preference learning (DPO), and several reinforcement‑learning (RL) algorithms (GRPO, CISPO, GSPO, PPO, REINFORCE). The same package also supports “classic” deep‑RL for single‑agent, multi‑agent, offline, and bandit problems, with built‑in evolutionary hyper‑parameter optimisation (HPO).

Why it matters – Most existing tools focus on either LLM fine‑tuning or classic RL, but not both, and they often require a lot of glue code to run at scale. AgileRL bundles the entire post‑training stack for LLMs and a full RL suite, offering:

• Unified YAML‑based experiment manifests that can be run locally or submitted to the managed Arena cloud service.
• Torch‑native distributed training (data‑parallel or FSDP2 sharding) with memory‑saving tricks (LoRA, QLoRA, activation checkpointing, CPU/off‑GPU optimizer offload).
• Fast inference rollouts via vLLM, optionally colocated with the trainer.
• Evolutionary population‑based HPO that tunes learning‑rate, KL penalty, etc., on‑the‑fly.
• A CLI and Python SDK that are deliberately “agent‑friendly,” meaning an AI coding assistant can generate a manifest, validate it, and launch a job without human interaction.

Key components

Component	Role
Algorithms	LLM: SFT, DPO, GRPO, CISPO, GSPO, PPO, REINFORCE. Classic RL: PPO, DQN, Rainbow DQN, DDPG, TD3, CQL, ILQL, MADDPG, MATD3, IPPO, NeuralUCB, NeuralTS.
Training back‑ends	Local GPU (torchrun) or remote Arena clusters (async, population‑based).
Model support	Any Hugging‑Face transformer, including dense, MoE, and hybrid Mamba families (e.g., Qwen, Granite, Gemma, Nemotron‑3.5 up to 120 B).
Adapters	LoRA and QLoRA for low‑memory fine‑tuning; per‑expert LoRA for MoE models.
Parallelism	Data‑parallel or Fully‑Sharded Data Parallel (FSDP2) with a single FSDPConfig that works across all LLM algorithms.
Rollout engine	vLLM for high‑throughput token generation; can be colocated on the same GPU as the trainer.
Hyper‑parameter optimisation	Evolutionary HPO runs a population of agents; hyper‑parameters evolve during training.
CLI / SDK	arena CLI for login, model listing, experiment submission, deployment, and generation. Python LocalTrainer and ArenaClient for programmatic use.

Typical workflow

1. Write a manifest – a single YAML file describing the base model, algorithm, environment (OpenEnv, Gymnasium, PettingZoo, or a custom remote server), and HPO settings.
2. Run locally – python -m agilerl.train my_manifest.yaml or use LocalTrainer.from_manifest(...).
3. Scale on Arena – arena experiments submit my_manifest.yaml --project my‑project. Arena handles async rollouts, population‑based HPO, and GPU provisioning.
4. Deploy – arena agent deploy <experiment> then arena agent generate --prompt "…" to chat with the fine‑tuned LLM.

Performance claim – The README cites a benchmark where AgileRL processes >4× more tokens per second than comparable libraries (ART, TRL) while achieving higher reward with half the GPU memory on a multi‑turn Sudoku task.

Getting started

pip install "agilerl[llm]"   # LLM post‑training extras
pip install agilerl           # classic RL + Arena CLI

Optional extras add Box2D physics, CPU‑only wheels, or the Arena SDK alone.

Documentation & community – Full docs at https://docs.agilerl.com, a Discord server for help, and an open‑source Apache‑2.0 licence.

---

All details are taken directly from the repository’s README; no external assumptions are added.