Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries
TL;DR – Hugging Face surveyed 16 open‑source reinforcement‑learning (RL) libraries and found that all successful async RL systems separate inference and training onto different GPU pools, use a rollout buffer, and push model weights asynchronously; Ray is the dominant orchestration framework, NCCL broadcast is the common weight‑sync method, and support for LoRA and Mixture‑of‑Experts (MoE) training is still sparse.
1. Why Async RL Matters
Async RL eliminates the generation bottleneck that makes synchronous training of large language models (LLMs) idle for up to 60 % of wall‑clock time. In synchronous pipelines, a single batch of 32 K‑token rollouts on a 32‑billion‑parameter model can take hours, while the GPUs allocated for gradient updates sit idle. By disaggregating inference and training onto separate GPU pools and connecting them with a rollout buffer, generation can continue while the trainer consumes previously generated data, dramatically improving GPU utilisation.
2. Survey Overview
Hugging Face examined sixteen open‑source async RL libraries (AReaL, ART, Atropos, MILES, NeMo‑RL, OAT, open‑instruct, PipelineRL, PRIME‑RL, ROLL, SkyRL, SLIME, TorchForge, Tunix, verl, verifiers‑rl). Each library was evaluated across seven orthogonal axes:
- Orchestration & concurrency primitive – how distributed components are coordinated.
- Rollout buffer design – the data structure that carries generated samples from inference to training.
- Weight synchronisation protocol – how updated parameters are pushed to the inference pool.
- Staleness management – strategies for handling off‑policy rollouts.
- Partial‑rollout handling – what happens to in‑flight generations when weights change.
- LoRA training support – ability to train adapter‑only parameters and sync them efficiently.
- Distributed training backend & parallelism – the parallelism strategy (FSDP, Megatron, DeepSpeed, JAX, etc.) and support for Mixture‑of‑Experts.
The full comparison table is available in the original blog post; the sections below summarise the key findings for each axis.
3. Orchestration & Concurrency Primitive
| Orchestration type | What it is | Libraries using it |
|---|---|---|
| Distributed actor model (Ray) | Stateful actors with asynchronous RPC, object store, and built‑in fault tolerance. | AReaL, verl, SkyRL, NeMo‑RL, SLIME, MILES, ROLL, OAT, open‑instruct, etc. |
| Native Python concurrency | Threads, asyncio, multiprocessing; no external runtime. |
verifiers‑rl, PipelineRL (intra‑pool), ART, AReaL (asyncio‑based) |
| Pub/Sub message bus | Decoupled producers/consumers via Redis streams or append‑only files. | PipelineRL (inter‑pool), SLIME (async mode) |
| HTTP microservices | Independent services communicating over REST. | Atropos |
Finding: Ray dominates the landscape, appearing in 8/16 libraries. Its actor model matches the heterogeneous components of RL (inference server, trainer, reward model, environment pool) and provides automatic scheduling, fault tolerance, and zero‑copy data transfer via the Ray object store. However, Ray adds a heavyweight runtime, which motivates lighter alternatives (native Python, Pub/Sub) for smaller deployments.
4. Rollout Buffer Design
| Buffer pattern | Depth (max in‑flight batches) | Libraries | Remarks |
|---|---|---|---|
| No buffer (synchronous) | 0 | TRL (current), ART (gather‑all‑then‑train) | Generation and training alternate; maximal staleness but zero overlap. |
| Double‑buffer | 1 | verifiers‑rl, SLIME (async mode), MILES, OAT | Overlaps exactly one generation batch with one training step; minimal staleness. |
| Bounded async queue | 2–K | SkyRL, verl, NeMo‑RL, ROLL, PRIME‑RL, TorchForge, Tunix, open‑instruct, AReaL | Multiple batches in flight; staleness bounded by queue capacity. |
| Unbounded / stream | Unlimited | PipelineRL (Redis streams), SLIME (full async), Atropos | Continuous generation; staleness must be controlled by version tags or importance‑sampling. |
A deeper queue improves throughput but requires explicit staleness management (see Axis 4).
5. Weight Synchronisation Protocol (Axis 3)
The protocol determines latency and interrupt granularity when pushing new weights from the trainer to the inference pool.
5.1 Transport mechanisms
| Mechanism | Typical latency | Libraries |
|---|---|---|
| NCCL broadcast | 100–500 ms | Most libraries (PipelineRL, SkyRL, SLIME, MILES, ROLL, OAT, NeMo‑RL, PRIME‑RL, open‑instruct, AReaL) |
| NCCL + bucketing | ~20 ms | verl |
| Shared‑memory / CUDA IPC | Very low | NeMo‑RL, MILES |
| Filesystem + HTTP | Medium (seconds) | PRIME‑RL, AReaL, ART |
| HTTP PUT | High (seconds) | verifiers‑rl |
| JAX cross‑mesh | Low | Tunix |
5.2 Interrupt granularity
| Granularity | Behaviour | Libraries |
|---|---|---|
| Never stop (per‑token swap) | Weights are swapped between forward passes; generation never aborts. | PipelineRL, open‑instruct (opt‑in) |
| Per HTTP request abort | Ongoing HTTP calls are cancelled and retried with a prefix. | SkyRL, SLIME |
| Soft pause (drain in‑flight) | New requests are blocked; existing generations finish before sync. | PRIME‑RL, AReaL, open‑instruct (default), verl (async) |
| Per‑batch/blocking | Generation and training take turns; sync blocks both sides. | NeMo‑RL, ROLL, OAT, TorchForge, Tunix, verifiers‑rl, Atropos |
Finding: Only PipelineRL achieves true never‑stop weight updates, swapping parameters between token‑level forward passes. All other libraries pause at a coarser boundary, which introduces a brief period where inference runs on stale weights.
6. Staleness Management (Axis 4)
When generation and training overlap, rollouts become off‑policy. Libraries adopt three orthogonal strategies:
- Per‑sample version rejection – discard samples whose
model_versionexceeds a configurable lag. - Depth bounding – limit the number of in‑flight batches, guaranteeing a maximum version gap by construction.
- Importance‑sampling (IS) correction – re‑weight stale samples by the ratio (\frac{\pi_{\theta}(a|s)}{\pi_{\text{old}}(a|s)}), often clipped.
| Library | Version rejection | Depth bounding | IS correction |
|---|---|---|---|
| AReaL | ❌ | ✅ | ⚠️ (optional) |
| ART | — (synchronous) | — | — |
| Atropos | ❌ | ✅ | ❌ |
| MILES | ❌ | ❌ | ✅ |
| NeMo‑RL | ✅ | ❌ | ❌ |
| OAT | ❌ | ❌ | ✅ |
| open‑instruct | ❌ | ✅ | ⚠️ (optional) |
| PipelineRL | ✅ | ❌ | ❌ |
| PRIME‑RL | ✅ | ✅ | ✅ |
| ROLL | ❌ | ❌ | ✅ |
| SkyRL | ❌ | ✅ | ❌ |
| SLIME | ❌ | ❌ | ✅ |
| TorchForge | ✅ | ❌ | ❌ |
| Tunix | ❌ | ✅ | ❌ |
| verl | ❌ | ❌ | ✅ |
| verifiers‑rl | ❌ | ✅ | ❌ |
Hybrid approaches (e.g., PRIME‑RL, open‑instruct) combine depth bounding with optional IS weighting to keep pipelines simple while retaining robustness.
7. Partial‑Rollout Handling (Axis 5)
Long‑context rollouts (tens of thousands of tokens) may still be generating when a weight update arrives. Strategies include:
| Strategy | Libraries | Description |
|---|---|---|
| Implicit continuation | PipelineRL | No interruption; weight swap occurs between token forward passes. |
| Abort + retry with prefix | SkyRL, SLIME | In‑flight generations are cancelled; the partial prefix is re‑submitted under the new policy. |
| Explicit save/resume | verl (full async) | Partial token IDs and KV cache are saved, sync occurs, then generation resumes from the saved state. |
| Group cancellation | PRIME‑RL | Stale rollout groups are discarded; new weight sync happens between HTTP requests. |
| Soft pause (drain) | AReaL | New tasks stop; existing tasks run to completion before sync. |
| No support | verifiers‑rl, OAT, Atropos, Tunix | Sync only after all in‑flight generations finish. |
Only PipelineRL and verl provide true never‑stop behaviour; the rest rely on abort‑or‑drain mechanisms.
8. LoRA Training Support (Axis 6)
LoRA reduces trainable parameters dramatically, enabling adapter‑only weight sync where only the small adapter delta is broadcast to the inference server. This can shrink NCCL transfer time from hundreds of milliseconds to sub‑millisecond for 7 B+ models.
| Library | LoRA supported? | Backend | Adapter‑only sync |
|---|---|---|---|
| AReaL | ✅ | HF peft (FSDP2/Megatron) |
✅ |
| ART | ✅ | Unsloth / Megatron | ✅ |
| Atropos | ✅ | HF peft |
✅ |
| MILES | ✅ | Megatron‑Bridge | ✅ |
| NeMo‑RL | ✅ (custom) | DTensor / Megatron | ❌ (no evidence) |
| OAT | ✅ | HF peft |
✅ |
| open‑instruct | ❌ (code present but not wired) | — | ❌ |
| PipelineRL | ✅ | HF peft |
❌ (full broadcast) |
| PRIME‑RL | ✅ | Custom MultiLoRA | ✅ |
| ROLL | ✅ (DeepSpeed only) | DeepSpeed | ❌ |
| SkyRL | ✅ | HF peft / Megatron‑Bridge |
✅ |
| SLIME | ❌ | — | ❌ |
| TorchForge | ❌ | — | ❌ |
| Tunix | ✅ | qwix (JAX) | ✅ |
| verl | ✅ | HF peft / Megatron‑Bridge |
✅ |
| verifiers‑rl | ✅ | HF peft + FSDP2 |
✅ |
LoRA‑only sync dramatically relaxes the interrupt model: even libraries that abort per‑request can afford frequent weight updates because the data transfer is tiny.
9. Distributed Training Backend & Parallelism (Axis 7)
The training backend dictates model size limits, collective communication patterns, and compatibility with MoE.
| Library | Backend | Parallelism (DP/TP/PP/EP) | MoE support |
|---|---|---|---|
| AReaL | FSDP2, Megatron, Archon | DP, SP, TP, PP, CP, EP | ✅ |
| ART | Unsloth, Megatron | DP, TP, EP | ✅ |
| Atropos | PyTorch native, TRL | DP | ❌ |
| MILES | Megatron, FSDP2 | DP, TP, PP | ✅ |
| NeMo‑RL | DTensor, Megatron | DP, SP, TP, PP, CP, EP | ✅ |
| OAT | DeepSpeed | DP, TP | ❌ |
| open‑instruct | DeepSpeed | DP, SP | ❌ |
| PipelineRL | DeepSpeed | DP, SP | ❌ |
| PRIME‑RL | FSDP2 | DP, TP, CP, EP | ✅ |
| ROLL | DeepSpeed, Megatron, FSDP2 | DP, SP, TP, PP, CP, EP | ✅ |
| SkyRL | FSDP, Megatron‑Bridge | DP, SP, TP, PP, EP | ✅ |
| SLIME | Megatron | DP, TP, PP, SP | ✅ |
| TorchForge | FSDP2 (Monarch) | DP, TP, CP | ❌ |
| Tunix | JAX/XLA | DP, TP | ❌ |
| verl | FSDP, Megatron | DP, SP, TP, PP, CP, EP | ✅ |
| verifiers‑rl | DeepSpeed | DP | ❌ |
Key implication: MoE training (expert parallelism) is only supported by libraries built on Megatron or FSDP2 with explicit EP handling (AReaL, verl, PRIME‑RL, SkyRL, ROLL, NeMo‑RL). Libraries that rely solely on ZeRO (DeepSpeed, PyTorch FSDP without EP) can load MoE checkpoints but lose the sparsity advantage because all experts are sharded across every rank.
10. Emerging Design Pressures
10.1 Critic‑Free Algorithms
Removing value networks (e.g., GRPO, REINFORCE++) frees memory, allowing larger rollout batches, but increases the frequency of weight updates. Staleness management becomes more critical because policy drift accelerates with larger group sizes (G = 8‑32). Per‑sample version tagging and IS correction are therefore essential for stable training.
10.2 Process Rewards
Scoring intermediate reasoning steps (process reward models) adds a non‑trivial compute stage between generation and training. Async pipelines must therefore include a reward‑actor that runs concurrently with the trainer, as done in PRIME‑RL and NeMo‑RL. Without this, reward computation becomes the new bottleneck.
10.3 Multi‑Agent Co‑Evolution
Multi‑agent self‑play compounds the straggler problem: the effective rollout length becomes the product of per‑agent lengths, dramatically increasing variance. Buffer designs must treat an entire multi‑agent episode as a single atomic unit, and staleness strategies need per‑episode version tracking rather than per‑sample.
10.4 Training‑Inference Mismatch for MoE
Two structural mismatches have been identified in DeepSeek‑v3.2:
- Expert routing inconsistency – inference and training may select different experts for the same token due to floating‑point differences. The solution (“Keep Routing”) requires the inference server to return routing decisions and the trainer to enforce them.
- Sampling‑mask mismatch – top‑p/top‑k truncation during generation leads to a different action space than the full‑vocab training forward pass. “Keep Sampling Mask” records the truncation mask and re‑applies it during training. No surveyed library currently implements either feature, highlighting a gap for future async‑RL systems.
10.5 Distillation as Async RL
On‑policy distillation (student generates, teacher scores) follows the exact same async pattern as RL: generation → scoring → gradient update → weight sync. Consequently, all design axes apply unchanged. A fully generic async trainer should expose the scoring step as a pluggable component rather than a hard‑coded verifier, enabling both RL and distillation workloads.
11. Design Choices for TRL’s Async Trainer
Based on the survey, the TRL team plans the following concrete decisions for its upcoming async trainer:
- Lightweight orchestration – avoid heavyweight runtimes where possible; use native Python
asynciocombined with a minimal actor abstraction. - Bounded queue with per‑token
model_version– each token carries the policy version that generated it, enabling fine‑grained IS correction and eliminating the need for later retro‑fits. - NCCL weight sync with packed transfers – leverage vLLM’s
NCCLWeightTransferEnginewith bucketing to broadcast weights in ~20 ms chunks, reducing sync latency dramatically. - Partial‑rollout support – implement a prefix‑resume mechanism for agentic workloads, allowing in‑flight generations to continue under the new policy after a weight update.
These choices aim to combine the best practices observed across the ecosystem while keeping the implementation approachable for the broader TRL community.
Bottom line: Async RL training is now the de‑facto standard for large‑scale LLM post‑training. The survey shows a clear convergence on disaggregated inference, rollout buffers, and asynchronous weight pushes, with Ray as the predominant orchestration layer and NCCL broadcast as the default sync method. Future work must address LoRA‑only sync, MoE routing consistency, and multi‑agent episode handling to keep pace with the next generation of frontier models.