radixark/miles

Miles is an enterprise-facing reinforcement learning framework for LLM and VLM post-training, forked from and co-evolving with slime.

What it solves

Miles is designed for large-scale model post-training, specifically addressing the instability and performance bottlenecks associated with reinforcement learning (RL) at the trillion-parameter scale. It provides an enterprise-ready framework to handle the high throughput required for rollouts and the massive scale of training, while ensuring numerical stability and fault tolerance.

How it works

Miles decouples rollout and training workers to enable fully asynchronous RL. It uses SGLang for high-throughput generation (rollout) and Megatron-LM (or PyTorch FSDP2) for scalable training. To maintain efficiency, it implements P2P RDMA for fast weight updates across disaggregated setups and supports low-precision training (MXFP8, NVFP4, INT4 QAT) to reduce memory and compute overhead.

Who it’s for

It is intended for model labs, hardware and cloud providers, and researchers who are training frontier-scale models (such as DeepSeek-V4, Kimi-K3, or Nemotron 3 Ultra) using RL recipes like GRPO, PPO, and REINFORCE++.

Highlights

  • Fully Async RL: Decouples rollout and training to minimize bubbles and support configurable on- and off-policy schedules.
  • Token-in-token-out (TITO): Eliminates the need for detokenize/retokenize round-trips between rollout and training.
  • Rollout Routing Replay (R3): Prevents MoE routing mismatch by replaying expert routing in the trainer's forward pass.
  • Broad Hardware Support: Compatible with NVIDIA (H100, B200, etc.) and AMD (MI300X, MI355X) GPUs via ROCm.
  • Agentic Environment Integration: Connects to sandboxes like E2B and Modal for training coding and computer-use agents.

Related

  • Project
    THUDM/slime

    An LLM post-training framework for RL scaling that integrates Megatron and SGLang to provide high-performance training and flexible data generation.

  • Project
    PrimeIntellect-ai/prime-rl

    A framework for large-scale asynchronous reinforcement learning designed to train 1T+ parameter MoE models across 1000+ GPUs using FSDP2 and vLLM.

  • Project
    nvidia-cosmos/cosmos-rl

    A scalable Reinforcement Learning framework specialized for Physical AI applications, providing a decoupled architecture for large-scale training workloads.

  • Project
    vllm-project/vime

    An LLM post-training framework for RL scaling that integrates Megatron for training and vLLM for high-performance rollouts and data generation.

  • Project
    alibaba/ROLL

    An efficient scaling library for reinforcement learning with LLMs that optimizes training and inference across large GPU clusters using a distributed Ray-based architecture.