radixark/miles

Miles is an enterprise-facing reinforcement learning framework for LLM and VLM post-training, forked from and co-evolving with slime.

What it solves

Miles is designed for the high-performance post-training of large-scale models, specifically focusing on reinforcement learning (RL) at a trillion-parameter scale. It addresses the stability, precision, and throughput issues that typically arise when training massive models across distributed systems.

How it works

Miles combines high-throughput rollout (generation) using SGLang and scalable training using Megatron-LM (with a PyTorch FSDP2 backend option). It uses a fully asynchronous RL pipeline where rollout and training workers are decoupled to minimize idle time. To maintain stability, it implements "Token-in-token-out" (TITO) to avoid re-tokenization errors and "Rollout Routing Replay" (R3) to prevent MoE routing mismatches between rollout and training.

Who it’s for

It is built for model labs, hardware and cloud providers, and researchers who need to train frontier-scale models (such as DeepSeek-V4 or Kimi-K3) using RL recipes like GRPO, PPO, and REINFORCE++.

Highlights

  • Trillion-parameter scale: Optimized for massive models with P2P RDMA for fast weight updates.
  • Low-precision training: Supports MXFP8, NVFP4, FP8, and INT4 QAT to reduce memory and compute costs.
  • Broad hardware support: Compatible with NVIDIA (H100, B200, etc.) and AMD (MI300X, MI355X) GPUs.
  • Agentic environment integration: Connectors for coding and computer-use agents via platforms like E2B, Modal, and NeMo Gym.
  • Fault tolerance: Automatically recovers failed SGLang engines without pausing the run.

Related

  • Project
    verl-project/verl

    A flexible and efficient RL training library for LLMs that simplifies the implementation of RLHF algorithms and scales to trillion-parameter models.

  • Project
    vllm-project/vime

    Vime is an open‑source framework that combines Megatron‑LM training with vLLM inference to enable high‑performance reinforcement‑learning fine‑tuning of large language models. It offers a unified data‑flow loop, flexible custom generation hooks, and support for many modern LLMs (Qwen, DeepSeek, LLaMA 3).

  • Project
    NVIDIA-NeMo/RL

    A scalable post-training library for multimodal models that streamlines reinforcement learning and alignment using high-performance backends like Megatron Core and vLLM.

  • Project
    nvidia-cosmos/cosmos-rl

    A scalable Reinforcement Learning framework specialized for Physical AI applications, providing a decoupled architecture for large-scale training workloads.

  • Project
    THUDM/slime

    An LLM post-training framework for RL scaling that integrates Megatron-LM and SGLang to provide high-performance training and flexible data generation.