vime RL Framework Release

vime RL Framework Release

vLLM has announced the release of vime, an open-source post-training framework designed to simplify and stabilize reinforcement learning (RL) for Large Language Models (LLMs). By integrating the training stack of slime with the inference capabilities of vLLM, vime creates a unified pipeline that allows distributed training and inference to operate reliably under a single architecture.

Architecture and Design

vime utilizes a three-stage, decoupled train-inference design based on the slime paradigm, replacing the standard rollout backend with vLLM. The architecture consists of three primary components:

  • Training (Megatron): Manages the main training loop, handles parameter updates, and synchronizes weights to the rollout side.
  • Rollout (vLLM + Router): Performs inference sampling to produce training samples accompanied by reward or verifier signals.
  • Data Buffer: Acts as the connector between training and rollout, managing custom rollout logic and prompt injection.

Key Technical Capabilities

vime is engineered for stability and flexibility across various model architectures and hardware configurations:

  • Train-Inference Alignment: vime maintains a controllable train_rollout_logprob_abs_diff across Dense and Mixture-of-Experts (MoE) scenarios. For MoE models, vime implements R3 (routing replay) to further minimize the mismatch between training and inference.
  • Broad Model and Algorithm Support: The framework provides end-to-end examples and CI-verified paths for RL algorithms including GRPO and PPO, and supports models such as Qwen3 Dense/MoE and GLM-4.5.
  • Unified Hardware Abstraction: Training, rollout resources, and cluster topology are abstracted uniformly, allowing the RL pipeline to be reused across different hardware backends as the vLLM ecosystem evolves.
  • Simplified Configuration: The system inherits parameter conventions from slime and Megatron, utilizing a --vllm- prefix for passing arguments to the vLLM side.

Performance Benchmarks and Validation

Hardware Efficiency

In tests using Qwen3-30B-A3B with GRPO on the dapo-math-17k dataset (8-GPU colocate), vime demonstrated significant performance gains on newer hardware:

  • GB200: Mean step time of approximately 147 seconds.
  • H200: Mean step time of approximately 252 seconds.
  • Comparison: The GB200 end-to-end step speed is approximately 1.72x faster than the H200.

Stability and Alignment

  • Qwen3-4B (A100): Using GRPO with 4 training and 4 inference non-colocate GPUs on gsm8k, vime's train_rollout_logprob_abs_diff remained stable at approximately 0.011, whereas the baseline drifted to roughly 0.77.
  • Qwen3-30B-A3B MoE (A100): With 4 training and 4 inference GPUs, enabling R3 routing replay reduced the logprob difference from 0.019 to 0.013.
  • GLM-4.5-Air (GB200): Using GRPO with 8-GPU colocate on dapo-math-17k, the raw_reward trended upward over 100 steps (mean of ~0.56) while the train_rollout_logprob_abs_diff remained stable between 0.02 and 0.03.

Development Roadmap

vime is currently evolving with a focus on three strategic areas:

  1. vLLM Integration: Adopting advanced vLLM features including PD disaggregation, FP8, Router, and multi-model serving.
  2. Hardware Expansion: Utilizing vLLM's hardware plugin system to extend efficiency to more accelerators and cluster configurations.
  3. Algorithmic Advancement: Developing fully asynchronous pipelines, Agentic RL for multi-turn tool calling, and support for new architectures such as VLMs and MoE.

Getting Started

vime is open-sourced under the Apache 2.0 license. Users can launch training via train.py or train_async.py after configuring Megatron training and vLLM rollout resources. The project is available on GitHub at github.com/vllm-project/vime.

Sources