vllm-project/vime

An LLM post-training framework with vLLM for RL Scaling

Vime – LLM post‑training framework for reinforcement‑learning at scale

What it is

  • Vime is a research‑grade framework that lets you fine‑tune large language models (LLMs) with reinforcement‑learning (RL) methods. It stitches together two proven stacks:
    1. Megatron‑LM – the high‑throughput training engine used for giant models.
    2. vLLM (plus the optional vllm‑router) – a fast inference server that can generate text on many GPUs in parallel.
  • The combination gives a single data‑flow loop where prompts are generated, the model rolls out responses, rewards are computed, and the resulting trajectories are fed back into Megatron for the next training step.

Key capabilities

Capability How Vime delivers it
High‑performance training Megatron handles the heavy‑weight gradient updates while Vime passes the same model weights to vLLM for rollout, keeping both sides in sync.
Flexible data generation Users plug in custom generation functions (multi‑turn dialogs, tool calls, sandboxed environments, etc.). The Data Buffer abstracts prompts, generated samples, and RL rewards, so any workflow can be expressed.
Agentic / long‑horizon support Built‑in hooks for multi‑agent, asynchronous, and code‑generation agents; examples include a coding‑agent RL run on Qwen‑3.8‑27B.

Supported models (inherited from the upstream slime project) – Qwen series, DeepSeek V3 series, LLaMA 3, and other Megatron‑compatible checkpoints.

Architecture at a glance

  • Training (Megatron) – reads batches from the Data Buffer, computes RL losses, and synchronises updated weights to the rollout side.
  • Rollout (vLLM + router) – launches one or more vLLM inference engines, routes generation requests, and can wrap them with custom logic (tool use, environment interaction, reward verification).
  • Data Buffer – the glue that stores prompts, generated trajectories, and rewards; it can be persisted via Ray’s object store or a “straw” shared‑storage backend for fault‑tolerance.

How to get started

  1. Follow the Quick‑Start guide (environment setup, data prep, launch script). All commands are driven by a single train.py entry point.
  2. Choose a rollout backend (--rollout-num-gpus-per-engine, --vllm‑router‑ip, etc.) and any Megatron parallelism flags you need.
  3. Optionally add a custom generate function (--custom-generate-function-path) to implement your own agentic loop.
  4. Run the provided examples – multi‑agent, fully‑async, or the Qwen‑3.8 coding‑agent RL demo.

Extensibility

  • Arguments are split into three groups (Megatron, vLLM, Vime‑specific) and are all exposed on the command line; the source files vime/utils/arguments.py and vime/backends/vllm_utils/arguments.py list the full set.
  • Engine deployment supports heterogeneous GPU groups, external rollout servers, and delta‑weight sync for disk‑based updates, making it possible to run rollouts on a different cluster than the trainer.
  • Testing & reliability – CPU unit tests, GPU end‑to‑end suites, and CI pipelines guard against silent RL bugs; fault‑tolerance docs describe how to recover from crashes using the straw transport.

Who should use Vime?

  • Researchers building RL‑based LLM agents (tool‑using bots, code‑generation assistants, multi‑agent simulations).
  • Teams that already use Megatron or vLLM and want a turnkey RL loop without rewriting the training or inference stack.
  • Anyone needing a production‑ready, configurable pipeline that can scale across many GPUs while keeping the data‑generation logic flexible.

Where to find more

Vime is a genuine open‑source project focused on LLM post‑training with reinforcement learning, tightly integrated with the vLLM ecosystem.

Related

  • Dispatch
    vime RL Framework Release

    vLLM introduces vime, an open-source RL post-training framework that integrates Megatron training with vLLM inference to provide a stable and efficient pipeline for LLM reinforcement learning.

  • Project
    THUDM/slime

    An LLM post-training framework for RL scaling that integrates Megatron-LM and SGLang to provide high-performance training and flexible data generation.

  • Project
    radixark/miles

    An enterprise-grade reinforcement learning framework for large-scale model post-training that integrates SGLang and Megatron-LM for high-throughput rollout and scalable training.

  • Dispatch
    vime ROCm Support for AMD Instinct GPUs

    vLLM has announced ROCm support for vime, enabling end-to-end reinforcement learning post-training workflows to run natively on AMD Instinct MI300X and MI355X GPUs.

  • Project
    verl-project/verl

    A flexible and efficient RL training library for LLMs that simplifies the implementation of RLHF algorithms and scales to trillion-parameter models.