vLLM Model Runner V2 release notes / what's new

vLLM has announced Model Runner V2 (MRV2), a complete re-implementation of the vLLM model runner designed to increase execution efficiency and reduce technical debt. MRV2 introduces a modular, GPU-native, and async-first core that improves performance without requiring any changes to the existing user-facing APIs.

GPU-Native Input Preparation and Persistent Batching

MRV2 increases throughput by decoupling persistent request state from per-step input tensors and moving input preparation directly to the GPU using Triton kernels.

In vLLM V1, request ordering was tightly coupled to the block table layout, which required complex reordering when requests were added or removed. MRV2 replaces this with a stable state table where each live request occupies a fixed row for its lifetime. The runner then uses a gather operation to produce the correctly ordered input block table for each step. This design eliminates the need for redundant backup state like CachedRequestState and simplifies state management.

By moving the construction of tensors—including input_ids, positions, query_start_loc, and seq_lens—to the GPU, MRV2 achieves:

  • Reduced CPU overhead: Minimizes Python and CPU-side tensor manipulation.
  • Simplified code: Removes constraints imposed by CPU-side operations.
  • Enhanced compatibility: Allows GPU-resident preparation to consume device-side results without synchronization, facilitating better async and speculative decoding.

Async-First Architecture

MRV2 is built on the assumption of zero synchronization between the CPU and GPU across all supported models and features. This async-first design maximizes GPU utilization by allowing the scheduler and worker to prepare step N+1 while the GPU is executing step N.

This architecture specifically solves previous difficulties in combining async scheduling with speculative decoding. Because input preparation now occurs on the device, preparation kernels can directly consume rejection sampling results produced by the GPU. Outputs are transferred to the CPU asynchronously via a separate CUDA stream, fully decoupling the main computation stream from host-side processing.

Triton-Native Sampling Improvements

Sampling has been reworked using optimized Triton kernels to improve memory efficiency and numerical control:

  • Gumbel-Max sampling: Now avoids explicit softmax materialization and utilizes stateless in-kernel RNG.
  • Top-k logprobs: Efficiency is increased by identifying top-k logits first and then computing logprobs only for the selected candidates.
  • Prompt logprobs: Memory usage is reduced through finer-grained chunking, including within a single prompt.
  • Speculative decoding: Compatibility is improved via the use of idx_mapping indirection inside kernels, avoiding the need to expand request state to match every logits vector.

Modularization via ModelState

To support a diverse range of model architectures without cluttering the shared execution path, MRV2 introduces the ModelState abstraction. This interface defines model-specific logic for multimodal embeddings, attention metadata, and CUDA graph capture, allowing the main runner to remain focused on common execution paths.

This modularization has significantly reduced code complexity. The original gpu_model_runner.py file, which exceeded 6,700 lines, has been broken down into smaller files, with the largest now being under 1,300 lines.

Performance Benchmarks

MRV2 delivers measurable performance gains, particularly in scenarios where host-side overhead is a bottleneck:

  • Throughput: In tests using Qwen3-0.6B on a single GB200 GPU, MRV2 achieved 25K output tokens per second compared to 16K for MRV1, representing a 56.2% throughput increase.
  • Latency: Using GLM-4.7-FP8 with MTP=1 on 4xGB200 GPUs, MRV2 achieved a 6.3% reduction in mean Time Per Output Token (TPOT) due to the elimination of CPU-GPU synchronization points during speculative decoding.

Current Status and Limitations

As of v0.18.0, MRV2 is experimental and not yet feature-complete. The following features are currently not supported:

  • Linear attention models (e.g., Qwen3.5, Nemotron 3 Super)
  • Speculative decoding methods other than Eagle, Eagle3, and MTP
  • Logits processors, LoRA, EPLB, and DBO

Users can enable MRV2 by setting the environment variable export VLLM_USE_V2_MODEL_RUNNER=1.

Sources