vLLM Performance Benchmarks: Topping Artificial Analysis Leaderboard

vLLM Performance Benchmarks: Topping Artificial Analysis Leaderboard

vLLM has achieved top-tier performance on the Artificial Analysis leaderboard, demonstrating that an open-source inference engine can outperform proprietary stacks on NVIDIA Blackwell Ultra silicon. Key achievements include a per-user output throughput of 230 TPS for DeepSeek V3.2 and a first-place ranking for Qwen 3.5 397B with Time to First Token (TTFT) under 1 second for 10,000-token prompts.

DeepSeek V3.2: Reducing Kernel Launch Overhead

DeepSeek V3.2 performance was primarily limited by GPU kernel launch overhead at low batch sizes rather than raw compute. vLLM addressed this by implementing aggressive op fusion across the attention path, collapsing approximately 33 per-layer kernel launches down to roughly 10. This fusion covers Q and KV normalization, rotary embedding for Q and KV, indexer's layer norm and rotary embedding, FP8 quantization, and KV cache writes.

Performance Gains:

  • Fusion Impact: Delivered a 1.28× speedup at batch size 1 (increasing from 85.8 to 109.3 tok/s on 4× GB200 without MTP).
  • Throughput Scaling: On a single 8× B300 node at concurrency 1, throughput reached 125 tok/s without MTP, 234 tok/s with MTP=1 (~90% draft acceptance), and 262 tok/s with prefill/decode disaggregation (TP=4 + TP=4 + MTP=3).
  • Router GEMM Kernel: A specialized kernel for MoE routing dimensions at small decode batch sizes provided an additional 6% speedup at batch 1 (PR #34302).
  • TopK Kernel: A new kernel for the sparse attention indexer improved per-token latency by up to 17% on 128K-context decode (PR #37421).

These optimizations now serve as the foundation for vLLM's DeepSeek V4 support.

MiniMax-M2.5: Speculative Decoding and Custom Fusion

For MiniMax-M2.5, vLLM combined targeted kernel fusion with a custom EAGLE3 draft model. The draft model was trained using TorchSpec, a torch-native online speculative decoding framework, to match the base model's exact token distribution by consuming live vLLM-generated hidden states.

Technical Enhancements:

  • Speculative Infrastructure: Improvements to the MRV2 path included a draft model metadata fix to increase acceptance rates at later positions (PR #38311) and CUDA graph support for draft prefill (PR #37588).
  • QK-Norm Fusion: A custom fuse_minimax_qk_norm kernel was implemented to handle non-standard attention normalization where Q and K variances are reduced across tensor-parallel ranks before per-channel scaling (PR #37045).
  • Ceiling Performance: With fusion and a perfect draft model (100% acceptance), the stack reached 326 tok/s at concurrency 1 (TP=4, EAGLE3 + 3 speculative tokens).

Qwen 3.5 397B: Linear Attention Optimizations

Qwen 3.5 397B utilizes linear attention and a non-standard normalization variant that initially bypassed vLLM's standard allreduce_rms fusion, causing approximately half of decode time to be spent on un-fused cross-device reduces.

Optimization Suite:

  • Normalization Fix: Updated the allreduce_rms fusion pass to recognize Qwen's specific normalization variant, resulting in a ~5% TPOT improvement at batch sizes greater than 1.
  • Post-Conv Fusion: Implemented kernel fusion for the post-projection convolution path specific to linear-attention architectures (PR #37813).
  • Execution Efficiency: Integrated kernel-level optimizations for the qk-norm + rope path and dual-stream execution to overlap independent compute branches.

Production Results:

  • Concurrency 1: Reached 163 tok/s (TEP=8, post-conv fusion).
  • Concurrency 256: Reached 7.33 req/s, a 10% increase over the 6.69 req/s baseline.

Implications for Open-Source Inference

All optimizations detailed above—including the DeepSeek V3.2 attention-path fusions, MiniMax EAGLE3 recipes, and Qwen 3.5 fusions—are either already merged into the vLLM main branch or are currently in flight. This ensures that the performance gains achieved in these benchmarks are available to all users of the open-source engine, challenging the assumption that peak inference performance requires proprietary software stacks.

Sources