vLLM Adds NVIDIA Vera Rubin NVL72 Support with 7.8× Throughput Gains

TL;DR

vLLM announced official support for NVIDIA Vera Rubin NVL72 GPUs, achieving up to 7.8× higher inference throughput per GPU compared to the previous‑generation GB200 NVL72 and delivering up to 3.7× higher Vision‑Language Model (VLM) throughput on MLPerf.

Overview of Vera Rubin NVL72

  • Compute advantage – The NVL72 provides roughly 5× the NVFP4 inference FLOPS of the GB200 NVL72.
  • Memory bandwidth – HBM4 on Rubin offers 2.4× the bandwidth of the GB200’s HBM3e.
  • Inter‑GPU networking – Sixth‑generation NVLink delivers 1.7× more bandwidth than the Blackwell generation, reducing collective‑communication latency.
  • Softmax acceleration – Exponential throughput is 2× faster in FP32 and 4× faster in BF16/FP16, helping attention keep pace with faster GEMM kernels.

These hardware improvements are purpose‑built for agentic AI workloads and form the foundation for the performance gains reported by vLLM.

Day‑0 Compatibility

  • Blackwell‑family kernels – Rubin is a Blackwell‑derived architecture (SM107). vLLM’s existing Blackwell kernels (targeting SM100f) run on Rubin without modification.
  • Container availability – Nightly Docker images built with CUDA 13.4 and PyTorch 2.15 are published as vllm/vllm-openai:cu134-nightly and can be pulled directly for Rubin deployments.
  • Model coverage – Out‑of‑the‑box support now includes DeepSeek, Kimi, GLM, MiniMax, and other community models.

Rubin‑Tuned Kernels

vLLM integrates several Rubin‑specific kernels via FlashInfer 0.7.0 and its own forks:

  • Attention, GEMM, and MoE kernels tuned for Rubin’s tensor‑core instructions.
  • MiniMax Sparse Attention (MSA) prefill kernel optimized for Rubin’s exponentials.
  • FP8 kernels for attention and MSA, leveraging Rubin’s high‑throughput FP8 paths.

These kernels are upstreamed to libraries such as FlashInfer, MSA, and Humming, ensuring broader ecosystem benefit.

Locality‑Aware Mixture‑of‑Experts (MoE)

  • CUDA 13.4 locality domains allow SMs to read weights from the nearest HBM region, reducing cross‑domain traffic.
  • Split‑N strategy shards MoE weight matrices column‑wise, assigning each shard to a distinct locality domain.
  • Performance impact – Early experiments on the MiniMax M3 MoE model show an average 1.2× speed‑up for small‑token decode when locality domains are enabled, even when only 200 of 212 SMs are active (default mode). Backfill mode restores full SM utilization with similar gains.

These results demonstrate that Rubin’s increased memory bandwidth can be effectively harnessed through software‑level memory‑locality optimizations.

Benchmark Results

AgentX (SemiAnalysis) – LLM Throughput

  • Throughput per GPU – vLLM on Rubin NVL72 achieves 7.84× the throughput of a GB200 NVL72 at matched interactivity.
  • TPS‑constrained scenario – Under a 150 TPS limit, Rubin delivers 5.18× higher throughput.

MLPerf Inference v6.1 – Vision‑Language Model

  • Model – Qwen3‑VL‑235B‑A22B served via vLLM (backend) and Dynamo (router).
  • Throughput gain – Rubin NVL72 provides up to 3.7× higher throughput than GB300 NVL72 across offline, server, and interactive modes.
  • Reference – Detailed results are published in NVIDIA’s blog post on Rubin’s MLPerf performance.

These early numbers confirm that Rubin’s architectural upgrades translate into substantial real‑world inference speedups for both text‑only and multimodal workloads.

Roadmap and Future Optimizations

The vLLM community plans several next steps to further exploit Rubin’s capabilities:

  • Integrate FlashInfer’s sm107 MegaMoE kernel.
  • Fully enable locality domains for all MoE layers.
  • Overlap kernels using PDL and Lamport Sync to hide latency.
  • Develop mega‑kernels for latency‑sensitive serving.
  • Optimize KDA/MLA kernels for Kimi K3 and Rubin‑specific CSA/HCA kernels for DeepSeek‑V4.1‑Flash.
  • Complete Rubin‑specific MSA decode kernels and the CFT counted‑write MoE all‑to‑all kernel via FlashInfer.

Continued collaboration among NVIDIA, Inferact, Red Hat, and the broader vLLM community will drive these enhancements.

Getting Started

  1. Pull the Rubin‑compatible nightly image:
    docker pull vllm/vllm-openai:cu134-nightly
  2. Launch vLLM with the appropriate backend flags (e.g., --linear-backend flashinfer_cutedsl, --moe-backend flashinfer_cutedsl).
  3. Enable locality domains by setting cudaDevSmResourceGroupBackfill=1 if full SM utilization is required.

The repository’s PR references for Rubin support include #55953, #53443, #54640, #56545, and #59288.

Acknowledgements

The release reflects joint effort from NVIDIA, Inferact, Red Hat, and the vLLM community. Special thanks go to the engineers who provided early hardware access, contributed Rubin‑specific kernel tuning, and maintained daily Docker builds.

Appendix: Kernel Configuration Cheatsheet

Kernel Enable Flag Notes
Dense NVFP4 / MXFP4 GEMM --linear-backend flashinfer_cutedsl (NVFP4) or flashinfer_cutlass (MXFP4) Default for matching checkpoints
NVFP4 MoE --moe-backend flashinfer_cutedsl
Batched expert‑format MoE (W4A4) `--enable-expert-parallel --data-parallel-size N --all2all-backend deepep_low_latency nixl_ep`
FP8 BMM (static per‑tensor) Autotuned via FlashInfer
FP8 attention (TRT‑LLM) --kv-cache-dtype fp8 + `--attention-backend FLASHINFER FLASHINFER_MLA`
FP8 MSA prefill --attention-config.minimax_m3_msa_decode_backend=cutlass

These flags expose the Rubin‑optimized paths that deliver the performance gains described above.

Sources