DeepSeek-V4.1-Flash on NVIDIA Vera Rubin NVL72 achieves 58× higher throughput than GB300

TL;DR

vLLM runs DeepSeek‑V4.1‑Flash on a single NVIDIA Vera Rubin NVL72 rack at 1.7 M tokens / s per GPU (each user sees 387 tokens / s, a 58× improvement over the GB300 NVL72 baseline) and peaks at 2.52 M tokens / s overall, by exploiting Rubin‑specific kernel optimizations, a specialized TP‑8 decode path, and rack‑scale KV‑cache and Engram pooling.


Hardware context: Vera Rubin NVL72

The Rubin NVL72 server introduces three architectural features that shape the optimizations:

  • Dual memory domains per GPU – each SM can read its own domain much faster than the other domain, making bandwidth‑bound data (MoE weights, LM head, MegaMoE buffers) highly sensitive to placement.
  • SM107’s wider block‑scaled MMA instructions – the new K64 mixed‑precision (FP8×FP4) and K128 FP4 MMAs double the work per instruction, accelerating GEMMs in MegaMoE, attention, and the MQA‑logits kernel.
  • 72 GPUs share a single NVLink domain – Tensor‑Parallel (TP) across two trays incurs virtually no extra all‑reduce cost, and prefill instances can read each other’s HBM directly, eliminating redundant weight copies.

These properties enable the dramatic throughput gains reported below.


Core kernel rewrites for Rubin

DEP prefill kernels – ~28 % faster per step

  • MegaMoE tiling now follows the larger of the local token count and the DP‑average token count, cutting MegaMoE time by 55 % over a 40‑step replay and reducing single‑request time‑to‑first‑token (TTFT) by 13–39 % for 8 K–32 K tokens.
  • Mega‑mHC residual stream lives in a die‑local pool striped across both memory domains; each SM drains its own domain before helping the other, reducing call latency from 354 µs → 317 µs at 32 K tokens.
  • MegaAttn query GEMM moves from cuDNN (SM100) to FlashInfer’s CuTe DSL on SM107’s K64 MMA, delivering 13–18 % speed‑up for 4 K–32 K rows.
  • Pipeline‑friendly attention issues SVs ahead of QKs, prefetches index rows into L2, and lets each TMEM lane score its own keys, yielding 15 % faster kernels at 16 K–32 K tokens.
  • Ticketed task ring for Mega‑mHC replaces fixed strides with an atomic ticket counter, improving large‑grid performance by 18 % at 32 K tokens.
  • MegaGate contender filtering evaluates sqrt‑softplus only for experts that can still make the top‑6, cutting kernel time by 3–13 % at 4 K tokens.

DEP decode kernels – ~25 % faster per step

  • Die‑local MegaMoE weights split expert columns across memory domains, giving a 10.1 % speed‑up at batch‑size 32 and 5.5 % at batch‑size 168.
  • Wide‑EP dispatch reduces NVLink round‑trips to one per token, improving MegaMoE forward time by 17 % at DEP8.
  • Combine‑without‑barrier uses a sentinel buffer so each CTA combines as soon as its GEMM finishes, adding 3.5–10.8 % decode speed‑up.
  • Split‑K GEMMs distribute K across an SM107 cluster, accelerating attention projections by 10–38 % and overall decode forward by up to 3.1 %.
  • MQA logits kernel packs 5–6 query rows into a single 192‑wide tile and uses SM107’s K128 MMA, achieving 15–22 % faster execution.
  • Split Mega‑mHC isolates only the critical Post, route‑collapse, and RMSNorm steps, delivering 2–4.8 % decode gains.
  • Deeper attention pipelines keep three index‑load blocks and four gather batches in flight, yielding 7–10 % faster decode kernels.

Low‑latency TP‑8 decode path

A dedicated decode pipeline targets the smallest per‑user latency:

  • TP‑8 across two trays halves per‑GPU weight reads with only a 0.3 µs NVLink penalty, giving a 35 % step‑time reduction.
  • Packed‑FP4 weight stages on SM107’s K64 MMA increase the number of stages from 7 to 11, saving 57 µs per step.
  • Die‑local LM head streams only half the weights per SM, eliminating cross‑domain stalls.
  • All‑reduce + mHC fused publishes GEMM epilogues directly into a Lamport mailbox, cutting 30 µs per step.
  • Sparse MLA runs an 8‑head kernel per rank; early KV‑cache inserts and fused RoPE reduce latency by 16–21 µs.
  • Shared expert inside routed GEMMs moves the MoE tail into the GEMM for layers with ≤16 rows, saving 220 µs per step.
  • Early kernel launches and Engram all‑gathers on the layer stream shave another 40 µs per step.

Result: on a single‑user workload (batch 1) the specialized path reaches 1,810 tokens / s (P90), up from 1,117 tokens / s in the GB300 baseline.


Dynamo workload‑aware routing & rack‑scale memory pools

Rubin’s speed enables far higher concurrency, exposing the limits of per‑rank KV caches. vLLM replaces per‑rank CPU tiers with three shared pools:

  • Mooncake KV pool – a DRAM‑wide store across all prefill/decode trays. Any rank can load a prefix over RDMA, turning a moved request into a KV load rather than a full recompute. Prefill steps under‑filled by pending lookups drop from 6–14 % to <0.5 %.
  • HBM pool – spare HBM on each GPU is exposed to the Mooncake store, extending the KV pool beyond DRAM and allowing the rest of the GPU memory to be used for other ranks.
  • Engram pool – the two 94.4 GiB hashed n‑gram tables are split across the HBM of a peer group (≈10.5 GiB per GPU) and accessed via NVLink instead of host memory, freeing host RAM for ~1,500 additional sessions.

Workload‑aware routing (Dynamo) now assigns each request to the rank with the least uncached work, eliminating the DP imbalance caused by session affinity. In a 4,800‑concurrency simulation, the combined pools and routing deliver:

  • 68 % more requests per second
  • TTFT P50 reduced from 29.8 s → 4.2 s
  • Hit rate improved from 91.8 % → 96.1 %
  • Idle‑rank time cut from 10.3 % → 1.3 %

Performance summary

Metric Vera Rubin NVL72 GB300 NVL72 Speedup
Tokens / s per GPU (total) 1.7 M 29 k 58×
Tokens / s per user (typical) 387 6.7 58×
Peak rack‑wide throughput 2.52 M 43 k 58×
Low‑latency per‑user rate (TP‑8) 1,810 tokens / s 1,117 1.6×

All numbers are measured with the SemiAnalysis AgentX benchmark, which reflects realistic agentic serving workloads.


Implications

  • Scale‑out serving – The Rubin‑optimized stack shows that a single NVL72 rack can replace dozens of older racks for high‑throughput LLM serving, dramatically reducing hardware footprint and power consumption.
  • Latency‑critical applications – The TP‑8 path demonstrates sub‑2 k tokens / s per‑user rates, enabling responsive chat or tool‑calling interfaces at scale.
  • Memory‑centric routing – Rack‑wide KV and Engram pools, combined with Dynamo’s workload‑aware routing, remove the need for per‑rank session pinning, simplifying orchestration and improving utilization.
  • Future hardware – The optimizations rely heavily on Rubin’s dual‑domain memory and SM107 MMAs; similar gains are expected on any future GPU that exposes comparable memory locality and mixed‑precision tensor cores.

Acknowledgements

We thank DeepSeek for open‑sourcing DeepSeek‑V4.1‑Flash, NVIDIA for early access to Vera Rubin NVL72, and the Mooncake team for the KV store. Special thanks to Hanjie Qiu, Lisa Wang, Jie Hao, Yaoyao Ding, Pei Li, Xin Li, and Pen Chung Li for their collaboration on the kernel work, and to SemiAnalysis for the AgentX benchmark.


Footnotes

  1. DEP2, DEP16, and DEP32 denote data‑parallel attention plus expert parallelism over 2, 16, and 32 GPUs respectively.

Sources