Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72

TL;DR: On one NVIDIA GB300 NVL72 rack, vLLM with P/D disaggregation and Wide-EP serves DeepSeek-V4.1-Flash on the SemiAnalysis AgentX benchmark at 954K tokens/s per GPU at a P90 of 196 tokens/s per user — 7.9x vLLM's earlier single-node result — and its decode throughput lands within 6% of DeepInfer, DeepSeek's own inference engine for the model. The result matters because it shows a general-purpose serving engine can match a model-specific one while also scaling online across a full NVL72 rack.

What vLLM announced

vLLM published a performance study of serving DeepSeek-V4.1-Flash at rack scale. The post is a follow-up to vLLM's earlier single-node work on the same model and describes the engineering that produced the scale-up.

The headline result: on one NVIDIA GB300 NVL72 rack, vLLM with P/D (prefill/decode) disaggregation and Wide-EP serves DeepSeek-V4.1-Flash at 954K tokens/s per GPU at a P90 of 196 tokens/s per user. That is 7.9x vLLM's aggregated single-node result from its first post on the model. The cost is 38.96 US dollars per GPU-hour at DeepSeek's off-peak prices. Throughput counts cached input, uncached input and output tokens.

A specialized decode path reaches 826 tokens/s for a single user.

Against DeepInfer, DeepSeek's own engine, vLLM's decode throughput is within 6% across batch sizes 8 to 256, and its prefill tracks DeepInfer's published curve.

The post opens with its own summary of the lessons:

  • A general-purpose engine can match a model-specific one
  • Understand the model before optimizing
  • Small DEP instances work best for prefill
  • Wide-EP decode scales well with EPLB
  • CPU overhead becomes critical at NVL72 scale
  • A specialized decode path unlocks low latency

Headline numbers at a glance

Metric Result
Throughput on SemiAnalysis AgentX, GB300 NVL72 954K tokens/s per GPU
P90 latency per user 196 tokens/s per user
Gain over vLLM's earlier aggregated single-node result 7.9x
Cost 38.96 US dollars per GPU-hour at DeepSeek's off-peak prices
Single-user latency on the specialized decode path 826 tokens/s
Decode throughput vs. DeepInfer (batch sizes 8 to 256) Within 6%

How vLLM compares with DeepInfer

DeepInfer is DeepSeek's own inference engine for DeepSeek-V4.1-Flash. It was released in the same week as vLLM's post, as an offline engine with published prefill and decode curves. vLLM treats it as the most relevant reference point for this model and compares against it before presenting its own optimizations.

Decode comparison

Under DeepSeek's decode configuration — 131K context, nextn=3 — vLLM on GB200 achieves throughput within 6% of DeepInfer's published results across batch sizes 8 to 256. vLLM read the DeepInfer values off the charts in its README.

Prefill comparison

vLLM is explicit that its prefill comparison is not a like-for-like engine benchmark. DeepInfer's published prefill curve uses EP4DP4 with 32 redundant experts and a 140-token bounded replay, on hardware the chart does not state, and vLLM read the values off the chart. vLLM's own run uses DEP2 on GB300 with no redundant experts, a 128-token replay and its own prompts.

Because the setups differ, vLLM states it does not read the gap between the two curves as an engine-to-engine speedup. What the comparison shows is that vLLM's DEP2 prefill tracks DeepSeek's own engine across the range DeepSeek published.

Why the two engines land close

The post attributes the similarity to shared lineage in both directions. DeepInfer was originally built on vLLM, and vLLM in turn has continuously integrated kernels from DeepSeek's libraries — including MegaAttn, MegaMoE and Mega-mHC — and has extensively optimized the rest of the stack around them.

The distinction vLLM draws: DeepInfer targets a single model in offline mode, while the same vLLM build also serves that model online with P/D disaggregation and Wide-EP across a full NVL72 rack.

Model characteristics that shaped the optimization strategy

Before tuning, vLLM quantified DeepSeek-V4.1-Flash's architectural properties. Three findings drove the rest of the work:

  • Small KV cache footprint. Only 890 bytes per token for the global KV cache, thanks to cross-layer KV sharing and FP4 compression, per the DeepSeek-V4.1-Flash Technical Report.
  • Lightweight attention. Layers 2, 8 and 14 use 2x compression for full-context indexing. With bounded replay, layer 20 writes global KV for all tokens but runs indexing and attention only on the last 128 tokens; layers 21 through 39 process only those tokens.
  • Decode is MoE-bound even in a long context. With DEP4 decoding at 128K context and 64 requests per rank, MoE takes 62% of the target step, while attention and the indexer together take only 13%.

vLLM also measured target-model forward time at DEP32, with 16 requests per rank, 6 tokens per request (one token plus five DSpark draft tokens), and every request at the same context length, on GB300. Long context barely moves decode latency: going from 128K to 855K tokens — 6.7x the context — adds only 0.6 ms (+7%) to the target model decode latency.

Context per request 64K 128K 256K 512K 855K
Target forward (ms) 8.86 8.93 9.03 9.25 9.53
vs. 128K -0.8% – +1.1% +3.6% +6.7%

These characteristics led to five strategy decisions, in vLLM's framing:

  • Prefill should run at a small DEP size. Even a DEP2 instance on B300 or GB300 can hold both the model weights and the KV cache for many concurrent sessions, and a smaller world size greatly simplifies DP/EP load balancing.
  • KV capacity rarely limits batch size. The KV cache fits comfortably in HBM, so each operation can be pushed toward its compute or memory bandwidth limits instead of working around KV capacity. This also keeps sweeps consistent and low-noise with no preemption or KV cache pressure.
  • Wide-EP is effective for decode. MegaMoE decode is bound by weight reads, and Wide-EP directly cuts this dominating MoE cost.
  • Decode DP load balancing is simpler. Decode step time varies little with context length for this model, so DP load balancing can be simplified to least-load by request count per rank.
  • Synthetic workloads remain representative for decode. Using average input and output lengths gives a good performance estimate without replaying the full agentic trace.

Prefill: DEP2 works best, plus three bottleneck fixes

Small DEP instances win for prefill

In an AgentX prefill-only run at concurrency 256, DEP4 gave only 83% of DEP2's per-GPU throughput (53.3K vs. 64.4K tokens/s). Mirror and lag replays with prefix caching off point to DP imbalance as the cause: when ranks fall out of step, DEP4 loses 13.6% of its throughput. vLLM calls DEP2 a simple default for this reason.

Prefill throughput per instance (tokens/s) mirror (balanced) lag (imbalanced) loss
DEP2 138.7K 130.5K 5.9%
DEP4 267.6K 231.1K 13.6%

vLLM notes the imbalance cost was measured on an earlier build, and that it is a property of DEP size rather than of the kernels.

Clean traces exposed context-length bottlenecks

On two GB300 GPUs, an 8K-token prefill takes 87.9 ms without prior context but 189.4 ms with 512K cached context. vLLM addressed three additional bottlenecks behind that trace:

  • Extending SWA bounded replay to layer 20 (PR #59894): +19.9% at 32K new tokens. SWA bounded replay already limited layers 21 through 39 to the last 128 tokens. Previously layer 20 still performed indexer computation across all new tokens. Now it writes KV for all new tokens but runs its remaining computation only on the 128-token replay batch. Gains increase from 10.1% at 4K to 23.1% at 64K.
  • 2 MB pages for shared Engram host tables (PR #62359): another +2.8% at 32K. Engram performs random reads from host tables shared across DP ranks. The previous /dev/shm mapping used 64 KB GPU pages, causing frequent GPU TLB misses. Moving the tables to host-NUMA cuMem allocations enables 2 MB GPU pages.
  • DeepSelect top-k in the prefill indexer (PR #62240): another +15.6% at 32K. The prefill indexer now uses DeepSelect to rank candidate blocks, matching the decode path. The benefit grows with the number of keys being ranked, reaching 15.2% at 64K.

Together, these changes improve new-token prefill throughput by 35.6% to 43.9% across 4K to 64K new tokens on top of a 128K cached prefix. At 32K new tokens, the combined gain is 42.5%. At 64K, throughput rises from 71.2K to 102.5K tokens/s per GPU (+43.9%).

Decode: Wide-EP scaling and EPLB expert balancing

Wide-EP with EPLB is an established practice, described in DeepSeek's inference system overview, and vLLM has supported both for a while. This section focuses on vLLM's new findings and empirical lessons.

Wide-EP scaling from DEP4 to DEP32

vLLM measured decode scaling from DEP4 to DEP32 with 64 requests per GPU at 128K context, with EPLB enabled. Target MegaMoE time drops from 13.4 ms to 6.0 ms as each rank holds fewer experts, while the remaining work stays around 10 ms. As a result, per-GPU throughput gains gradually diminish. At DEP32, the draft's communication overhead also starts to offset the benefits of wider parallelism. These measurements were taken on NVIDIA GB200.

EPLB results

vLLM configured EPLB to rearrange experts every 3,000 steps, based on the load of the previous 1,000 steps. EPLB raises balancedness from 0.35 to 0.94 for the target model and from 0.21 to 0.89 for the draft, recovering about 97% and 93% of the gap to uniform routing.

Two changes make this work:

  • PR #62009 introduces an NVLink-based symm_mem backend that rebalances all of DEP32 in under a second without stopping decode, fuses EPLB's replica mapping and load recording into the MegaMoE kernels (saving about 0.2 ms per step), and removes host synchronization from the decode loop.
  • PR #61395 supports EPLB with heterogeneous target and draft models. The DSpark draft has 128 routed experts with top-3 routing, against the target's 384 with top-6, so EPLB tracks expert placement and load separately for each model and rebalances both.

CPU overhead becomes critical at NVL72 scale

Wide-EP decode synchronizes all ranks at every layer, so one slow rank stalls the instance. Async scheduling hides CPU overhead well at a small scale, but more ranks make this harder in two ways: GPU steps get shorter, leaving less slack for CPU work, and it becomes more likely that one rank falls behind.

In practice, vLLM observed that an unoptimized DEP32 ran slower than DEP16 — 17.57 ms versus 13.84 ms per step — even though its GPU execution was 1.1 ms faster. The post lists every cause it found, ordered by effect. Because the effects come from different setups, each row carries its own.

Cause Fix PRs Effect Setup
KV read posted synchronously on the decode worker thread NIXL push with UCX scatter-gather; packed KV rows sent as one NIXL region NIXL push connector, #61121 141K to 440K tokens/s per GPU; P90 about 40 to 159 tokens/s per user GB300, 8P1D, concurrency 3,072
Gloo all-reduce latency grows linearly with the rank count Recursive-doubling CPU all-reduce #61767, building on #60267 DP sync 10.8 to 1.5 ms per step; step 16.10 to 13.15 ms DEP32 decode
Admission bursts: up to 140 requests admitted in one step Opt-in cap of 4 new KV loads per step #61725 460K to 516K tokens/s per GPU; P90 109 to 153 tokens/s per user GB300, 7P1D, concurrency 4,800, with two other CPU fixes
Admission work that grows with the prompt length Prompt as one int32 buffer; skip the null blocks of sliding-window block tables; chunked prompt copy kernel #59908, #62702, #61724, #62698 855K prompt: 38.8 to 6.0 ms of CPU per admission Engine microbenchmark

Low latency: a specialized decode path for small batches

vLLM extended low-latency kernel support for DeepSeek-V4.1-Flash through PRs #61868 and #61869, continuing work from its previous post on the Kimi K3 megakernel.

On real AgentX sessions, the specialized decode path reaches 826 P90 tokens/s for a single user on four B300 GPUs, 44% higher than the standard decode path (575). The measurement covers SemiAnalysis AgentX with TP4 on four B300 GPUs at concurrency 1, 8 and 16, before and after InferenceX PR #4332 switched to the specialized decode path.

How vLLM measured performance

vLLM breaks accurate measurement into two questions: what should a prefill or decode step cost, and how much additional overhead comes from DP and EP imbalance?

  • Prefill and decode step time. The vllm bench forward tool (PR #61339) measures model-forward time for an exact prefill or decode batch through the actual execute_model path across DP and EP ranks. This measures GPU time only and gives a clean trace. Separately, vLLM's DecodeBenchConnector measures engine-level end-to-end decode performance at a given concurrency. The gap between GPU-only and serving step times reveals unexpected overhead.
  • Prefill DP imbalance: mirror vs. lag replays. Agentic traces vary from run to run, so DP imbalance is hard to isolate. vLLM therefore replays identical requests on every DP rank. In a mirror run, all ranks receive their requests at the same time. In a lag run, rank r receives the same requests r x 5 seconds later, so ranks are out of step but do the same total work. The difference between the two runs is the cost of DP imbalance.
  • Decode EP imbalance: real vs. uniform routing. For each MoE layer, balancedness is defined as mean rank load divided by max rank load, per step and over EPLB's window. As the balanced reference, vLLM runs the same decode batch with tokens routed uniformly at random. That reference still has sampling noise: at DEP32 with 72 requests per GPU, uniform routing reaches about 0.96 balancedness for the target (top-6 routing) and 0.94 for the draft (top-3). The gap between real and uniform routing is the most that expert balancing can recover, and the EPLB run shows how much of that gap it actually recovers.

Terminology

DEP followed by a number means data-parallel attention plus expert parallelism over that number of GPUs. DEP2 is two GPUs, DEP4 is four, DEP32 is 32. Target forward is the main model's full forward pass at each decode step, which verifies the draft tokens and produces the output; draft forward is the DSpark draft model's full forward pass, which proposes the tokens for the next step.

Credits

vLLM thanks DeepSeek for open-sourcing and sharing DeepSeek-V4.1-Flash and its optimized kernels, which provided the foundation for the work, and for publishing DeepInfer and its performance results. It also thanks SemiAnalysis for the AgentX benchmark platform.

vLLM singles out Pei Li, Hanjie Qiu, Lisa Wang, Yaoyao Ding, Rohit Nagraj and Xin Li from NVIDIA for collaboration on performance optimization, support across vLLM and Dynamo, and help with AgentX submissions, and Canlin Guo from the vLLM community for prefill optimization. The work was led by Inferact.

Sources