DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Optimization

vLLM and Inferact have optimized the serving of DeepSeek-V4.1-Flash, achieving a 5.3× throughput improvement under a 150 TPS constraint and a 1.9× speedup at low concurrency. These gains are driven by the implementation of SWA bounded replay and the integration of high-performance kernels including MegaAttention, Mega-mHC, and Mega-Gate.

SWA Bounded Replay for Prefill Efficiency

SWA bounded replay reduces prefill computation and memory overhead by trading exactness for efficiency, with negligible impact on model quality as verified on GSM8K and GPQA benchmarks.

Reducing KV Cache and Prefill Costs

DeepSeek-V4.1-Flash utilizes two KV caches: a compressed global KV (FP4, ~890 bytes per token) and an uncompressed sliding-window (SWA) KV (FP8, covering the last 128 positions across 40 layers). To avoid the high storage cost of SWA KV in prefix caching and the redundant computation of layers 21–39 during prefill, vLLM implements SWA bounded replay. This technique reruns only the last 128 tokens and clips the SWA window at the replay start.

  • Encoder-side replay: vLLM caches only the global KV. On a prefix hit of length H, it reruns tokens [H − 128, H) to rebuild the SWA KV.
  • Decoder-side replay: In the causal encoder-decoder (CED) architecture, vLLM runs layer 20 for every token to produce global KV, but runs layers 21–39 only on the last 128 tokens of each request, skipping nearly half the model for long prompts.

Accelerating Trimmed Layers with CUDA Graphs

Because layers 21–39 perform minimal GPU work after trimming, kernel launch overhead can dominate. vLLM uses a breakable PIECEWISE graph to capture layers 0–20 on the full batch and layers 21–39 separately on the trimmed batch. This optimization reduces prefill computation time by 30–40%, particularly for short prompts where launch overhead is the primary bottleneck.

Kernel-Level Optimizations

vLLM integrated several open-source kernels from DeepSeek to eliminate memory round trips and fuse operations.

Specialized Mega-Kernels

  • Mega-mHC: Fuses the mHC chain (post step, delayed-pre step, and RMSNorm) into one kernel, performing 1.14–1.51× faster than previous TileLang versions on NVIDIA GB200.
  • Mega-Gate: Fuses the MoE router (gate GEMM, expert scoring, bias, and top-k selection), resulting in 1.18–1.31× kernel speedups at medium batch sizes.
  • MegaAttention: Performs query RoPE, sparse attention, inverse RoPE, and FP8 casting in a single launch. It utilizes a new NVFP4 compressed KV format that is 45% smaller than FP8, improving kernel efficiency by 1.45×.

Latency and Memory Optimizations

  • mHC Multistream Overlap: Computes the next mHC block's coefficients on a side CUDA stream in parallel with attention and FFN, reducing latency by ~4% in TP4 low-latency scenarios.
  • Sparse MQA Logits: Scores only candidate positions rather than the entire context. This is 1.2× faster at 8K tokens and 14–23× faster at 512K tokens on NVIDIA GB300, improving prefill speed by 2× at 1M context.
  • Low-latency fused WO-A kernel: Fuses inverse RoPE, FP8 quantization, WO-A batch GEMM, and MXFP8 requantization for small decode batches on Blackwell, reducing inter-token latency by 6–7%.
  • Engram Optimizations: Implements asynchronous prefetching for CPU-offloaded Engram lookups and supports transparent huge pages (THP), which can deliver up to 10× faster lookup kernels for prefills.

Agentic Serving Performance

Using the SemiAnalysis AgentX benchmark, vLLM demonstrated significant improvements over its day-0 implementation.

Throughput and Latency Trade-offs

  • Low-Latency Serving: Utilizing TP4 with FlashInfer attention, vLLM achieved a 1.9× speedup. FlashInfer was found to be faster than MegaAttention in this specific low-concurrency, memory-bandwidth bound scenario.
  • High-Throughput Serving: Utilizing DEP2 with DP attention and MegaAttention, vLLM achieved a 5× throughput improvement. DP attention avoids KV cache duplication across GPUs, and NVFP4 compression further increases per-GPU concurrency.

TTFT Improvements

At approximately 100K throughput, Time to First Token (TTFT) decreased by nearly 70%. This reduction is the combined result of SWA bounded replay (cutting prefill computation in half), CUDA graphs (minimizing CPU launch overhead), and general kernel accelerations.

Sources