vLLM AgentX release: Optimizing real‑world agentic serving

TL;DR

vLLM introduced a coordinated stack of KV‑cache, parallelism, and scheduling improvements that boost agentic serving to 130K total tokens per GPU‑second and achieve a 14.6×–106× serving‑cost advantage over Opus 5 API pricing.

Agentic workload characteristics

Median session length: 43 turns.
Median input context: 142 K tokens; median output: 444 tokens.
Prefix‑cache hit rate: >96%.
44% of sessions contain subagents, with a median of four sub‑agent rollouts.

These numbers come from the SemiAnalysis AgentX benchmark, which captures real‑world coding‑assistant traces. Each turn appends the latest tool result to the accumulated context, so inputs grow while only a short new prefill is added, and most of the request is a previously seen prefix.

Core challenges for serving agents

Prefix cache pressure – Replaying the full conversation each turn forces frequent KV‑cache offloading across GPUs and replicas.
Execution efficiency – Long contexts and tight latency SLOs require higher per‑token throughput and lower decode latency.
P/D ratio selection – Varying context lengths and cache‑hit rates make it difficult to pick the optimal prefill‑to‑decode (P/D) balance, especially under concurrency.

Full‑stack optimization approach

vLLM addresses the three challenges across three planes: data, execution, and control.

Data plane – Warm, close‑to‑compute KV caches

Hybrid KV‑cache manager

Uses a uniform memory page as the allocation unit for all attention types (full, sliding‑window, linear).
Maintains a single shared block pool, allowing dynamic reallocation based on concurrency and prefix‑reuse patterns.
Replaces fragmented per‑type allocations (e.g., DeepSeek V4’s 92 tensors) with a packed layout, reducing padding and P/D transfer overhead by ~10% when FP4 indexing is enabled.

Hierarchical KV‑cache offloading

Integrates Mooncake Store as a distributed KV‑cache pool with CPU‑memory and disk tiers.
Supports model‑parity for sparse, compressed, and linear attention.
Implements two retention policies to keep high‑hit‑rate prefixes:
1. Interval‑based retention – Saves prompt‑end caches at each turn.
2. Marconi‑style selective retention – Saves a checkpoint when a prefix is observed a second time.
Optimizations (PR #46188, #45444, #45659, #47317) cut CPU lookup cost and move work off the scheduler’s critical path.

Execution plane – Faster token generation

Model‑specific parallelism

Kimi K3 – Uses decode context parallelism (DCP) instead of tensor parallelism. DCP shards KV along the sequence dimension, lowering decode latency and increasing KV capacity. Symmetric‑memory buffers fuse query gather and partial‑output reduction, cutting per‑layer latency by ~13%.
DeepSeek V4 – Prefill context parallelism (PCP) outperforms TP for long prompts (2.65× speedup on a 32 K prompt). Data‑plus‑expert parallelism (DEP) is the default for mixed workloads because DCP incurs excessive communication.

Mixed‑traffic scheduling

Head‑of‑line blocking mitigation – --long-prefill-token-threshold caps tokens per step (e.g., 512) so short cached turns can interleave with long prefills, improving total tokens per GPU‑second by up to 93% and P90 interactivity by ~2.3×.
DEP prefill cadence alignment – --prefill-schedule-interval co‑alesces prefills onto the same engine steps across ranks, freeing intermediate steps for pure decode and reducing lock‑step stalls.

Control plane – Optimal P/D disaggregation

Phase 1: Saturation profiling – Benchmark prefill‑only and decode‑only configurations across parallelism strategies and GPU counts to obtain max request‑per‑second limits.
Phase 2: P/D sweep – Combine the best prefill and decode configurations, vary the P/D ratio, and measure latency‑cost trade‑offs for the full deployment.

Open‑source kernel contributions

MiniMax M3 – CuteDSL long‑context indexer (+3%–31% latency), MSA top‑k path (up to 4× worst‑case speedup), speculative‑verification path (+20% decode throughput).
Kimi K3 – GEMM & reduce‑scatter fusion, latent‑tail MoE fusion (≈5% latency reduction).
DeepSeek V4 – MXFP4 MoE and HCA compression improvements, multi‑stream C4A, cluster‑based top‑k.

All patches are publicly available in the vLLM repository.

Performance results on AgentX

Model	GPUs / concurrency	Total tokens per GPU‑second (TPGS) @ P90 > 50 tok/s	P90 interactivity
DeepSeek V4 Pro 1.6T	12 GB300 GPUs / 256	83 K	58.3 tok/s
MiniMax M3 428B	2 B300 GPUs / 24	70 K	74.2 tok/s
Kimi K3 2.8T	16 GB300 GPUs / 48	11.8 K	62.7 tok/s

Cost comparison against Opus 5 (conservative cache‑hit assumptions):

Model	GPU TCO / hour	Equivalent Opus 5 cost / hour	Cost advantage
DeepSeek V4 Pro	$27.72	$2,926	106×
MiniMax M3	$4.52	$384	85×
Kimi K3	$36.96	$538	14.6×

The advantage stems from >96% prefix‑cache reuse, turning cached tokens into cheap compute across all three models.

Lessons learned (the “bitter” ones)

Pipeline parallelism works for cold, long prefills but adds bubbles for warm, prefix‑heavy turns; it should not be the default for agentic traffic.
DCP does not generalize to models with complex sparse attention stacks (e.g., DeepSeek V4); parallelism must match model architecture.
Load‑balance alone is insufficient – Session‑aware sticky routing preserves cache locality and outperforms aggressive queue‑depth balancing for short inter‑turn delays.

Future roadmap

Control‑plane routing – Separate first‑turn (long fresh prefills) from subsequent turns (high cache reuse) to avoid head‑of‑line blocking.
Agent hints API – Accept metadata about session structure, tool‑call latency, and branching points to guide scheduling and cache eviction.
Programmable KV cache – Expose interfaces for custom prefetch, eviction, and soft‑pinning policies.
Session‑based KV migration – Prefetch KV state to the next worker during inter‑turn idle time to hide transfer latency.

Acknowledgments

The work was led by Inferact with extensive contributions from the vLLM community, SemiAnalysis (benchmark design and infrastructure), and hardware partners NVIDIA and AMD.