Serving Agentic Workloads at Scale with vLLM x Mooncake

Agentic workloads create large reusable prefixes

Agentic workloads produce long contexts with high reuse, making prefix caching critical. The Codex/SWE‑bench Pro traces show that by turn 30 the context length reaches roughly 80 K tokens and can exceed 180 K tokens, while each turn adds only a few hundred to a few thousand new tokens. Across 610 traces with a median of 33 turns per trace the dataset exhibits:

• 94.2 % cache hit rate
• 131:1 input‑to‑output token ratio
• average context growth of about 2 242 tokens per turn
• median context growth from 12 K to 80 K tokens per trace
• inter‑turn delays ranging from 5.2 s median to 81.4 s P99 These numbers confirm that most of the input is reusable prefix, so caching it eliminates redundant prefills.

Distributed KV cache pool with Mooncake Store solves prefix sharing

Integrating Mooncake Store gives a shared KV cache that increases hit rates and enables cross‑instance reuse. Mooncake is an open‑source library for KV cache transfer and distributed storage. vLLM already uses Mooncake for prefill‑decode disaggregation via the MooncakeConnector. The new integration builds a distributed KV cache pool where multiple vLLM instances embed Mooncake clients and connect to a cluster‑wide Mooncake Store master. The master manages KV‑block metadata, service discovery, and client health; workers transfer KV blocks between GPU HBM and the distributed DRAM/SSD pool over RDMA. On the scheduler side, vLLM hashes prompt token blocks, queries the Mooncake master for matching KV cache blocks, and uses the result to guide scheduling. On the worker side, each GPU worker runs a Mooncake client, registers GPU KV cache memory as RDMA buffers, and moves data via background I/O threads.

Design highlights: SM‑free GPUDirect RDMA, asynchronous transfers, MultiConnector

The implementation uses GPUDirect RDMA for zero‑copy, runs transfers on background I/O threads, and works with PD disaggregation via MultiConnector. KV block movement leverages GPUDirect RDMA, avoiding staging buffers and SM consumption, and can pool multiple RNICs for higher bandwidth. All RDMA operations run on a dedicated background I/O thread, making the transfer path fully asynchronous and preventing stalls on the main CPU path. The MultiConnector interface chains sub‑connectors; the prefill instance stores KV blocks in the distributed pool and can retrieve them for cache hits, while the decode instance writes blocks into the pool for visibility to prefill.

Performance results on Codex traces and scaling

On realistic agentic traces the cache hit rate jumps from 1.7 % to 92.2 %, yielding 3.8× throughput, 46× lower TTFT, 8.6× lower latency; scaling to 60 GB200 GPUs retains >95 % hit rate and near‑linear throughput growth. In the Codex agentic trace experiment (1P1D, 12 GB200 GPUs) the distributed KV cache pool increased throughput by 3.8×, reduced P50 TTFT by 46×, and cut end‑to‑end latency by 8.6×. These gains stem from the cache hit rate rising from 1.7 % (only the system prompt cached) to 92.2 % (nearly the entire prefix cached). For scaling, a synthetic dataset derived from the Codex workload was used with round‑robin routing across nodes. The system maintained a cache hit rate above 95 % at all scales and scaled throughput nearly linearly from 12 to 60 GB200 GPUs, demonstrating that the distributed pool prevents the cross‑instance misses that would otherwise degrade performance.

Future work

Planned extensions include distributed disk offloading, hybrid model support, cache‑aware routing, and datapath optimizations. Future work will extend the storage hierarchy to NVMe SSDs and distributed file systems for larger cache capacity. Support for hybrid models with mixed attention mechanisms will be added, allowing different caching strategies per layer. Cache‑aware routing will co‑design the request router with the KV cache pool to direct turns to instances holding the relevant prefix, maximizing local hits before falling back to the distributed pool. Datapath optimizations will explore NVIDIA multi‑node NVLink and DualPath‑like simultaneous KV loading from prefill and decode instances to boost aggregate bandwidth.

Acknowledgements

The vLLM Mooncake Store integration was inspired by prior work in vLLM‑Ascend. Thanks go to Chao Lei from Ant Group for the initial implementation and Zijing Liu from Inferact for the agentic trace and analysis. Additional gratitude is extended to Jiahao Lu, Zuoyuan Zhang, Zihan Tang, Ke Yang from Approaching.AI; Pengbo Zhao, Fuqiao Duan, Tianyu Xu from Huawei; Tianchen Ding, Xuchun Shang, Xingrui Yi, Teng Ma from Alibaba Cloud Computing; Yunxiao Ning, Dejiang Zhu, Shoujian Zheng from Ant Group; and Feng Ren from 9#AISoft for technical feedback. The broader vLLM and Mooncake communities are acknowledged for their support, as is the Inferact team for close collaboration.