vLLM x Novita AI: PegaFlow for Production-Grade External KV Cache

TL;DR

In collaboration with Novita AI, PegaFlow integrates with vLLM as an external KV cache service implemented as a standalone Rust process, moving KV cache lifetime out of the vLLM worker process, pooling cache across local instances and remote nodes, and combining pinned host memory, RDMA-accessible remote memory, and SSD into a three-level cache hierarchy.

Why KV cache needs a process boundary

KV cache is one of the most expensive runtime assets in production LLM serving, often occupying hundreds of GiB per host, taking time to allocate and warm, and outliving the request pattern that created it. In a conventional in-process design, the KV cache is tightly coupled to the inference engine process, causing pain during engine crashes, rolling upgrades, and model switches because the host KV pool disappears with the engine restart. PegaFlow addresses this by moving the KV cache runtime into a standalone daemon on each machine, owning the host KV pool, SSD cache, topology metadata, RDMA resources, indexing state, and background tasks, while vLLM workers connect through CUDA IPC and gRPC. This design allows one cache server to serve multiple engines and multiple models on the same host, providing namespace isolation while sharing the same memory pool, SSD capacity, and cross-node network bandwidth, resulting in cleaner failure domains.

Faster restarts with external cache ownership

To isolate the startup-path impact of host KV pool ownership, an 8 x RTX 5090 setup running Qwen3-8B with TP8 was measured using dummy weights and eager mode. With an embedded KV cache design, vLLM took 71.4 seconds to reach ready state. With PegaFlow, after the standalone server was ready, vLLM reached ready state in 33.2 seconds, a 2.15x faster startup path driven by decoupling long-lived host cache allocation from the inference process lifecycle.

Rust data path and tail-latency stability

Moving KV cache into an external process was primarily motivated by lifecycle management, sharing, and CPU resource isolation. Implementing that process in Rust avoids Python interpreter overhead, GIL contention, and stop-the-world garbage collection, which matters because a production cache service runs background tasks such as statistics collection, index uploads, prefetching, health checks, metrics reporting, eviction, and SSD cache management. In PegaFlow, those tasks run in the same standalone Rust service without sharing an interpreter runtime with vLLM, giving the system more room to run control-plane and maintenance work without disturbing the data-plane path.

Pooling cache across instances and nodes

In production deployments, the same logical KV content is often replicated many times because process, model, or node boundaries make caches invisible to each other. PegaFlow turns these isolated cache fragments into a shared cache pool. On a single host, all local instances connect to the same PegaFlow server and share one CPU KV pool. Across hosts, a PegaFlow MetaServer maintains an approximate global index, allowing nodes to fetch remote KV blocks through one-sided RDMA READs with zero CPU involvement on the remote side after connection setup.

Single-node multi-instance sharing

We evaluated eight Qwen3-8B instances on one host with the same 500 GiB cache budget.

Setup	Cache layout	Throughput	Mean TTFT	Request hit rate
PegaFlow	500 GiB shared pool	11.97 req/s	5.26 s	52.35%
In-process	8 x 62.5 GiB isolated pools	7.68 req/s	8.22 s	11.77%
Throughput improved by 56%, mean TTFT dropped by 36%, and request hit rate increased by 4.4x.

MLA logical KV deduplication

We also evaluated DeepSeek-V3.2 MLA with TP8 under a 500 GiB cache budget.

Setup	Cache layout	Throughput	Mean TTFT	Request hit rate
PegaFlow	Logical KV stored once	1.81 req/s	35.66 s	97.23%
In-process	KV stored per TP rank	1.05 req/s	60.88 s	65.18%
Throughput improved by 72%, mean TTFT dropped by 41%, and request hit rate approached the practical upper bound for the trace.

Cross-node RDMA sharing

In an internal production inference cluster equipped with 8 x 400 Gbps RDMA NICs per node, we sampled thousands of recent online remote reads. For large prefix pulls of at least 1 GiB, PegaFlow sustained 194 GB/s average effective throughput under production traffic, with 250 GB/s P99 and a peak of 261.6 GB/s. At this transfer rate, a 24 GiB KV cache segment can be pulled from a remote node in roughly 100 ms, replacing a prefill computation that would otherwise consume seconds of GPU time.

Three-level cache hierarchy

Pooling makes cache capacity more useful, but host memory is still finite. PegaFlow addresses this with a three-level cache hierarchy: hot local blocks stay in pinned DRAM, remote hits can be fetched over RDMA, and colder reusable blocks can spill to local SSD.

Level	Medium	Access path	Typical role
L1	Local pinned DRAM	Local memory	Fast local KV reuse
L2	Remote DRAM	RDMA READ	Cross-node cache sharing
L3	Local SSD	io_uring	Large-capacity spillover
The SSD cache is implemented in Rust on top of io_uring. In internal tests, a single SSD delivered roughly 6.9 GB/s peak read throughput, with PegaFlow keeping online steady-state throughput around 6.5-6.6 GB/s per disk.
With RAID0 across multiple disks, total throughput scales approximately linearly.
For scan-heavy workloads or hosts with smaller cache budgets, PegaFlow can enable a TinyLFU admission policy, which admits blocks only when they are likely to be reused, protecting the cache from one-time traffic.
TinyLFU is disabled by default because the best admission policy depends on workload shape.

Measuring distance from the theoretical hit-rate ceiling

Online hit rate alone can be misleading. PegaFlow estimates the theoretical hit-rate upper bound online using HyperLogLog:

r* = (N - U) / N

Here, N is the total number of block requests in a window, and U is the number of first-seen unique blocks. HyperLogLog keeps this estimate inexpensive: a 24-hour window uses less than 1 MiB of memory with roughly 0.8% error. PegaFlow exports rolling HLL windows, with defaults of 15 minutes, 1 hour, and 24 hours. By placing measured hit rate and theoretical upper bound on the same dashboard, operators can distinguish three cases:

• The cache is already close to the workload ceiling, so adding capacity may not help much.
• The measured hit rate is far below the ceiling, suggesting room for better capacity, admission, prefetching, or cross-node discovery.
• The theoretical ceiling itself is low, indicating that the workload has limited reuse and the bottleneck is not primarily the cache implementation.

Integrating with vLLM through the external connector

External KV cache systems often require invasive changes to the scheduler, block manager, or attention kernels. PegaFlow instead integrates through vLLM's external KV connector mechanism. The connector is configured through kv_transfer_config, and external packages can be loaded dynamically with kv_connector_module_path. This lets PegaFlow take over key KV cache operations at runtime without modifying vLLM source code or carrying a long-lived fork. From vLLM's perspective, PegaFlow is not a replacement for the serving engine; it is an external cache backend attached through the KV transfer interface, while vLLM continues to handle scheduling, model execution, batching, and the OpenAI-compatible serving path. This boundary is useful for both projects: PegaFlow can iterate on its Rust data plane, SSD cache, RDMA path, indexing, and connector logic independently, while vLLM can continue improving the core serving engine while exposing a stable connector contract for external cache systems.

Quick start

Install the package for your CUDA version:

uv pip install pegaflow-llm        # CUDA 12
uv pip install pegaflow-llm-cu13   # CUDA 13

Start a single-node PegaFlow server with pinned host memory and SSD cache:

pegaflow-server \
  --pool-size 30gb \
  --ssd-cache-path <ssd-cache-file-path> \
  --ssd-cache-capacity 512gb

For online deployments, we recommend adding --use-hugepages. Huge pages should be reserved in advance. For multi-node deployments, start the MetaServer first, then start a PegaFlow server on each node with RDMA configuration. When P2P is enabled, each PegaFlow server's --addr must be a routable IP address, not 0.0.0.0 or 127.0.0.1, because other nodes use it for the gRPC handshake and block queries.

pegaflow-metaserver --addr 0.0.0.0:50056

pegaflow-server \
  --addr this-node:50055 \
  --pool-size 30gb \
  --ssd-cache-path <ssd-cache-file-path> \
  --nics mlx5_0 mlx5_1 \
  --metaserver-addr http://metaserver-host:50056

Connect vLLM without modifying vLLM source code. The examples in this post use vllm>=0.20.0:

vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "PegaKVConnector",
    "kv_role": "kv_both",
    "kv_connector_module_path": "pegaflow.connector"
  }'

The PEGAFLOW_HOST and PEGAFLOW_PORT environment variables point the connector to the PegaFlow service. By default, they are http://127.0.0.1 and 50055.

Public reference benchmark

The PegaFlow repository also includes a public KV cache benchmark on H800 with Llama-3.1-8B, using 8 prompts, 10K-token prefill, 1-token decode, and 4.0 req/s. In that setup, the warm cache path reduces mean TTFT from 572.5 ms to 61.5 ms, with P99 TTFT dropping from 1113.7 ms to 77.0 ms.

Try PegaFlow

PegaFlow is available on GitHub: novitalabs/pegaflow. The repository includes installation instructions, server configuration, P2P RDMA setup, metrics documentation, and vLLM connector examples.

Acknowledgements

We would like to thank the Novita AI team for building and productionizing PegaFlow, and the vLLM maintainers and broader vLLM community for the discussions, reviews, and connector infrastructure that made this integration possible.