vLLM Tiered KV Cache Offloading

TL;DR

vLLM added tiered KV cache offloading that keeps evicted key‑value (KV) data in host memory, filesystem, object storage, or peer nodes, eliminating costly recomputation, reducing latency, and increasing the effective serving capacity of LLM clusters.

Host‑Centric Design Guarantees Fast Memory Release and Consolidated I/O

The framework routes all KV data through host memory (CPU DRAM) before it reaches any secondary tier. Offloading copies KV chunks from the accelerator to the host via PCIe, frees accelerator memory immediately, and then asynchronously writes the data to secondary tiers (filesystem, object storage, or peers). Reloading reverses the flow: a secondary tier promotes a chunk back to host memory, after which the accelerator receives it. This just‑in‑time allocation pattern ensures accelerator memory is occupied only while actively needed.

Consolidating shards from multiple GPUs into a single shared host region reduces the number of I/O operations, improving storage and network throughput in multi‑accelerator setups.

The host region uses a canonical memory layout—each page stores one block of one layer, with all KV heads from across tensor‑parallel ranks gathered contiguously. Because the layout is configuration‑independent, nodes with different parallelism settings or attention backends can share KV data without conversion.

Routing all data through the host also makes secondary tiers simple to implement: they operate as a single process per vLLM instance, use standard CPU‑based libraries (POSIX I/O, S3 SDKs, RDMA verbs), and never touch accelerator memory.

Offload and Reload Mechanics Operate on Fixed‑Size Chunks

KV data is partitioned into chunks, each covering a group of tokens. By default a chunk maps to a single accelerator block; the blocks_per_chunk parameter can enlarge chunks to increase I/O granularity.

Offload path

  1. Asynchronously DMA KV chunks from accelerator to host.
  2. Free accelerator memory as soon as the host copy is complete.
  3. The tiering manager concurrently pushes the host copy to all configured secondary tiers.
  4. The host tier acts as an LRU/ARC cache; chunks stay in host memory until capacity is exceeded, then they are evicted only to secondary tiers.

Reload path

  1. Scheduler checks the host cache; a hit returns the chunk immediately.
  2. On a host miss, secondary tiers are queried in order; the first tier with the chunk serves it.
  3. The tier promotes the chunk back to host memory asynchronously; the scheduler receives a RETRY and re‑checks on the next cycle.
  4. Different chunks of the same request may be satisfied by different tiers (e.g., one from filesystem, another from a remote peer).

Supported Secondary Tiers

Filesystem Tier

  • Stores each KV chunk as a file on local or networked storage using content‑addressed naming.
  • Automatic sharing when multiple vLLM instances mount the same directory.
  • Non‑blocking lookups, atomic writes, separate read/write thread pools.
vllm serve Qwen/Qwen3.6-35B-A3B \
    --kv-transfer-config '{
        "kv_connector_extra_config": {
            "spec_name": "TieringOffloadingSpec",
            "cpu_bytes_to_use": 107374182400,
            "secondary_tiers": [{"type": "fs", "root_dir": "/mnt/kv-cache"}]
        }
    }'

Object‑Storage Tier

  • Persists chunks in S3‑compatible stores via NIXL, using the same content‑addressed scheme.
  • Provides a cost‑effective, network‑wide cache.
--kv-transfer-config '{
    "kv_connector_extra_config": {
        "spec_name": "TieringOffloadingSpec",
        "cpu_bytes_to_use": 107374182400,
        "secondary_tiers": [{
            "type": "obj",
            "bucket": "my-kv-cache",
            "endpoint_override": "http://minio:9000"
        }]
    }
}'

Peer‑to‑Peer (P2P) Tier

  • Enables cross‑instance KV sharing over the network using ZMQ for coordination and RDMA for bulk transfers.
  • All transfers are host‑to‑host; no accelerator memory is involved.
  • Orchestration (e.g., via llm‑d) decides which peer to pull from.
--kv-transfer-config '{
    "kv_connector_extra_config": {
        "spec_name": "TieringOffloadingSpec",
        "cpu_bytes_to_use": 107374182400,
        "secondary_tiers": [{"type": "p2p", "host": "10.0.0.1", "port": 5710}]
    }
}'

Key P2P use‑cases

  • Prefill/Decode disaggregation: a prefill node writes KV chunks to its host tier; a decode node pulls them via RDMA, overlapping computation and data movement.
  • Load balancing: overloaded instances can offload chunks to under‑utilized peers, improving overall throughput.

Hybrid Model Compatibility

The tiered offloading framework integrates with vLLM’s hybrid memory allocator, normalizing all KV formats (full attention, sliding‑window, MLA, Mamba, etc.) into a uniform byte‑buffer representation. Each chunk has a fixed byte size on the host regardless of layer type, allowing consistent offloading across heterogeneous architectures. Consequently:

  • Sliding‑window layers reload only the active window tokens.
  • State‑space layers (e.g., Mamba) offload and reload their internal state together with attention KV. Supported hybrid models include DeepSeek V4, GLM 5.3, Nemotron 3, among others.

Observability and Metrics

vLLM exposes Prometheus metrics at the standard /metrics endpoint, covering:

  • Host cache utilization (fill ratio).
  • Accelerator↔host transfer throughput.
  • Per‑tier lookup and transfer latencies.
  • Per‑tier hit rates. Secondary tiers can register custom counters, histograms, or gauges that are automatically exposed.

KV Events Enable Intelligent Orchestration

Whenever a chunk moves between tiers, the framework emits a structured KV event indicating the chunk key, source tier, destination tier, and locality (local vs. remote). Orchestration systems such as llm‑d and Dynamo consume these events to route requests to instances with the highest likelihood of a cache hit and to trigger P2P transfers, yielding higher throughput and lower latency than cache‑unaware scheduling.

Extending the System with New Secondary Tiers

A secondary tier implements four methods:

class SecondaryTierManager(ABC):
    def lookup(self, key, req_context) -> LookupResult: ...
    def submit_store(self, job_metadata: JobMetadata) -> None: ...
    def submit_load(self, job_metadata: JobMetadata) -> None: ...
    def get_finished_jobs(self) -> Iterable[JobResult]: ...

The manager receives a direct memoryview into the shared host region, allowing zero‑copy reads/writes. Eviction policies are managed independently per tier. An in‑memory reference implementation is available at vllm/v1/kv_offload/tiering/example/. Out‑of‑tree tiers can be loaded by specifying a module_path in the tier configuration.

Performance at Scale: Offloading Beats Re‑computation

Benchmarking with Qwen 3.6‑35B‑A3B on 2 × NVIDIA H100 (TP=2) shows:

  • Up to ~64 concurrent conversations: accelerator memory holds the entire working set; all caching strategies perform similarly.
  • 64–128 conversations: accelerator memory saturates; throughput collapses without offloading, while CPU‑offloading maintains performance.

  • 128 conversations: CPU cache also fills; storage‑backed offloading retains a high hit ratio and more than doubles throughput compared to full recompute.

The storage tier used a local NVMe filesystem. Although storage latency is higher than CPU memory, a cache hit from storage is still far cheaper than re‑prefilling the model.

Performance scaling chart

Full benchmark scripts and results are hosted at neuralmagic/fs-offload-experiments.

Acknowledgements

Thanks to Liran Schour, Chang Guo, Srinivas Krovvidi, Rotem Shavitt, Effi Ofer, Omer Paz, Kfir Toledo, Michal Malka, and the broader community for their contributions to the tiered KV cache offloading framework.

Sources