ovg-project/kvcached

kvcached – Elastic KV‑Cache for LLMs on Shared GPUs

What it does – kvcached is a library (and daemon) that sits between an LLM serving engine (e.g., SGLang or vLLM) and the GPU. It gives the engine a virtual‑memory‑like view of the KV‑cache, so the cache can be reserved lazily and backed by real GPU memory only when a request actually needs it. This makes GPU memory usage elastic: multiple models can share the same GPU, idle models can “sleep”, and the same GPU can also run other workloads such as training or vision models.

Why it matters – Current serving stacks allocate a fixed chunk of GPU memory for each model’s KV‑cache at start‑up. When traffic spikes, you must over‑provision GPUs; when traffic is low, most of that memory sits idle. kvcached’s on‑demand allocation reduces the time‑to‑first‑token (TTFT) by 2‑28× in multi‑model scenarios and can cut the number of GPUs you need, translating directly into cost savings.

Key features

• Elastic KV cache – allocate and reclaim cache blocks as load changes.
• GPU virtual memory abstraction – logical KV addresses are mapped to physical GPU pages at runtime.
• Memory limits & control – CLI and a revisioned API let you enforce per‑model caps.
• Frontend router & sleep mode – automatically route requests and put idle models to sleep.
• Prefix caching (APC / RadixCache) – reuse common prompt prefixes across requests while staying elastic.
• Supports major engines – works with SGLang (≥ v0.5.11) and vLLM (≥ v0.17) and a wide range of attention types (MHA, GQA, MLA, sliding‑window, hybrid).
• AMD / ROCm aware – automatically switches to a non‑contiguous layout on ROCm.

Typical use‑cases

Scenario	Benefit
Multi‑LLM serving	Several models share one GPU, eliminating static partitioning.
Serverless LLM	Models spin up only when a request arrives; cache is allocated on‑demand.
Compound AI pipelines	Retrieval → reasoning → summarisation stages can share memory, enabling complex pipelines on a single GPU.
GPU colocation	LLM inference can coexist with training, fine‑tuning, or vision jobs on the same hardware.

How to get started

1. Install the package (or Docker image) – pip install kvcached --no-build-isolation or pull the appropriate ghcr.io/ovg-project/kvcached‑sglang / …‑vllm image.
2. Enable it via environment variables:

   export ENABLE_KVCACHED=true
   export KVCACHED_AUTOPATCH=1
   

3. Launch your usual SGLang or vLLM server; kvcached will patch the engine and manage the KV‑cache automatically.
4. Use kvctl to list, monitor, or delete control files, and adjust limits with the provided CLI.

Performance snapshot – In a benchmark with three Llama‑3.1‑8B models on an A100‑80 GB, kvcached reduced TTFT by up to 28× compared with the baseline static‑allocation engines.

Documentation & community – Full architecture, API reference, and deployment guides are on the project’s DeepWiki page and the official homepage kvcached.org. A Slack channel, GitHub issues, and a growing list of adopters (Google, Red Hat, AMD, etc.) provide support.

License – Apache 2.0.