ovg-project/kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

What it solves

LLM serving engines typically require static GPU memory reservation at startup, which leads to inefficient GPU utilization and high costs when workloads are dynamic or mixed. kvcached addresses the "GPU cost crisis" by allowing multiple LLMs to share a single GPU's memory elastically.

How it works

It introduces a virtual memory abstraction that decouples logical KV caches from physical GPU memory allocation. This allows serving engines to reserve virtual memory upfront and only back it with physical memory when the cache is actively used. It integrates as a plugin with mainstream engines like vLLM and SGLang and includes features like a frontend router, sleep mode for idle models, and automatic prefix caching.

Who it’s for

  • LLM Service Providers: Looking to reduce costs through better GPU utilization and multi-model serving.
  • Developers building Compound AI systems: Needing to allocate memory dynamically across specialized models in a pipeline.
  • Cloud/Infrastructure Engineers: Seeking to support serverless LLM deployments where models spin up and down on demand.
  • Users with limited hardware: Needing to colocate LLM inference with other GPU workloads like training or vision models.

Highlights

  • Elastic KV cache: Dynamically allocates and reclaims memory based on live load.
  • Engine Support: Compatible with vLLM and SGLang, supporting various attention types (MHA, GQA, MLA).
  • Prefix Caching: Supports automatic prefix caching (APC) and RadixCache for cross-request reuse.
  • Performance Gains: Can achieve a 2-28x reduction in Time To First Token (TTFT) compared to current static serving engines.