ovg-project/kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

What it solves

LLM serving engines typically reserve a fixed amount of GPU memory for the KV cache at startup, which leads to rigid memory partitioning and poor GPU utilization when running multiple models or dynamic workloads. kvcached solves this by enabling elastic, demand-driven KV cache allocation, allowing multiple LLMs to share GPU memory more flexibly.

How it works

It introduces an OS-style virtual memory abstraction to LLM systems. By decoupling GPU virtual addressing from physical memory allocation, serving engines can reserve virtual memory initially and only back it with physical GPU memory when the cache is actively used. This allows for on-demand allocation and reclamation of memory to match the live load.

Who it’s for

Developers and operators deploying LLMs in resource-constrained environments who need to run multiple models on shared GPUs, implement serverless LLM deployments, or colocate LLM inference with other GPU workloads like training or fine-tuning.

Highlights

  • Elastic KV Cache: Dynamically allocates and reclaims memory based on actual demand.
  • GPU Virtual Memory: Decouples logical KV cache from physical memory via runtime mapping.
  • Engine Integration: Works as a plugin for mainstream serving engines like vLLM and SGLang.
  • Prefix Caching: Supports automatic prefix caching (APC) and RadixCache for cross-request reuse.
  • Memory Control: Includes a CLI to enforce memory limits.
  • Multi-Model Support: Enables concurrent deployment of multiple LLMs without rigid partitioning.

Related

  • Project
    LMCache/LMCache

    A KV cache management layer for LLM inference that reduces TTFT and improves throughput by making KV caches persistent and reusable across requests and engines.

  • Project
    ModelEngine-Group/unified-cache-management

    A cache management framework for LLMs that reduces inference latency and GPU memory usage by persisting KV caches and implementing sparse attention retrieval.

  • Project
    cchenax/px-kvstore

    A lightweight, high-performance in-memory KV store optimized for deterministic LLM caching and fast local AI inference experimentation.

  • Project
    microsoft/vattention

    A memory manager for LLM KV-caches that uses CUDA virtual memory APIs to enable dynamic allocation while maintaining virtual contiguity for unmodified attention kernels.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.