ModelEngine-Group/unified-cache-management

Persist and reuse KV Cache to speedup your LLM.

Unified Cache Manager (UCM)

What it is – UCM is a library that sits between LLM inference engines (e.g., Transformers, vLLM, etc.) and the storage used for their key‑value (KV) attention cache. It provides a single KV‑cache layer that can:

  • Reuse previously computed attention states across different inference requests (e.g., multi‑turn chats, multimodal pipelines).
  • Move cache data between fast accelerator memory and slower external storage so that the limited GPU/TPU memory can be extended.
  • Plug‑in different storage back‑ends (in‑memory pools, file‑system blobs, networked stores) and engine adapters without changing the application code.

Why it matters

LLM inference repeatedly recomputes the same attention context when the same prompt prefix appears in many requests. By keeping that KV cache alive beyond a single forward pass and by spilling it to external storage when needed, UCM can cut compute time and reduce memory pressure on accelerators—especially useful for long‑running agents, chat bots, or multimodal apps that revisit earlier context.

Core concepts

Component Role
Applications / inference engines Produce KV cache during a pre‑fill step and later consume it during decode.
Unified Cache Decides what cache to keep, where to keep it, and when to move it (e.g., prefill‑decode transfer, sparse‑attention loading).
Global KV Store Abstract storage layer; implementations can be pure RAM, disk‑based, or distributed.

Key features (as described in the README)

  • Pluggable adapters for multiple inference engines.
  • Pluggable storage back‑ends (memory pools, file systems, custom stores).
  • Cache lifecycle management – automatic reuse, eviction, and movement.
  • Support for advanced reuse patterns such as shared prefixes, multimodal context, and KV‑Bridge style chunk caching.
  • Observability tools to inspect cache reuse and runtime behavior.
  • Extensible developer guide for adding new engines, stores, or cache mechanisms.

Typical use cases

  1. Chat‑bot servers that keep the conversation history in KV cache so that each new turn only needs to compute the new token.
  2. Multimodal pipelines where image‑text embeddings share large common prefixes.
  3. Batch inference where many requests share the same system prompt or few-shot examples.
  4. Resource‑constrained deployments that need to spill cache to SSD or network storage to fit larger contexts on limited GPU memory.

Getting started (quick steps from the README)

  1. Check compatibility – consult the support matrix to ensure your engine, model, and hardware are supported.
  2. Follow the Quickstart – install the appropriate artifact (pip package or container image), configure a storage backend, and launch an engine with the --ucm flag (or equivalent).
  3. Validate – use the observability guide to watch cache reuse metrics and confirm that cache movement is happening as expected.

All detailed commands, Docker image tags, and configuration examples live in the online documentation.

Community & contribution

  • Contributing – add new engine adapters, storage back‑ends, or cache policies via the developer guide.
  • Bug reports / feature requests – open an issue on the GitHub repo.
  • Documentation – extensive user‑guide, developer‑guide, and troubleshooting sections.

License

UCM is released under the MIT license with additional conditions (see the LICENSE file in the repository).

All information above is taken directly from the repository’s README and linked documentation; no external assumptions have been added.

Related

  • Project
    ovg-project/kvcached

    kvcached is a Python library/daemon that adds virtual‑memory‑style, on‑demand KV‑cache allocation to LLM serving engines (SGLang, vLLM). It lets multiple models share a GPU, puts idle models to sleep, and reduces time‑to‑first‑token by up to 28×, saving GPU costs.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Project
    LMCache/LMCache

    LMCache is a vendor-neutral KV cache management layer that offloads and persists LLM inference cache to improve throughput and reduce latency for long-context and agentic workloads.

  • Project
    kvcache-ai/Mooncake

    Mooncake is a high-performance infrastructure project that provides a disaggregated, KV cache-centric architecture to improve the efficiency, scalability, and latency of large-scale LLM inference and training.

  • Project
    microsoft/vattention

    A memory manager for LLM KV-caches that uses CUDA virtual memory APIs to enable dynamic allocation while maintaining virtual contiguity for unmodified attention kernels.