ModelEngine-Group/unified-cache-management

Persist and reuse KV Cache to speedup your LLM.

What it solves

Unified Cache Manager (UCM) addresses the high GPU memory consumption and computational redundancy associated with the KV cache in Large Language Models (LLMs), particularly during long-sequence inference. It solves the problem of GPU memory bottlenecks by offloading KV caches to external storage and reducing redundant calculations through efficient retrieval mechanisms.

How it works

UCM implements a storage-compute separation architecture that persists the LLM KV cache. It uses a modular framework where different sparse attention algorithms can be plugged in via a base class (UcmSparseBase). A SparseKVManager handles custom block allocations, while a KVStoreBase decouples the sparse algorithms from the physical storage (such as local filesystems or NFS), allowing for flexible data movement between GPU memory and external storage.

Who it’s for

This project is designed for developers and researchers working with LLM inference engines (specifically vLLM) who need to optimize performance for multi-turn dialogues, long-context reasoning, and scenarios requiring heterogeneous computing resources.

Highlights

  • Prefix Caching: Supports persisting and reusing common prompt prefixes to avoid redundant computation.
  • Sparse Attention: Offers training-free sparse attention retrieval methods to improve performance on extremely long sequences.
  • PD Disaggregation: Provides a Prefill-Decode (PD) disaggregation solution to better manage heterogeneous computing resources.
  • vLLM Integration: Achieves a 3-10x reduction in inference latency when integrated with vLLM.
  • Flexible Storage: Supports various external storage options, including NFS for multi-server environments.

Related

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Project
    ovg-project/kvcached

    A KV cache library that brings virtual memory abstraction to LLM serving, enabling elastic and demand-driven GPU memory sharing across multiple models.

  • Project
    LMCache/LMCache

    A KV cache management layer for LLM inference that reduces TTFT and improves throughput by making KV caches persistent and reusable across requests and engines.

  • Project
    NVIDIA/kvpress

    A framework for implementing and benchmarking KV cache compression methods to reduce memory usage and increase inference speed for long-context LLMs.

  • Project
    lucienhuangfu/eLLM

    An LLM inference framework for CPU servers that optimizes long-horizon inference by trading storage for computation to outperform GPUs in long-context scenarios.