ModelEngine-Group/unified-cache-management
Persist and reuse KV Cache to speedup your LLM.
What it solves
Unified Cache Manager (UCM) addresses the high GPU memory consumption and computational redundancy associated with the KV cache in Large Language Models (LLMs), particularly during long-sequence inference. It solves the problem of limited GPU memory by offloading the KV cache to external storage and reducing redundant calculations through efficient retrieval mechanisms.
How it works
UCM implements a storage-compute separation architecture that persists the KV cache and replaces redundant computations with retrieval. It integrates with vLLM and uses several key components:
- UcmSparseBase: A base class that allows various sparse attention algorithms to be plugged in as modules.
- SparseKVManager: Manages custom KV block allocations for different algorithms.
- KVStoreBase: Decouples sparse algorithms from external storage, allowing them to work with various storage systems (e.g., local filesystems or NFS) using block IDs and offsets.
- KVStoreConnector: Links the system with vLLM's KVConnectorBase to enable prefix caching.
Who it’s for
It is designed for developers and researchers working with LLM inference engines (specifically vLLM) who need to optimize performance for multi-turn dialogues, long-context reasoning, and extremely long sequence inference tasks.
Highlights
- Performance Boost: Achieves a 3-10x reduction in inference latency when integrated with vLLM.
- Flexible Retrieval: Supports prefix caching and various training-free sparse attention retrieval methods.
- Storage-Compute Separation: Enables heterogeneous PD (Prefill-Decode) disaggregation for more flexible resource management.
- Comprehensive Feature Set: Includes prefill offload, cache blend, and model window extrapolation.