LMCache/LMCache

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

What it solves

LMCache addresses the high memory and computational costs associated with LLM inference by managing the Key-Value (KV) cache as a persistent, reusable resource. By offloading this cache from GPU memory to other storage tiers and enabling sharing across different inference engines and sessions, it significantly reduces the time-to-first-token (TTFT) and improves overall system throughput, particularly for long-context, RAG, and agentic workloads.

How it works

LMCache operates as a standalone daemon process that sits alongside your inference engine. It intercepts and manages the KV cache, allowing it to be stored in a tiered hierarchy including CPU RAM, local SSDs, or remote backends like Redis and S3. Because it is decoupled from the inference engine, the cache persists even if the engine crashes. It supports advanced features like non-prefix cache reuse (reusing blocks at any position in a prompt) and provides a pluggable interface for KV transformation, such as compression or custom serialization.

Who it’s for

This project is designed for infrastructure engineers, MLOps practitioners, and researchers building high-performance, scalable LLM serving systems who need to optimize memory usage and reduce latency in production environments.

Highlights

  • Engine-independent: Decouples KV cache management from the inference engine to prevent data loss during engine crashes.
  • Tiered Storage: Supports offloading KV caches to CPU memory, local disk, and remote storage backends (Redis, S3, etc.).
  • Observability: Provides comprehensive metrics for monitoring cache hits, lifecycle, and performance at the token and request levels.
  • Flexible Integration: Vendor-neutral design that works with various hardware and serving frameworks, including support for NVLink and RDMA for fast cache transfer.

Related

  • Project
    AstraNetLab/CacheRoute

    A lightweight LLM scheduling framework that reduces prefill overhead by reusing KV caches for frequently accessed external knowledge across LLM systems.

  • Project
    cchenax/px-kvstore

    A lightweight, high-performance in-memory KV store optimized for deterministic LLM caching and fast local AI inference experimentation.

  • Dispatch
    vLLM Tiered KV Cache Offloading

    vLLM introduced tiered KV cache offloading, which preserves evicted key‑value data across host memory, storage, and remote peers to avoid recomputation, cut latency, and boost serving capacity.

  • Project
    ModelEngine-Group/unified-cache-management

    UCM (Unified Cache Manager) is an open‑source library that provides a shared KV‑cache layer for LLM inference engines. It lets multiple requests reuse attention cache, spills cache to external storage to extend accelerator memory, and offers pluggable engine adapters and storage back‑ends. Ideal for chat‑bot servers, multimodal pipelines, and any workload that repeatedly accesses the same context.