LMCache/LMCache

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

What it solves

LMCache addresses the inefficiency of the Key-Value (KV) cache in LLM inference, which is typically treated as temporary state. By transforming the KV cache into reusable, persistent knowledge, LMCache reduces the time-to-first-token (TTFT) and increases throughput, particularly for long-context agentic workloads, multi-turn conversations, and RAG applications where repeated prefill computation is a major bottleneck.

How it works

LMCache acts as a vendor-neutral management layer that sits between the inference engine and storage. It manages the KV cache independently as a standalone daemon process, preventing cache loss during engine crashes. It employs a tiered storage hierarchy (CPU memory, local SSD, and remote backends like Redis or S3) to offload and reuse caches across different requests, sessions, and engine instances.

Beyond simple prefix caching, it supports non-prefix KV reuse via CacheBlend to selectively recompute tokens. It also enables Prefill-Decode (PD) disaggregation, transferring KV caches from prefill workers to decode workers via NVLink, RDMA, or TCP.

Who it’s for

Developers and infrastructure providers building scalable LLM serving systems who need to optimize performance for long-context or multi-turn interactions without being locked into a specific inference engine or hardware vendor.

Highlights

  • Engine-independent deployment: KV cache is managed by a separate daemon, ensuring it persists even if the inference engine crashes.
  • Persistent tiered storage: Offloads KV caches to CPU RAM, local disk, and remote backends (e.g., Redis, S3) for reuse across sessions.
  • Pluggable backends: Supports a wide range of transport and storage providers including NVLink, RDMA, Redis, and S3-compatible storage.
  • Non-prefix reuse: Reuses cached KV blocks at any position in the prompt, not just the beginning.
  • Production observability: Provides detailed metrics for cache hits, lifecycle, and performance diagnostics.
  • KV transformation: Offers a flexible interface for researchers to implement custom compression and serialization.