kvcache-ai/Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

What it solves

Mooncake addresses the performance bottlenecks in large-scale LLM inference and training by optimizing how KV (Key-Value) caches and model data are managed and moved across clusters. It enables "disaggregated" architectures, where compute-heavy tasks (like prefill and decoding) can be separated and scaled independently, while still sharing a high-performance, distributed cache pool to avoid redundant computation.

How it works

Mooncake functions as an infrastructure layer that sits beneath inference engines like vLLM or SGLang. It consists of two primary components: the Transfer Engine (TE), which handles low-latency, high-bandwidth data movement across heterogeneous networks (RDMA, TCP, etc.) and hardware accelerators; and the Mooncake Store, a distributed storage engine that manages these cached KV states across DRAM and SSDs. By using content-addressed storage and topology-aware routing, it allows different inference instances to share and reuse cache data efficiently.

Who it’s for

This project is designed for ML engineers and infrastructure teams building large-scale LLM serving systems, particularly those deploying models that require high throughput, long-context support, or disaggregated inference pipelines (e.g., separating prefill and decode nodes).

Highlights

  • High-Performance Data Transfer: Utilizes multiple RDMA NICs to achieve massive bandwidth (up to 190 GB/s in 8x400 Gbps networks).
  • Ecosystem Integration: Deeply integrated with major serving frameworks including vLLM, SGLang, TensorRT-LLM, and LMDeploy.
  • Hardware Agnostic: Supports a wide range of accelerators and interconnects, including CUDA, ROCm, NPU, MUSA, and EFA.
  • Multi-Tier Caching: Implements a hierarchical cache strategy across DRAM and NVMe/SSD to maximize capacity and reuse.
  • Production Proven: Powers the Kimi LLM service, enabling significantly higher request handling capacity within existing Service Level Objectives (SLOs).

Related

  • Dispatch
    Serving Agentic Workloads at Scale with vLLM x Mooncake

    vLLM integrates Mooncake's distributed KV cache store to boost agentic LLM serving, delivering 3.8× higher throughput, 46× lower TTFT, and 8.6× lower end‑to‑end latency on realistic traces while scaling to 60 GB200 GPUs.

  • Project
    LMCache/LMCache

    LMCache is a vendor-neutral KV cache management layer that offloads and persists LLM inference cache to improve throughput and reduce latency for long-context and agentic workloads.

  • Project
    llm-d/llm-d

    A high-performance distributed inference serving stack for Kubernetes that optimizes LLM production deployments across various hardware accelerators.

  • Project
    ovg-project/kvcached

    kvcached is a Python library/daemon that adds virtual‑memory‑style, on‑demand KV‑cache allocation to LLM serving engines (SGLang, vLLM). It lets multiple models share a GPU, puts idle models to sleep, and reduces time‑to‑first‑token by up to 28×, saving GPU costs.