kvcache-ai/Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

What it solves

Mooncake addresses the inefficiencies of large-scale LLM serving, specifically the bottlenecks caused by the KV cache. It solves the problem of underutilized hardware resources (CPU, DRAM, and SSD) in GPU clusters and the high latency associated with moving large amounts of data (like KV caches and model weights) across networks during inference and training.

How it works

Mooncake implements a KV cache-centric disaggregated architecture that separates prefill and decode clusters. It uses three primary components to manage data and execution:

  • Transfer Engine (TE): A high-performance data transfer framework that provides a unified interface for moving tensors across different storage types (DRAM, VRAM, NVMe) and network protocols (RDMA, TCP, AWS EFA, etc.) with topology-aware routing and bandwidth aggregation.
  • Mooncake Store: A distributed key-value cache storage engine that manages reusable KV caches and model weights across clusters, utilizing a multi-tier hierarchy (DRAM and SSD/NVMe) and zero-copy data transfer.
  • Mooncake EP & PG: Tools for fault-tolerant distributed execution, specifically for Mixture-of-Experts (MoE) models, allowing the system to route around failed ranks and recover processes without restarting the entire service.

Who it’s for

It is designed for developers and operators of large-scale LLM inference and training infrastructure, particularly those using frameworks like vLLM, SGLang, or TensorRT-LLM, who need to maximize throughput and reduce latency in multi-node GPU clusters.

Highlights

  • High-Bandwidth Transfer: Delivers up to 190 GB/s in 8x400 Gbps RoCE networks, significantly faster than standard TCP.
  • Disaggregated Storage: Decouples KV cache storage from the inference engine, allowing for elastic scaling of storage nodes independent of engine restarts.
  • Multi-Tier Caching: Supports a hierarchy of DRAM and SSD/NVMe to increase total cache capacity.
  • Fault-Tolerant MoE: Enables expert-parallel inference that can detect and bypass failed ranks to maintain service availability.
  • Broad Ecosystem Integration: Deeply integrated with vLLM, SGLang, and TensorRT-LLM for efficient KV cache and weight transfer.