kvcache-ai/Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

What it solves

Mooncake addresses the efficiency bottlenecks in large-scale LLM serving, specifically the high cost of KVCache management and the performance gaps in data movement across heterogeneous hardware. It solves the problem of underutilized CPU, DRAM, and SSD resources in GPU clusters by creating a disaggregated KV cache pool, allowing prefill and decode clusters to be separated to improve throughput and reduce latency.

How it works

Mooncake employs a KVCache-centric disaggregated architecture. It consists of three primary components:

  1. Transfer Engine (TE): A high-performance framework that provides a unified interface for batched data movement across diverse networks (TCP, RDMA, AWS EFA) and accelerators (CUDA, ROCm, Ascend, etc.), utilizing topology-aware routing and multi-NIC bandwidth aggregation.
  2. Mooncake Store: A distributed key-value storage engine that manages reusable KV caches and model weights across clusters. It uses a multi-tier cache hierarchy (DRAM and SSD/NVMe) and supports zero-copy data transfer to maximize bandwidth.
  3. Mooncake EP & PG: A fault-tolerant distributed execution layer for MoE (Mixture-of-Experts) inference, providing expert-parallel dispatch and a PyTorch-compatible process-group backend that can recover failed ranks without restarting the entire service.

Who it’s for

It is designed for developers and operators of large-scale LLM inference and training infrastructures, particularly those deploying massive models (like MoE models) across multi-node GPU clusters who need to maximize hardware utilization and maintain strict service-level objectives (SLOs).

Highlights

  • High-Bandwidth Data Transfer: Delivers up to 190 GB/s in 8x400 Gbps RoCE networks, significantly faster than standard TCP.
  • Disaggregated Storage: Decouples KVCache storage from inference engines, allowing storage nodes to be scaled independently of engine restarts or upgrades.
  • Fault-Tolerant MoE Serving: Enables MoE inference to route around failed ranks and recover processes elastically.
  • Broad Ecosystem Integration: Deeply integrated with vLLM, SGLang, TensorRT-LLM, and the PyTorch ecosystem.

Related

  • Dispatch
    Serving Agentic Workloads at Scale with vLLM x Mooncake

    vLLM integrates Mooncake's distributed KV cache store to boost agentic LLM serving, delivering 3.8× higher throughput, 46× lower TTFT, and 8.6× lower end‑to‑end latency on realistic traces while scaling to 60 GB200 GPUs.

  • Project
    MoonshotAI/checkpoint-engine

    A middleware for LLM inference engines that enables rapid, in-place model weight updates across large GPU clusters, essential for reinforcement learning.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.

  • Project
    lemonade-sdk/lemonade

    Lemonade is an open‑source local AI server that runs LLMs, speech, TTS, image and other multimodal models on your own CPU/GPU/NPU. It provides a standard OpenAI‑compatible HTTP API, a CLI for model management, and two distribution flavors – a full server service and a portable embeddable binary. Supports Windows, macOS, Linux (many distros, Docker, Snap) and a broad set of hardware back‑ends (CUDA, ROCm, Vulkan, Metal, AMD NPU, etc.).