MoonshotAI/checkpoint-engine

Checkpoint-engine is a simple middleware to update model weights in LLM inference engines

What it solves

Checkpoint Engine is a middleware designed to solve the problem of slow model weight updates in LLM inference engines. This is particularly critical for reinforcement learning workflows where weights must be updated frequently and efficiently across large clusters of GPUs without causing significant downtime or performance degradation.

How it works

It operates as a ParameterServer service colocated with inference engines, utilizing two primary update methods:

  • Broadcast: The fastest method for synchronous updates across many instances. It uses a three-stage process (Host-to-Device transfer, broadcasting among workers via CUDA IPC buffers, and reloading by the inference engine) organized into a pipeline to overlap communication and copying.
  • P2P: Used for dynamic scaling or restarts. It leverages the mooncake-transfer-engine to send weights from the CPUs of existing instances to the GPUs of new instances, optimizing bucket assignment to maximize network bandwidth.

The engine orchestrates these transfers by gathering metadata, planning bucket sizes, and controlling the inference engine via ZeroMQ sockets.

Who it’s for

It is built for developers and researchers working with massive LLMs (such as the 1-trillion parameter Kimi-K2) who need to perform rapid weight updates across thousands of GPUs, specifically those using vLLM or SGLang as their inference backend.

Highlights

  • Extreme Scale: Capable of updating a 1-trillion parameter model across thousands of GPUs in approximately 20 seconds.
  • Flexible Update Modes: Supports both high-speed synchronous broadcasting and flexible peer-to-peer transfers for dynamic instance availability.
  • Pipelined Execution: Overlaps data transfer and memory copying to maximize GPU throughput.
  • Framework Integration: Provides out-of-the-box support and integration paths for vLLM and SGLang.