MoonshotAI/checkpoint-engine

Checkpoint-engine is a simple middleware to update model weights in LLM inference engines

What it solves

Checkpoint Engine is a middleware designed to update model weights in LLM inference engines efficiently. This is particularly critical for reinforcement learning workflows where model weights must be updated frequently and rapidly across large-scale GPU clusters without interrupting service.

How it works

It utilizes a ParameterServer class that colocates with inference engines to manage weight updates via two primary methods:

  • Broadcast: The fastest method, used for synchronous updates across many instances. It uses a three-stage process (Host-to-Device transfer, broadcasting among workers via CUDA IPC buffers, and reloading into the inference engine) organized into a pipeline to overlap communication and copying.
  • P2P: Used for dynamic scaling, such as when new instances are added or restarted. It uses the mooncake-transfer-engine to send weights from the CPUs of existing instances to the GPUs of new instances, optimizing bucket assignment to maximize network bandwidth.

Who it’s for

It is intended for developers and researchers working with massive LLMs (e.g., 1 trillion parameters) and distributed inference frameworks like vLLM and SGLang who need high-performance, in-place weight updates.

Highlights

  • Extreme Scale: Capable of updating a 1 trillion parameter model across thousands of GPUs in approximately 20 seconds.
  • Flexible Update Paths: Supports both synchronous broadcast and asynchronous P2P transfers for dynamic availability.
  • Hardware Support: Compatible with NVIDIA GPUs (CUDA) and Intel XPUs (via SYCL IPC memory).
  • Framework Integration: Integrated with vLLM and SGLang to reduce model loading and update times.

Related

  • Project
    ServiceNow/PipelineRL

    A scalable asynchronous reinforcement learning framework for LLM agents that uses in-flight weight updates to maximize GPU utilization while maintaining on-policy data freshness.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Project
    UbiquitousLearning/mllm

    A fast, lightweight multimodal LLM inference engine for mobile and edge devices that optimizes vision-language model execution across various NPUs and GPUs.

  • Project
    NVIDIA/TensorRT-LLM

    An open-source optimization library from NVIDIA that maximizes inference performance, throughput, and efficiency for LLMs and visual generation models on NVIDIA GPUs.

  • Project
    PaddlePaddle/PaddleNLP

    A comprehensive LLM development suite based on PaddlePaddle that provides tools for efficient training, lossless compression, and high-performance inference across multiple hardware platforms.