sqliteai/warp

Run the full 2.78-trillion-parameter Kimi K3 model or GLM-5.3-Flash beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

What it solves

WARP (Weight-Aware Runtime and Paging) is an embeddable inference engine designed to run massive frontier models—such as the 2.78-trillion-parameter Kimi K3—on consumer-grade hardware. It addresses the memory bottleneck of huge Mixture-of-Experts (MoE) models by allowing them to run when the model weights exceed available RAM, utilizing fast local storage (NVMe SSDs) as the primary weight repository.

How it works

WARP keeps the shared "trunk" of a model in RAM while streaming only the specific experts needed for each token directly from disk. To optimize performance, it employs several techniques:

  • Expert Caching: Unused RAM is used as a bounded cache to store frequently accessed experts, reducing disk reads.
  • Lookahead Routing: A router predicts which experts will be needed for the next layer and begins reading them from disk before they are required.
  • Aligned Reads: The model container is structured so that one expert requires exactly one aligned read, overlapping disk I/O with computation.
  • Quantization: Experts use 3-bit residual vector quantization, while shared weights use 4 or 8 bits.
  • Efficient Attention: It leverages linear attention and compressed latent KV caches to keep memory requirements low (e.g., 0.21 GB for K3 at 4K context).

Who it’s for

It is for developers and researchers who want to run trillion-parameter multimodal models locally on machines with limited RAM (e.g., 16 GB to 64 GB) using fast NVMe storage.

Highlights

  • Massive Model Support: Capable of running Kimi K3 (2.78T parameters) and GLM-5.3-Flash (313B parameters).
  • Zero Dependencies: Written in C with no third-party runtime dependencies (no BLAS, CUDA, or Python required for inference).
  • Multimodal Capabilities: Supports both text and image inputs for Kimi K3 and GLM-5.3-Flash.
  • Hardware Efficient: Can run a 313B model on a machine with as little as 16 GB of RAM.
  • Broad Compatibility: Supports macOS, Linux (arm64/x86_64), and Windows (via MinGW-w64).

Related

  • Dispatch
    WASTE Inference Engine: Running Kimi K3 2.78T on Consumer Hardware

    WASTE is a dependency-free C inference engine that enables running the 2.78-trillion-parameter Kimi K3 model on consumer laptops by streaming activated weights from NVMe storage.

  • Project
    FareedKhan-dev/kimi-k3-in-c

    A portable C99 inference engine that enables a 2.78-trillion-parameter model to run on a single CPU with as little as 8 GB of RAM by streaming weights from disk.

  • Project
    JustVugg/colibri

    A pure C inference engine that enables running frontier MoE models (up to 2.8T parameters) on consumer hardware by treating VRAM, RAM, and disk as a unified memory hierarchy.

  • Project
    MoonshotAI/Kimi-K3

    Kimi K3 is an open-weight, 2.8T-parameter native multimodal MoE model designed for long-horizon coding, deep research, and complex agentic knowledge work.

  • Dispatch
    Kimi K3 Architecture Overview

    Kimi K3 is a 2.8T parameter open-weight model that optimizes inference efficiency through LatentMoE, Multi-Head Latent Attention, and a total removal of positional embeddings (NoPE).