FareedKhan-dev/kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

What it solves

This project provides a portable C99 implementation of the Kimi K3 inference engine, allowing a massive 2.78-trillion-parameter model to run on consumer-grade hardware. It solves the problem of extreme memory requirements by enabling the model to run on machines with as little as 8 GB of RAM, regardless of the model's 1.56 TB checkpoint size, by streaming weights from disk.

How it works

The engine uses several memory reduction techniques to fit the model into limited RAM:

  • Streaming Trunk: The dense layers (the "trunk") are packed into a single file and streamed from disk, turning memory requirements into a adjustable "dial" where more RAM increases speed but not the output quality.
  • Expert Cache: Routed experts (1.45 TB) are never fully resident in memory; instead, they are multiplied straight from their packed 4-bit form and managed via an LRU cache.
  • Architectural Optimizations: It implements KDA (attention with fixed memory) and MLA (Multi-head Latent Attention) to reduce the memory footprint of the attention mechanism.
  • Zero Dependencies: The engine is written in portable C99 with no BLAS, no GPU, and no heavy AI frameworks, relying only on libm and OpenMP for parallelism.

Who it’s for

Researchers and developers who want to run massive MoE (Mixture-of-Experts) models on local, CPU-only hardware without needing a cluster of GPUs.

Highlights

  • Extreme Memory Efficiency: Runs a 2.78T parameter model in as little as 8.24 GB of peak RSS.
  • Byte-Identical Output: Produces the exact same results regardless of whether it is running on 8 GB or 224 GB of RAM.
  • Portable C99: No GPU or external ML frameworks required; builds quickly with GCC or Clang.
  • Flexible Memory Presets: Includes presets (laptop, desktop, workstation, server) to optimize performance based on available system RAM.

Related

  • Project
    xboot/libonnx

    A lightweight, portable pure C99 ONNX inference engine designed for embedded devices with support for hardware acceleration.

  • Project
    sqliteai/warp

    An embeddable C-based inference engine that enables trillion-parameter MoE models to run on consumer hardware by streaming experts from disk to RAM.

  • Dispatch
    WASTE Inference Engine: Running Kimi K3 2.78T on Consumer Hardware

    WASTE is a dependency-free C inference engine that enables running the 2.78-trillion-parameter Kimi K3 model on consumer laptops by streaming activated weights from NVMe storage.

  • Project
    JustVugg/colibri

    A pure C inference engine that enables running frontier MoE models (up to 2.8T parameters) on consumer hardware by treating VRAM, RAM, and disk as a unified memory hierarchy.

  • Project
    vitoplantamura/OnnxStream

    A memory-optimized inference library that enables large AI models like Stable Diffusion and LLMs to run on ultra-low-RAM devices by streaming weights and using attention slicing.