FareedKhan-dev/kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

What it solves

This project provides a portable C99 implementation of the Kimi K3 inference engine, allowing a massive 2.78-trillion-parameter model to run on consumer-grade hardware. It solves the problem of extreme memory requirements by enabling the model to run on as little as 8 GB of RAM, regardless of the machine's total memory, by streaming model weights from disk rather than requiring them to be fully resident in memory.

How it works

The engine uses several techniques to reduce memory footprints:

  • Weight Streaming: The "trunk" (dense layers) is packed into a single file and streamed from disk. The amount of memory allocated for this is adjustable via presets (e.g., laptop, server).
  • Expert Caching: The 1.45 TB of routed experts are never fully resident; instead, they are multiplied straight from their packed 4-bit form and managed via an LRU (Least Recently Used) cache.
  • Architectural Optimizations: It implements KDA attention (memory that never grows) and MLA (Multi-head Latent Attention) to reduce the memory overhead of the KV cache.
  • Portable C99: The engine is written in portable C99 with no dependencies on BLAS or heavy AI frameworks, using OpenMP for parallelism and AVX2/NEON for hardware acceleration.

Who it’s for

  • AI Researchers and Developers interested in high-efficiency inference and memory-constrained environments.
  • Users with limited hardware who want to run the Kimi K3 model on a standard laptop or desktop without needing a GPU cluster.

Highlights

  • Extreme Memory Efficiency: Runs a 2.78T parameter model in as little as 8.24 GB of peak RSS.
  • Zero GPU Dependency: Operates entirely on the CPU.
  • Byte-Identical Output: Produces the exact same results regardless of whether it is running on 8 GB or 224 GB of RAM.
  • Minimal Footprint: The entire inference engine is only 176 KB.
  • Cross-Platform: Supports Linux, macOS, and Windows.

Related

  • Dispatch
    WASTE Inference Engine: Running Kimi K3 2.78T on Consumer Hardware

    WASTE is a dependency-free C inference engine that enables running the 2.78-trillion-parameter Kimi K3 model on consumer laptops by streaming activated weights from NVMe storage.

  • Project
    xboot/libonnx

    A lightweight, portable pure C99 ONNX inference engine designed for embedded devices with support for hardware acceleration.

  • Project
    FlashML-org/FreeVideo

    A local inference engine that allows MiniMax H3 video generation to run on consumer GPUs with as little as 8 GB of VRAM through hardware-adaptive execution and memory optimization.

  • Dispatch
    vLLM Kimi K3 Support

    vLLM has released day-0 support for Kimi K3, a 2.8-trillion-parameter multimodal MoE model, featuring optimizations for Kimi Delta Attention and DSpark speculative decoding to achieve up to 370 tok/s.

  • Dispatch
    AirLLM enables 70B LLM inference on a single 4GB GPU

    AirLLM lets you run 70‑billion‑parameter LLMs on a single 4 GB GPU by streaming one layer at a time, without quantization, as demonstrated in the lyogavin/airllm repository.