JustVugg/colibri

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

What it solves

Colibrì is a high-performance inference engine designed to run massive Mixture-of-Experts (MoE) models—ranging from 744B to 2.8T parameters—on consumer-grade and heterogeneous hardware. It removes the need for hyperscaler-class hardware by treating storage, RAM, and VRAM as a single, unified memory hierarchy, allowing models that are too large for fast memory to be streamed from disk without changing the model's mathematical behavior.

How it works

The engine treats VRAM, RAM, and NVMe storage as placement tiers. The dense parts of a model (like attention and embeddings) stay resident in RAM, while the thousands of routed experts are stored on disk and streamed on demand.

Key technical optimizations include:

  • Weight JIT: A learning cache that tracks "routing heat" to pin frequently used experts in faster memory tiers.
  • I/O Optimization: Uses O_DIRECT to bypass page caches, batched expert unions to reduce read calls, and a "PILOT" thread that prefetches experts one layer ahead.
  • Dual-SSD Striping: Allows the engine to stream weights from two separate SSDs simultaneously to double read bandwidth.
  • Heterogeneous Execution: Supports a single runtime across CPU, CUDA, Metal, and Vulkan backends.
  • Compressed State: Implements a 57x smaller MLA KV state to reduce memory overhead and persist conversations across restarts.

Who it’s for

It is built for researchers and users who want to run frontier-scale MoE models locally on their own hardware rather than relying on proprietary APIs, as well as systems engineers interested in optimizing the software/hardware boundary for LLM inference.

Highlights

  • Massive Model Support: Runs models like GLM-5.2 (744B) and Kimi K3 (2.8T).
  • Zero Dependencies: Written in pure C as a single file with no BLAS or Python requirements at runtime.
  • Hardware Agnostic: Works on everything from 25GB dev boxes (disk-streaming) to multi-GPU workstations (full residency).
  • Visual Tooling: Includes a web dashboard with a "Brain" view to visualize expert routing in real-time and an "Atlas" view to map expert topic affinity in 3D.

Related

  • Dispatch
    colibrì runs GLM‑5.2 744B‑parameter MoE on a 25 GB RAM consumer machine

    colibrì demonstrates that the 744‑billion‑parameter GLM‑5.2 Mixture‑of‑Experts model can be run on a 25 GB‑RAM consumer PC using a pure‑C engine that streams experts from disk.

  • Project
    sqliteai/warp

    An embeddable C-based inference engine that enables trillion-parameter MoE models to run on consumer hardware by streaming experts from disk to RAM.

  • Project
    FareedKhan-dev/kimi-k3-in-c

    A portable C99 inference engine that enables a 2.78-trillion-parameter model to run on a single CPU with as little as 8 GB of RAM by streaming weights from disk.

  • Project
    kvcache-ai/ktransformers

    A framework for efficient LLM inference and fine-tuning using CPU-GPU heterogeneous computing, specifically optimized for ultra-large MoE models on limited hardware.

  • Dispatch
    Lumabri: P2P Swarm Inference for Mixture-of-Experts Models

    Lumabri is a dependency-free C engine that enables running large Mixture-of-Experts (MoE) models across a peer-to-peer swarm by distributing expert weights and compute across multiple CPU or GPU nodes.