Colibrì – Tiny C‑engine for frontier Mixture‑of‑Experts models

What it is – An open‑source inference engine written in a single C file that can run extremely large mixture‑of‑experts (MoE) language models (from 744 B to 2.8 T parameters) on consumer‑grade hardware. It treats VRAM, RAM and NVMe storage as a unified, multi‑tier memory hierarchy, streaming expert weights from disk when they are needed.

Why it matters – Frontier models are usually only accessible on expensive cloud GPUs. Colibrì’s “AI memory multitiering” lets you run a 744 B‑parameter model on a laptop or workstation by keeping only the dense core in RAM and loading the sparsely‑used experts from fast storage on‑the‑fly. The engine is deliberately small (one C source file, no BLAS, no Python runtime) so anyone can read, modify, and benchmark the system‑level ideas.

Core ideas

Idea	How it works (as described in the README)
Unified memory hierarchy	VRAM, system RAM and NVMe are treated as placement tiers for the same weight tensors. Limited fast memory only slows execution; it never changes model semantics.
Weight JIT / learned cache	The router records which experts are hot, keeps a per‑layer LRU cache, and can “pin” frequently used experts. A look‑ahead thread pre‑fetches the next‑layer experts, achieving ~71 % prediction accuracy one layer ahead.
I/O as part of the engine	Reads are performed with O_DIRECT, batched expert unions are streamed, and dual‑SSD striping is supported to increase read bandwidth. Mirrors are validated at start‑up and fall back to the primary drive on errors.
Heterogeneous execution	The same runtime can schedule work on CPU, CUDA, Metal, Vulkan, or a mix of them. Users can enable GPU tiers, NUMA‑aware RAM placement, or run a pure‑CPU configuration.
Compressed KV cache	The attention KV cache is stored at 57× smaller size (576 floats per token) while remaining token‑exact, enabling long‑context chats with modest memory.
Speculative decoding	Native MTP drafts and optional grammar‑forced drafts are supported; they are measured end‑to‑end and can be disabled when they do not improve throughput.

---

Supported model families (one C file per model)

• GLM‑5.2 / 5.3 – up to 744 B parameters
• GLM‑5.3‑Flash – 321 B with vision
• Inkling – 975 B
• Kimi K3 – 2.8 T
• DeepSeek V4 Flash – 284 B
• DeepSeek V4.1 Flash – 552 B with vision
• Qwen 3.8‑Flash‑Next – 125 B + 51 B n‑gram
• Qwen 3.6 – 35 B‑3 B range
• OLMoE – 7 B

Each model is compiled from a single c/colibri.c source plus small headers; the front‑ends (coli chat, coli serve, coli web) are identical across models.

Typical workflow

# start a local chat session (model already converted to int4 safetensors)
./coli chat --model /path/to/glm52_i4

# launch a web UI with live metrics
./coli web

# (optional) run a multi‑node cluster – start a coordinator then workers
./coli cluster coordinator --host 0.0.0.0 --port 8765
./coli cluster worker --model /nvme/glm52_i4 --port 9100 \
    --coordinator http://COORDINATOR:8765 --advertise-host <worker_ip>

The engine automatically decides which experts live in VRAM, RAM or on‑disk, and it can overlap I/O with compute using the async I/O pool and the pilot pre‑fetch thread.

Research‑oriented features

• Experiment‑ready hypothesis framework – every optimisation is treated as a hypothesis that must be validated with reproducible end‑to‑end A/B tests (see the benchmark protocol in docs/benchmarking.md).
• Open hypotheses table – the README lists current research questions (e.g., multi‑SSD striping, learned routing vs. LRU, hardware‑aware planner) and invites contributions.
• Profiling UI – the web dashboard shows per‑turn phase breakdown, expert heat maps, and a “Brain” explorer that visualises the measured expert atlas of GLM‑5.2.
• Brio mode – a deterministic “read‑only” inference that returns probabilities for a fixed set of candidate answers without generating text, useful for retrieval‑augmented generation or evaluation.

---

Community & resources

• Website: https://justvugg.github.io/colibri – documentation, model conversion guides, and download links.
• Discord: https://discord.gg/RXV83nSZdk – discussion, support, and experiment coordination.
• Contributing: detailed guidelines in CONTRIBUTING.md and a benchmark protocol for publishing both positive and negative results.
• Multilingual READMEs: Chinese (simplified/traditional), Italian, Japanese.

---

TL;DR

Colibrì is a minimalist C inference engine that makes frontier‑size MoE language models accessible on ordinary machines by streaming expert weights from storage and overlapping I/O with compute. It ships with a handful of 744 B‑2.8 T models, a web UI for monitoring, and a research‑first mindset that encourages anyone to test system‑level ideas and publish rigorous measurements.