JustVugg/colibri

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

colibri is a tiny, pure-C engine that runs very large open-weights AI models (mixture-of-experts language models, a vision model, and decision models) on ordinary computers—no GPU required, though it can use one when available. Its core trick is that a huge model like GLM-5.2 (744B parameters) only activates a small fraction of its weights for each token, so colibri keeps the always-needed "dense" part in RAM and streams the rarely-used "expert" weights from disk on demand. This lets models that would normally need hundreds of GB of VRAM run on a machine with 8–16 GB of RAM and a fast NVMe drive.

How it works

  • Mixture-of-experts (MoE) models have many parallel "expert" sub-networks; only a few are used per token. colibri exploits this sparsity.
  • The dense layers (attention, shared experts, embeddings) stay resident in RAM. Routed experts are read from disk through a cache that learns which ones your conversations use.
  • If a GPU is present, it can hold the hottest experts and dense layers, accelerating inference. The engine supports Vulkan (any GPU with a Vulkan 1.2 driver) and CUDA (NVIDIA cards on Linux/Windows).
  • The engine is written in C, one file per model family, and is designed to be auditable and measurable—an open research platform.

What it runs

  • Language models: GLM-5.2/5.3, DeepSeek V4 Flash, Kimi K3 (2.8T params), Qwen3.6, Qwen3-Coder, OLMoE, and more.
  • Image generation: Qwen-Image-2.1.
  • Decision models (fast, single-answer classifiers): Laya, GLiNER2.5-Decide, Clef.
  • All models are open-weights and downloaded from Hugging Face.

Getting started

  • One-step installer for Windows, Linux, macOS: download, run START-HERE.bat or ./start-here.sh, answer one question (which model), and it detects your hardware, recommends a fitting model, builds/downloads the engine, downloads the model, and starts a local server with a browser dashboard.
  • The server exposes OpenAI- and Anthropic-compatible APIs, so you can plug it into existing tools.
  • An MCP server (coli mcp) lets AI coding assistants install and manage colibri programmatically.

Key features

  • Streaming from disk: models much larger than RAM run by reading experts on demand.
  • GPU acceleration: optional Vulkan or CUDA builds; the setup automatically picks the best backend for your hardware.
  • Transparent and measurable: every optimization is backed by reproducible benchmarks; the dashboard shows expert activation in real time.
  • Resumable downloads and a robust setup that handles missing packages, GPU driver issues, and disk space problems.

Hardware requirements

  • Minimum 8 GB RAM (16 GB+ recommended), 22 GB free disk for the smallest model, internet connection. A GPU is optional.
  • For large models, a fast NVMe SSD is crucial—disk speed limits token generation.

Why it matters colibri democratizes access to frontier-scale open models by making them run on hardware people already own, without sacrificing fidelity—it never silently changes model precision or routing semantics. It's a practical tool and a research platform for efficient MoE inference.

Written about in

Related

  • Dispatch
    colibrì runs GLM‑5.2 744B‑parameter MoE on a 25 GB RAM consumer machine

    colibrì demonstrates that the 744‑billion‑parameter GLM‑5.2 Mixture‑of‑Experts model can be run on a 25 GB‑RAM consumer PC using a pure‑C engine that streams experts from disk.

  • Dispatch
    Lumabri: P2P Swarm Inference for Mixture-of-Experts Models

    Lumabri is a dependency-free C engine that enables running large Mixture-of-Experts (MoE) models across a peer-to-peer swarm by distributing expert weights and compute across multiple CPU or GPU nodes.

  • Project
    illuin-tech/colpali

    ColPali is a research library for visual document retrieval. It encodes document images with vision‑language models (e.g., PaliGemma, Qwen‑VL) into multi‑vector embeddings and scores them against query embeddings using a ColBERT‑style late‑interaction method. The original `colpali-engine` package (now deprecated) provides model classes, processors, optional fused kernels, fast‑Plaid indexing, token‑pooling, and interpretability tools. New projects should use Sentence‑Transformers v6’s `MultiVectorEncoder`, which integrates ColPali‑style models.

  • Dispatch
    AI & Frontier Tech Roundup: Localized Frontier Models, Agentic Infrastructure, and Physical AI

    The AI landscape is shifting toward making frontier-class intelligence cheap and local, with breakthroughs in MoE inference on consumer hardware and a surge in agentic infrastructure and physical robotics.

  • Project
    Edge0-AI/Edge0

    edge0 is an open‑source framework for streaming inference of large sparse‑MoE language models. It streams expert weights from SSD, uses a prerouter to pre‑fetch experts, and recovers quantisation loss with LoRA adapters, enabling 35B‑parameter models to run on a laptop with < 3 GiB RAM. The repo ships Python, macOS, iOS, Android and Windows runtimes, two model tiers (35B and 8B), benchmarks, and a roadmap for a unified cross‑platform API.