colibrì runs GLM‑5.2 744B‑parameter MoE on a 25 GB RAM consumer machine

TL;DR – What colibrì achieves and why it matters

colibrì proves that a 744‑billion‑parameter GLM‑5.2 Mixture‑of‑Experts (MoE) model can be executed on a consumer laptop with only ~25 GB of RAM by streaming the majority of the model’s experts from an SSD. This shows that frontier‑class LLMs are no longer confined to multi‑GPU servers; they can be run on inexpensive hardware, albeit with modest token‑per‑second throughput.

Core design – Streaming MoE to fit in RAM

The GLM‑5.2 MoE activates roughly 40 B parameters per token, but only ~11 GB of those change between tokens. colibrì exploits this by:

  • Keeping the dense part resident – attention, shared experts, and embeddings (~17 B parameters) stay in RAM at int4 precision, consuming ~9.9 GB.
  • Storing the 21,504 routed experts on disk – each expert (~19 MB at int4) lives in a ~370 GB container. An LRU cache plus an optional pinned hot‑store keeps frequently used experts in RAM, while the OS page cache acts as a secondary cache.

The entire runtime engine is a single C file (c/glm.c, ~1,300 lines) with no BLAS, Python, or GPU dependencies.

Implemented features – What the engine can do today

  • Faithful GLM‑5.2 forward – token‑exact validation against a transformers oracle.
  • MLA attention with compressed KV cache – 576 floats/token (57× smaller than the full 32,768‑float cache).
  • DeepSeek‑V3‑style sigmoid router for expert selection.
  • Native MTP speculative decoding – multi‑token‑prediction head drafts tokens; at int8 draft precision the acceptance rate is 39–59 % (2.2–2.8 tokens per forward). At int4 the acceptance drops to 0–4 %.
  • True temperature + nucleus sampling tuned for int4 quantization (default 0.7 / 0.90).
  • AVX2 integer‑dot kernels – int8 matmuls 1.4–2.5× faster, int4 1.8× faster in batch.
  • Async expert readahead (WILLNEED) to overlap I/O and compute.
  • Quantization kernels for int8, packed int4, and packed int2 with per‑row scales.
  • Byte‑level BPE tokenizer (GPT‑2 style, 320 k merges) implemented in C.
  • RAM safety – expert cache size auto‑derived from /proc/meminfo to avoid OOM.
  • Offline FP8→int4 converter that streams shards, never requiring the full 756 GB FP8 checkpoint on disk.

Performance numbers on the reference setup

Metric Value
Model on disk (int4 container) ~370 GB
Resident RAM (dense part) 9.9 GB
Load time ~30 s
Peak RSS during chat ~20 GB
Cold decode I/O cost ~11 GB reads per token
Disk bandwidth (VHDX random) ~1 GB/s → 0.05–0.1 tok/s cold
MTP speculation (int8 head) 2.2–2.8 tok/forward

The engine is not fast – a cold token costs ~0.05–0.1 tokens per second on the author’s WSL2 box. Warm caches, pinned hot experts, and MTP speculation reduce latency dramatically, making interactive use possible on a machine that costs less than a single H100 fan.

SSD wear considerations

Cold starts issue ~11 GB of random reads per token. While reads are safe, the OS page cache may generate writes, potentially accelerating wear on cheap SSDs. Users should monitor drive health or use a dedicated read‑only partition for the model.

Getting started – Quick start guide

cd c
./setup.sh               # checks gcc/OpenMP, builds, self‑tests
./coli convert --model /path/to/GLM-5.2-FP8   # streams FP8 shards, converts to int4
COLI_MODEL=/path/to/GLM-5.2-int4 ./coli chat

Python is only required for the one‑time converter.

Useful runtime knobs

  • --temp T – sampling temperature (default 0.7).
  • --topp 0.7 – adaptive expert top‑p, reduces disk reads by 30–40 %.
  • AUTOPIN=0 – disable automatic hot‑expert pinning.
  • DRAFT=n – set MTP draft depth.
  • PIN=stats.txt PIN_GB=20 – pin the hottest experts using a usage log.

Scaling expectations on better hardware

Machine Expected token‑per‑second (cold)
Reference WSL2 box (1 GB/s random) 0.05–0.1
Native Linux PCIe 4 NVMe (3–5 GB/s) + 32 GB RAM 0.5–1
PCIe 5 NVMe or RAID0 (8–12 GB/s) + 64 GB RAM (40 GB pinned) 2–4
128–256 GB RAM, 12 cores (matmul‑bound) 2–4
128 GB RAM + 24–32 cores or AVX‑512/VNNI 5–15

Real‑world measurements from community contributors:

  • Intel Core Ultra 7 270K Plus (24 threads, WSL2) – 0.07 tok/s cold, 0.11 tok/s with --topp 0.7.
  • Apple M5 Max (18 cores, 128 GB unified, internal SSD) – 1.06 tok/s, expert hit‑rate 23 %. These confirm that RAM budget becomes the bottleneck on low‑RAM machines, while faster SSDs shift the limit back to compute.

Quality evaluation – Open benchmark request

The repository includes a coli bench harness that runs MMLU, HellaSwag, and ARC (40 questions each). Full‑precision GLM‑5.2 scores are 85–95 % on these tasks. The int4 container’s accuracy has not been measured at scale because a full run takes a day on the reference hardware. The author invites anyone with faster machines to run the benchmark and submit results, which will guide future quantization improvements.

Community feedback highlights

"Yeah this idea makes instant sense. Very well done, this deserves a github star on concept alone." – barent

"I’ve seen locally hosted LLMs that are as slow as 1 tok/s still be useful for overnight jobs. Running a 744B model at 0.05 tok/s is impressive for cheap hardware." – walrus01

"On Apple Silicon we can stream weights similarly, but with Metal kernels and unified memory. The idea is portable across architectures." – Archit3ch

"MTP speculation gives 2.2–2.8 tokens per forward at int8, but on a cold cache it can increase disk reads, so the engine disables it automatically until the cache warms up." – project author (vforno)

"Running this on a ThreadRipper with 128 GB RAM and a 7 GB/s NVMe yielded 0.44 tok/s after tuning, confirming the scalability estimates." – efficax

These comments reinforce that the core concept is sound, that performance scales with I/O bandwidth and RAM, and that speculative decoding is a key optimization.

How to contribute

  • Star the repository and share it.
  • Open issues with benchmark numbers from your hardware.
  • Sponsor development or donate spare SSDs/NVMe drives.
  • Submit pull requests for additional quantization (e.g., int2) or platform ports.

Repository layout (key files)

c/glm.c          # core engine (forward, MoE streaming, MTP)
c/st.h           # safetensors reader, no mmap
c/tok.h          # byte‑level BPE tokenizer
c/coli           # CLI (chat, bench, convert, info)
c/iobench.c      # disk micro‑benchmark used by the engine
c/convert_fp8_to_int4.py  # offline FP8→int4 converter

License and model terms

colibrì is released under Apache 2.0. The GLM‑5.2 weights are provided by Z‑ai under the MIT license.

Sources

Related

  • Project
    JustVugg/colibri

    A pure C inference engine that enables running frontier MoE models (up to 2.8T parameters) on consumer hardware by treating VRAM, RAM, and disk as a unified memory hierarchy.

  • Project
    kacper-daftcode/vLLM-Moet

    A patched vLLM runtime for NVIDIA Blackwell GPUs that uses 2-bit quantization and tiered NVMe offloading to serve massive MoE models like GLM-5.2 and DeepSeek-V4-Flash on consumer hardware.

  • Project
    zai-org/GLM-5

    GLM‑5 is an open‑source series of very large LLMs (320 B‑744 B total parameters) optimized for multimodal input, 1 M‑token contexts, and long‑horizon agentic tasks. It introduces hybrid sparse/linear attention, IndexShare, and a custom RL engine (slime) to achieve frontier‑level performance on coding and planning benchmarks while offering cheap FP8 inference. Models are downloadable from Hugging Face/ModelScope and can be served with SGLang, vLLM, KTransformers, etc., with optional ‘reasoning_effort’ control.

  • Dispatch
    Soup Enables Fine‑Tuning an 8B Model on a 4 GB Laptop GPU via Layer Streaming

    Soup lets you fine‑tune an 8B parameter LLM on a 4 GB laptop GPU using layer streaming and QLoRA, achieving bit‑exact results with minimal overhead.

  • Dispatch
    Qwen 3.6 27B: The New Sweet Spot for Local LLM Development

    Qwen 3.6 27B is emerging as a highly capable local model that balances intelligence and performance, though its hardware requirements spark debate over the cost-effectiveness of local vs. cloud inference.