Colibrì: Running GLM-5.2 744B MoE on Consumer Hardware

Colibrì: Running GLM-5.2 744B MoE on Consumer Hardware

Colibrì enables the execution of the GLM-5.2 744B-parameter Mixture-of-Experts (MoE) model on consumer-grade hardware with approximately 25 GB of RAM. It achieves this by keeping only the dense portions of the model resident in memory and streaming the routed experts from disk on demand.

Architecture: Streaming MoE and Memory Management

Colibrì leverages the architectural properties of Mixture-of-Experts models, where only a small fraction of total parameters are activated per token. For GLM-5.2, while the model has 744B parameters, only about 40B are activated per token, and only 11 GB of those (the routed experts) change between tokens.

Memory Split

  • Resident RAM: The dense components—including attention, shared experts, and embeddings (~17B parameters)—are stored in int4 quantization and remain resident in RAM, occupying approximately 9.9 GB.
  • Disk Storage: The 21,504 routed experts (~19 MB each at int4) are stored on disk, totaling approximately 370 GB.

Loading Mechanism

To mitigate the latency of disk I/O, Colibrì implements several optimization strategies:

  • LRU Cache: A per-layer Least Recently Used (LRU) cache manages expert residency in RAM.
  • Async Readahead: The engine uses WILLNEED to read the next block of experts while the current block is being processed by the CPU.
  • Learning Cache: The engine records expert routing patterns in a .coli_usage file, allowing it to automatically pin the most frequently used "hot" experts in spare RAM upon startup.
  • OS Page Cache: The engine utilizes the OS page cache as a secondary layer of caching.

Technical Implementation Details

Colibrì is written in pure C (approximately 1,300 lines in glm.c) with zero dependencies, avoiding BLAS, Python (at runtime), and GPU acceleration.

Key Features

  • MLA Attention: Implements Multi-head Latent Attention (MLA) with a compressed KV-cache, reducing memory requirements from 32,768 floats per token to 576 floats (a 57× reduction).
  • MTP Speculative Decoding: Uses GLM-5.2's native Multi-Token Prediction (MTP) head. When the head is quantized to int8, it achieves an acceptance rate of 39–59%, resulting in 2.2–2.8 tokens per forward pass.
  • Integer-Dot Kernels: Utilizes AVX2 maddubs for int8 matmuls, which are 1.4–2.5× faster than standard implementations.
  • Weight Absorption: Employs a DeepSeek-style trick for MLA weight absorption during decoding to eliminate per-token k/v reconstruction.
  • Quantization: Supports int8, packed int4, and packed int2 kernels with per-row scales and dequantization-on-use.

Performance and Hardware Constraints

Performance is primarily bound by disk read speeds (the "disk ceiling"). A cold token requires approximately 11 GB of expert reads.

Measured Benchmarks

Hardware Disk Speed Configuration Performance
WSL2, 12-core, 25GB RAM ~1 GB/s Default 0.05–0.1 tok/s
WSL2, 24-thread, 24GB RAM 1.96–2.74 GB/s --topp 0.7 0.11 tok/s
Apple M5 Max, 18-core, 128GB RAM 14.2 GB/s MTP Off 1.06 tok/s
ThreadRipper PRO 5975WX, 128GB RAM 7 GB/s --topp 0.7 0.44 tok/s

Hardware Requirements

  • OS: Linux or WSL2.
  • CPU: gcc with OpenMP and AVX2 support.
  • RAM: Minimum 16 GB.
  • Storage: Local NVMe (ext4) with ~370 GB free space for the int4 model.

Critical Considerations

SSD Wear

Because cold starts involve heavy random reads (~11 GB/token), and the OS page cache may generate writes, heavy usage can accelerate wear on consumer-grade SSDs. Users are advised to monitor drive health.

Quantization Accuracy

While the engine is validated as token-exact against a transformers oracle for the architecture, the specific accuracy loss associated with the int4 quantization of the 744B model has not yet been fully benchmarked due to the extreme time required for a full run on consumer hardware.

Community Insights and Counterpoints

Discussion among users highlights both the conceptual brilliance and the practical limitations of the streaming approach:

"On small-RAM machines the RAM cap, not the disk, is the binding constraint... --topp 0.7 alone bought a clean 1.6× end-to-end speedup."

Some users questioned the practical utility of sub-1 token-per-second speeds, while others suggested that for long-running asynchronous tasks, such speeds remain viable. Technical suggestions from the community include using RAID0 NVMe arrays to increase bandwidth or utilizing Intel Optane memory to bridge the gap between SSD and RAM speeds.

Sources