sybil-solutions/glm-flash-lite

What it solves

Running a giant language model (125 GB) on a single consumer GPU with only 24 GB of memory seems impossible. This project makes it workable by cleverly splitting the model across the GPU's VRAM, system RAM, and fast NVMe storage. It targets the GLM-5.3-Flash model, which has over 12,000 small expert components, and moves the least-used ones off the GPU. The result is that you can get decent speeds (up to about 28 tokens per second) on a regular RTX 3090, without needing a multi-thousand-dollar enterprise GPU.

How it works

The model is a mixture-of-experts (MoE) design: for each token, it only activates 8 of its 12,096 experts per layer. The system keeps the most frequently used experts in VRAM, a middle tier in system RAM (capped by a memory limit like 55 GB or 16 GB), and the rest on NVMe. Two lanes work in parallel: the GPU computes experts in VRAM, while the CPU computes those in RAM. NVMe reads use direct I/O with prefetching to avoid delays. A controller thread manages everything so the GPU never stalls. It also avoids storing duplicate experts in VRAM and RAM, and uses a small ring buffer to temporarily park evicted experts before copying them to RAM. The system reports near-exact output quality (decoding KL divergence around 0.005) and supports a bit-exact mode for full reproducibility.

Who it's for

This is for people who want to run a very large open-weight LLM on a single consumer GPU (specifically an RTX 3090 24 GB) without buying expensive hardware. It's suitable for researchers, hobbyists, and tinkerers who are comfortable with Docker, have a fast NVMe drive, and want to self-host a powerful model. The project also provides a plugin for Omarchy Local AI (a simple UI) and a setup skill that coding agents like Claude Code can use to automatically tune performance on your machine.

Highlights

• Runs GLM-5.3-Flash (125 GB) on a single RTX 3090 24 GB with up to 28.2 tok/s decode when using all-RAM mode (needs ~238 GB free RAM).
• Two NVMe modes with capped RAM (55 GB or 16 GB) still achieve 19.5 and 14.7 tok/s respectively.
• Near-exact output quality (decode KL ~0.005) and a bit-exact mode for exact replication.
• Detailed technical breakdowns and tunables in the docs; a setup skill helps you adapt to your hardware.
• Optional acceleration via an Intel Arc Pro B70 as a second expert tier (experimental).