kacper-daftcode/vLLM-Moet

What it solves

It enables the serving of massive frontier Mixture-of-Experts (MoE) models—such as GLM-5.2 (753B) and DeepSeek-V4-Flash (159B)—on consumer or workstation-grade Blackwell GPUs (like the RTX 5090 or RTX PRO 6000) where these models would normally exceed available VRAM.

How it works

The project uses a combination of quantization, tiered memory management, and custom kernels to fit large models on limited hardware:

• 2-bit Expert Compression: Routed experts are compressed to a sign-symmetric 2-bit codebook using hand-written SM120 SASS kernels. To maintain quality, an FP4 "delta cache" and a confidence gate surgically restore precision where needed.
• Tiered Expert Residency: When models exceed VRAM, the system moves the 2-bit base to pinned host RAM, and further down to NVMe pack files. The GPU then acts as an expert cache, fetching missing experts from RAM or disk and replaying the computation step to ensure bit-identical results.
• Optimized Serving Base: It patches vLLM v0.24.0 to fix SM120 compatibility issues and introduces an NVFP4 KV cache to reduce memory usage per token (352 B/token).
• Speculative Decoding: Implements MTP (Multi-Token Prediction) speculative decoding to increase throughput, even under pipeline parallelism.

Who it’s for

Developers and researchers with Blackwell-architecture GPUs who want to run state-of-the-art, trillion-parameter scale MoE models on a limited number of GPUs without sacrificing significant precision or correctness.

Highlights

• Massive Model Support: Runs GLM-5.2 (753B) on just two RTX PRO 6000 cards.
• NVMe Offloading: Uses pack files and pinned-RAM arenas to bypass host RAM limitations and speed up boot times via a persistent quantization cache.
• High Performance: Achieves high decode speeds (e.g., 105 tok/s for GLM-5.2 on 4x PRO 6000) using custom SASS kernels and MTP.
• Precision Recovery: Uses a confidence gate to re-run low-confidence steps at FP4 precision, correcting artifacts caused by 2-bit quantization.