kacper-daftcode/vLLM-Moet

What it solves

This project enables the serving of massive Mixture-of-Experts (MoE) models—such as GLM-5.2 (753B) and DeepSeek-V4-Flash (159B)—on consumer and workstation-grade NVIDIA Blackwell (SM120) GPUs (e.g., RTX 5090, RTX PRO 6000) where these models would normally exceed VRAM limits.

How it works

The project implements three primary technical strategies to reduce memory footprints and maintain performance:

1. 2-bit Quantization with FP4 Recovery: Routed experts are compressed to a sign-symmetric 2-bit codebook using custom SASS kernels. To prevent quality loss, a "delta cache" and a "confidence gate" surgically restore FP4 precision for critical steps.
2. Tiered Expert Residency: When models are too large for VRAM, the system moves experts to pinned host RAM or NVMe pack files. The GPU then acts as an expert cache, fetching missing experts from host memory or disk and replaying the computation graph to ensure bit-identical results.
3. Optimized Serving Base: It patches vLLM v0.24.0 to fix SM120-specific bugs and adds features like MTP speculative decoding, an NVFP4 KV cache for higher token capacity, and specialized tool/reasoning parsers.

Who it’s for

Developers and researchers with NVIDIA Blackwell hardware who want to run frontier-scale MoE models on limited VRAM without sacrificing significant accuracy or throughput.

Highlights

• Extreme Compression: Runs a 753B model on just two 96GB GPUs.
• NVMe Offloading: Moves expert stores to disk to drastically reduce host RAM requirements.
• High Throughput: Achieves up to 161 tok/s for DeepSeek-V4-Flash on a single RTX PRO 6000.
• Hardware Specific: Specifically optimized for SM120 (Blackwell) architecture with hand-written SASS kernels.
• KV Cache Optimization: Implements an NVFP4 KV cache that increases pool size by 38% over FP8.