MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks

What it solves

This project provides a highly optimized deployment recipe and runtime overlay for serving the GLM-5.3-Flash model on specific high-end hardware (2x NVIDIA GB10 / DGX Sparks). It solves the problem of running this specific model architecture (NoPE MLA) on SM12x architecture, which otherwise crashes in stock vLLM due to incompatible KV cache layouts and missing sparse-MLA kernels.

How it works

The project uses a custom overlay on top of vLLM to implement several critical fixes:

• Sparse-MLA Support: It zero-pads the model's latent dimensions to fit the FLASHINFER_MLA_SPARSE_SM120 kernel, enabling efficient attention on SM12x GPUs.
• Quantization: It serves an EXL3 4bpw quantized version of the model, utilizing fused EXL3 MoE kernels to keep routed experts packed.
• Speculative Decoding: It integrates DFlash2 (k=7) as a speculator to increase decoding speed.
• KV Cache Optimization: It implements a packed fp8_ds_mla KV cache and a "padded slot-share" allocator that allows the draft model and target model to share memory more efficiently.
• Abliteration: Includes an optional runtime hook (ABLIT=1) to apply refusal-direction ablation to the model's weights at load time to bypass safety refusals.

Who it’s for

AI engineers and researchers with access to NVIDIA GB10/DGX Sparks hardware who need to deploy GLM-5.3-Flash with maximum throughput, long context (up to 1M tokens), and OpenAI-compatible API access.

Highlights

• High Throughput: Achieves up to 146.5 aggregate tokens per second across 4 concurrent requests.
• Massive Context: Supports a maximum model length of 1 million tokens.
• Efficient Memory: Uses 4bpw EXL3 quantization, matching official FP8 quality at 54% of the memory footprint.
• Hardware Specific: Optimized specifically for sm_121a cubins and CX7 fabric for tensor-parallel size 2.
• Custom Patches: Includes backports for XGrammar termination and Kpool slot-map clamping to prevent crashes during long generations.