blazux/qwen3.8-Flash-DGX

What it solves

This project enables the deployment of the Qwen3.8-Flash-Next model (176B parameters) on a single NVIDIA DGX Spark / ASUS GX10 system. It specifically addresses the memory constraint where the model's NVFP4 checkpoint (125 GiB) exceeds the available 128 GB unified memory pool when combined with a necessary KV cache. It also fixes critical bugs in vLLM related to prefix caching and non-deterministic decoding on GB10 hardware.

How it works

• NVMe mmap for Weights: To save memory, the project patches vLLM to serve the large n-gram embedding (PLE) table from NVMe via mmap instead of keeping it in RAM, reducing resident weights from ~125 GiB to ~75 GiB.
• Custom Patches: It applies 12+ patches to the official vLLM image to fix prefix caching (Mamba state bug), non-deterministic top-k in sparse attention, and tool-call parsing errors.
• Hybrid Checkpoint: Offers an optional hybrid layout (NVFP4 experts + fp8 side layers) to increase decoding speed by approximately 20%.
• Optimized Serving: Uses a flash CLI tool to manage Docker builds, weight downloads (via Xet), and serving profiles (e.g., speed, context, default).

Who it’s for

• Developers and researchers using NVIDIA DGX Spark or ASUS GX10 hardware who want to run massive Qwen models with high context (up to 500k-1M tokens) on a single machine.

Highlights

• Memory Efficiency: Reduces weight footprint to 75 GiB, freeing up space for a large KV cache (680k tokens).
• High Performance: Achieves ~34 tok/s single-stream decode and ~2,500–2,800 tok/s prefill on GX10.
• Deterministic Output: Includes a custom deterministic kernel to ensure consistent greedy decoding without sacrificing prefill speed.
• Robust Tool Parsing: Implements guards to prevent the model from accidentally triggering tool calls when documenting syntax inside code blocks.
• Easy Deployment: Provides a streamlined ./flash command for setup, health checks, and serving.