albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4

What it solves

This project provides a set of optimizations to significantly increase the inference throughput of the Qwen3.5-122B-A10B model on NVIDIA DGX Spark hardware. It improves performance from a baseline of 28.3 tokens per second to 52 tokens per second (an 82% increase) without degrading model quality.

How it works

The project employs a multi-step optimization stack:

• Hybrid Quantization: Replaces BF16 shared expert weights with FP8 to reduce memory bandwidth pressure.
• MTP-2 Speculative Decoding: Uses Multi-Token Prediction to generate multiple tokens per step, increasing throughput.
• INT8 LM Head: Implements a patched INT8 Language Model head with Triton autotuning to accelerate the final layer of the model.
• Hardware-Specific Compilation: Compiles vLLM specifically for the SM121 (Blackwell) architecture used in DGX Spark, including a specific CUTLASS blockwise FP8 GEMM patch (PR #38325).
• Context Management: Supports up to 256K context length out of the box, with an optional TurboQuant KV cache compression for higher concurrency.

Who it’s for

Developers and researchers using NVIDIA DGX Spark workstations who need to deploy the Qwen3.5-122B-A10B model with maximum efficiency and high throughput.

Highlights

• 82% Speedup: Increases throughput from 28.3 to 52 tok/s.
• Blackwell Optimized: Custom vLLM build for SM121 architecture.
• Large Context: Native support for 256K context window.
• Speculative Decoding: Integrated MTP-2 support for faster generation.
• Automated Setup: Includes an install.sh script to handle the complex build process, including Docker image creation and model preparation.