MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks

What it solves

This project provides a multi-node inference setup for the Qwen3.8-Flash-Next model, specifically optimized for deployment across two NVIDIA DGX Spark nodes. It addresses the challenge of running large, quantized models with high memory requirements and long context windows (up to 1 million tokens) by leveraging distributed inference and specific hardware optimizations.

How it works

The system uses vLLM as the inference engine, employing a combination of Tensor Parallelism (TP2), Expert Parallelism (EP), and Multi-Token Prediction (MTP3) to distribute the model across two nodes. It supports multiple quantization formats, primarily NVFP4, but also official FP8.

Key technical implementations include:

• Weight Distribution: Weights are transferred from a head node to a worker node via rsync or shared via NFS to save disk space.
• Custom Patching: The project applies runtime patches to the vLLM image to enable specific features like PLE (Parallel Linear Experts) and MXFP8 support, avoiding the need for full image rebuilds.
• Memory Optimization: It implements an FP8 KV cache patch that significantly increases the available token capacity (up to 3.65M tokens) compared to standard BF16.
• Context Extension: It uses YaRN rope scaling to extend the native 262K context window up to 1M tokens.

Who it’s for

It is designed for users with access to NVIDIA DGX Spark hardware (GB10, 128 GB unified memory) and those looking to deploy high-performance, long-context LLMs using vLLM in a multi-node configuration.

Highlights

• Multi-Node Scaling: Optimized for 2 DGX Spark nodes via ConnectX RoCE/IB.
• Massive Context: Supports up to 1M tokens using YaRN scaling.
• Enhanced KV Cache: Custom FP8 KV cache implementation increases token capacity by ~1.7x over BF16.
• Flexible Weight Management: Choice between local rsync copies or NFS sharing for model weights.
• Abliterated Model Support: Ability to serve a gated, safety-refusal-removed version of the model.