MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks

Qwen3.8-Flash-Next-NVFP4-vLLM-Dual-DGX-Spark

What it solves

This project provides a specialized deployment framework for running the Qwen3.8-Flash-Next-NVFP4 large language model across two NVIDIA DGX Spark nodes. It solves the complexity of multi-node inference by automating weight distribution, patching model kernels for compatibility, and managing distributed memory resources to maximize KV cache capacity.

How it works

It uses Docker containers to orchestrate a vLLM-based inference server across two nodes connected via high-speed RoCE/IB networking. The project includes scripts to download model weights, synchronize them between the head and worker nodes (either via rsync or a shared NFS volume), and apply specific patches to the model's layers and kernels to ensure they run correctly on the Spark hardware. It also manages memory by evicting page caches to prevent out-of-memory errors and supports advanced configurations like FP8 KV caching to significantly increase the number of tokens the model can handle in its context window.

Who it’s for

This project is for infrastructure engineers and AI researchers who have access to multi-node DGX Spark hardware and need to deploy high-performance, large-context Qwen models. It is intended for users comfortable with Linux, Docker, and distributed systems who require a turn-key solution for multi-node inference.

Highlights

  • Multi-node Optimization: Specifically tuned for dual-DGX Spark setups using Tensor Parallelism (TP2), Expert Parallelism (EP), and Multi-Token Prediction (MTP3).
  • High-Capacity KV Cache: Enables up to 3.65 million tokens of KV cache capacity using FP8 quantization.
  • Automated Patching: Includes automatic patching for FlashInfer kernels and model layers to ensure compatibility with the NVFP4 architecture.
  • Flexible Deployment: Supports multiple checkpoint types, including stock NVFP4, official FP8, and abliterated checkpoints, with options for NFS-based or rsync-based weight distribution.

Related

  • Project
    eugr/spark-vllm-docker

    A collection of Docker configurations and automation scripts for deploying high-performance vLLM inference on single or multi-node NVIDIA DGX Spark clusters.

  • Project
    blazux/qwen3.8-Flash-DGX

    A specialized environment for running the 176B-parameter Qwen3.8-Flash-Next model on a single NVIDIA DGX Spark by offloading embedding tables to NVMe and patching vLLM for memory efficiency.

  • Project
    MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold

    This project provides a ready-to-run recipe for serving the GLM-5.3-Flash language model across two NVIDIA DGX Sparks using TensorFold, delivering an OpenAI-compatible API with a 1M-token context, image/video input, and high throughput at low latency.

  • Dispatch
    vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation

    vLLM provides a high-performance local inference endpoint for NVIDIA DGX Spark, enabling the deployment of large NVFP4 models like Nemotron-3-Super using a unified-memory architecture and OpenAI-compatible API.

  • Project
    MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark

    A deployment recipe for serving the Qwen3.8-Flash-Next vision-language model on a single DGX Spark using vLLM, with memory optimizations, speculative decoding, and extensive benchmark data.