MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark

This repository is a practical recipe for serving a large vision-language model (Qwen3.8-Flash-Next) on a single NVIDIA DGX Spark workstation. It's not a tutorial or link list—it's a working deployment setup with scripts, configuration files, and detailed performance measurements.

What it does:

• Provides a self-contained way to run the Mia-AiLab/Qwen3.8-Flash-Next-NVFP4 checkpoint (99 GiB) on one DGX Spark's 121 GiB unified memory using vLLM.
• Handles the full lifecycle: downloading the checkpoint, starting/stopping the server, and managing memory.
• Supports text, images, and video out of the box.

Key optimizations:

• Offloads the PLE (positional lookup) table to disk and memory-maps it to save GPU memory.
• Uses speculative decoding (MTP) with a reduced draft vocabulary to speed up token generation.
• Includes a memory budget system with safety rules to avoid out-of-memory crashes.
• Offers knobs for context length, KV cache size, and chunk width to trade off speed vs. memory.

Measured performance (from the README):

• Single-stream decode: ~48.7 tokens/s on prose; up to 162.9 tokens/s aggregate with 8 concurrent streams.
• Prefill: ~2,200 tokens/s at 8k context, dropping to ~1,900 tokens/s at 256k.
• Time to health: ~10–12 minutes on first launch.

Notable caveats:

• The README is extremely detailed and technical, with many benchmark tables and configuration notes.
• The project points to a newer "TensorFold" recipe that is recommended instead.
• Performance varies with content type (predictable text decodes faster).

Overall, this is a serious, hands-on project for deploying a large multimodal model on a single high-end workstation, with a strong focus on squeezing out maximum performance through careful memory management and speculative decoding.