MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark

What it solves

This project provides a self-contained recipe and deployment scripts to serve the Mia-AiLab/Qwen3.8-Flash-Next-NVFP4 vision-language model on a single DGX Spark machine. It specifically addresses the challenge of fitting a large (99 GiB) checkpoint into the machine's unified memory while maintaining high performance for text, images, and video processing.

How it works

The system uses vLLM as the inference engine, with the PLE table offloaded and memory-mapped to optimize memory usage. It employs several performance optimizations, including:

• Speculative Decoding: Uses MTP (Multi-Token Prediction) with a reduced-vocabulary drafter to increase decode speed.
• Memory Management: Implements a host-side memory cap (HOST_RESERVE_GIB) to prevent system instability and memory leaks.
• KV Cache Optimization: Supports FP8 KV cache to increase the number of tokens that can be stored in memory.
• Multimodal Support: Integrates a 27-layer vision tower, allowing the model to process images and video out of the box using OpenAI-compatible API shapes.

Who it’s for

Developers and researchers with access to DGX Spark hardware who want to deploy a high-performance, multimodal vision-language model with minimal configuration overhead.

Highlights

• Multimodal Capabilities: Native support for text, images, and video (temporal order recognition verified).
• High Throughput: Achieves up to 162.9 aggregate tokens per second across 8 streams in specific configurations.
• Optimized Prefill: Includes a PLE page-fault prefetch mechanism to reduce time-to-first-token (TTFT).
• Flexible Configuration: Allows easy tuning of context length, speculative depth, and memory budgets via environment variables.