tonyd2wild/DeepSeek-v4-Flash-Vision-Exp-DSpark-1M-NVFP4-KV-2x-DGX-Spark

What it solves

This project provides a production-ready deployment recipe for the DeepSeek-V4-Flash-Vision-Exp multimodal model on DGX Spark clusters. It specifically addresses the lack of a production serving engine for the vision variant, providing the necessary ports and patches to enable native image input and high-throughput inference using vLLM.

How it works

The project implements a serving stack on 2x DGX Spark nodes using Tensor Parallelism (TP=2) and vLLM. It utilizes a custom runtime image and a series of critical bind-mounts to apply patches without rebuilding the container. Key technical components include:

• DSpark Speculative Decoding: Uses a drafter to accelerate token generation, optimized with a specific k=5 configuration.
• NVFP4 KV Cache: Employs an experimental nvfp4_ds_mla KV cache to support massive context windows (up to 1M tokens).
• Vision Port: Integrates a 32-block ViT and aligner directly into vLLM, bypassing the need for VLM proxies.
• Critical Patches: Includes Patch 3 (scheduler fix), Patch 4 (shared-expert loader fix to prevent silent performance drops), and Patch 5 (suppressing stop strings during reasoning segments).

Who it’s for

ML engineers and infrastructure researchers deploying DeepSeek-V4-Flash models on DGX Spark hardware who need high-throughput, long-context multimodal capabilities.

Highlights

• Native Multimodal Support: Direct integration of vision capabilities into the serving engine.
• Massive Context: Calibrated for 1M-token context windows.
• High Throughput: Achieves up to 80.1 tok/s on specific workloads (with Patch 4).
• Silent Failure Prevention: Provides specific scripts and patches to fix a critical bug where the draft's shared expert loads uninitialized, which would otherwise halve decode speed without affecting output quality.