tonyd2wild/DeepSeek-v4-Flash-Vision-Exp-DSpark-1M-NVFP4-KV-2x-DGX-Spark

What it solves

This project provides a high-performance deployment recipe for the DeepSeek-V4-Flash-Vision-Exp multimodal model on a 2-node DGX Spark cluster. It specifically addresses the lack of a production serving engine for this model, providing the necessary ports and patches to enable native image input and speculative decoding within vLLM.

How it works

The project utilizes vLLM as the serving engine, configured for Tensor Parallelism (TP=2) across two nodes. It employs several critical components to optimize performance:

• DSpark Speculative Decoding: Uses a drafter model to predict multiple tokens per step, significantly increasing throughput.
• NVFP4 KV Cache: Implements an experimental nvfp4_ds_mla KV cache to support a massive 1M-token context window.
• Custom Bind-Mounts: Since the model's vision architecture isn't natively supported in standard vLLM classes, the project uses read-only bind-mounts to inject the necessary vision-model files (ds4v_model.py, etc.) and critical bug fixes (Patch 3 and Patch 4) directly into the runtime environment.

Who it’s for

ML engineers and researchers with access to DGX Spark hardware who want to deploy the DeepSeek-V4-Flash-Vision-Exp model with high throughput, large context windows, and native vision capabilities.

Highlights

• Native Vision Support: Implements a genuine port of the 32-block ViT and aligner, avoiding VLM proxies.
• Massive Context: Supports a calibrated context of up to 1,048,576 tokens.
• Speculative Speedups: Achieves high token-per-second rates (e.g., 80.1 tok/s on specific workloads) using DSpark with a configured k=5 draft length.
• Critical Bug Fixes: Includes Patch 4 to fix a silent failure in the DSpark draft shared-expert loader that otherwise halves decode speed.