MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark

DeepSeek-v4-Flash 0731 recipe for 2x DGX Sparks

What it solves

This project provides a deployment recipe and optimized runtime for running the DeepSeek-V4-Flash-Vision-Exp model on a cluster of two (or three) NVIDIA DGX Spark nodes. It specifically addresses the challenges of managing massive context windows (up to 1 million tokens), optimizing throughput via speculative decoding, and enabling native image support on the Anemll runtime.

How it works

The system utilizes vLLM with Tensor Parallelism (TP=2) and DSpark speculative decoding to accelerate inference. It employs a specialized KV cache (nvfp4_ds_mla) to handle the 1M-token ceiling. The deployment is containerized using Docker and can be configured to either duplicate model weights on each node or mount them from a head node via NFSv4 over a ConnectX link to save storage.

Who it’s for

It is designed for operators and researchers who have access to DGX Spark hardware and want to deploy DeepSeek's Flash Vision model with high throughput, long-context capabilities, and image processing functionality.

Highlights

  • Massive Context: Supports a context ceiling of 1,048,576 tokens.
  • Speculative Decoding: Uses DSpark to increase generation speed.
  • Native Vision Support: Includes a startup hotfix for the Anemll runtime to support images via OpenAI-compatible image_url and <image> tags.
  • Flexible Weight Management: Supports both local weight caching on worker nodes and NFS-based mounting from the head node.
  • Abliteration Support: Optional runtime refusal-direction projection to modify model behavior.
  • Optimized Throughput: Capable of reaching ~160–190 aggregate tokens per second across multiple concurrent streams.

Related

  • Project
    MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark

    A single-node launcher for DeepSeek V4 Flash on NVIDIA DGX Spark, enabling high-throughput serving with speculative decoding and context windows up to 384k tokens.

  • Project
    MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks

    A deployment framework for running Qwen3.8-Flash-Next models across dual-DGX Spark nodes using vLLM, featuring automated kernel patching, multi-node synchronization, and optimized KV cache management.

  • Dispatch
    DeepSeek V4 Flash runs on a single AMD MI300X – performance, fixes, and deployment guide

    DeepSeek V4 Flash can be served on a single AMD MI300X GPU with 168 tok/s decode throughput and 8 K tok/s prefill without quantization, thanks to a set of ROCm patches, AITER tuning, and a hybrid KV cache.

  • Project
    MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark

    A Docker‑based deployment kit for serving the 27‑billion‑parameter Qwen 3.8 model on NVIDIA DGX Spark using SGLang. Includes ready‑made scripts for three speculative decoding engines (EAGLE/MTP, DSpark, DFlash2), benchmark utilities, and easy configuration of context length (up to 1 M tokens), concurrency (up to 10), and quantisation. Designed for engineers who need high‑performance LLM inference on a‑arch64 GPU server.

  • Project
    MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks

    A Docker‑based, OpenAI‑compatible vLLM server for the 4‑bpw EXL3 checkpoint of GLM‑5.3‑Flash, optimized for 2‑4 NVIDIA DGX Spark boxes. It offers 850k‑token context, DFlash2 speculative decoding, and optional speed‑boost or abliteration profiles, delivering 30‑90 tokens/s decode. Configuration is via a single `.env` file; start with `./start.sh` after setting head/worker IPs.