MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks

DeepSeek-V4.1-Flash-1017 EXL3 on 2x DGX Sparks with TensorFold

What this project is

This repository is a recipe and set of scripts for running a specific large language model — DeepSeek-V4.1-Flash-1017 — on two NVIDIA DGX Sparks (specialised computers with 128 GB of unified memory each). The goal is to serve the model through an OpenAI-compatible API with a very large context window (up to 1,048,576 tokens), support for image input, tool calling, and optional "thinking" mode, while using speculative decoding (a technique that speeds up text generation) and a compressed version of the model (2.9 bits per weight) to fit on the hardware.

How it works

The two Sparks are linked by a high-speed network (ConnectX-7 ports). The model is split across both machines (tensor parallelism) so each handles half the computation. The engine is TensorFold, a native Zig-based inference engine. The model weights come from a quantised checkpoint on Hugging Face, and additional unquantised tables (called Engram's tables) are used for certain parts.

A single command — ./start.sh on the first Spark — sets up both machines (downloads the model, builds the Docker image, prepares the weight files) and then starts both ranks (the two processes that run the model). ./stop.sh stops them. The setup can be customised with environment variables (e.g., to enable an "abliterated" mode that removes safety filters, or to use a different quantisation pack).

Performance

The README includes measured performance numbers from a real pair of Sparks. For example:

  • Decode speed (generating new tokens): about 75–110 tokens per second for a single stream, and up to 130–155 tokens per second when handling four concurrent streams.
  • Prefill speed (processing a long prompt): about 2,000–2,200 tokens per second for prompts up to 256K tokens.
  • A needle-in-a-haystack test (finding a hidden phrase in a 479,012-token prompt) succeeded 3 out of 3 times.

These numbers depend on the prompt type (prose vs. code) and on the hardware being properly configured.

Requirements

To use this project you need:

  • Two DGX Sparks (or equivalent GB10 systems) with 128 GB unified memory each, and nothing else large running on them.
  • Same GPU clock ceiling on both Sparks (otherwise the slower one limits performance).
  • A direct ConnectX-7 link between the two machines, with RoCE v2 enabled and a private subnet.
  • Key-based SSH from the first Spark to the second.
  • Docker with the NVIDIA container runtime on both Sparks.
  • Sufficient disk space: about 505 GB on the head (first) Spark and 310 GB on the worker, plus space for the Docker image.
  • NFS (network file system) so the worker can read the model files from the head, or alternatively a local copy on each machine.

Additional details

  • The project includes engine patches that fix known issues (e.g., preventing a long prompt from freezing other streams, and enabling image-prompt caching).
  • There is an optional systemd service for supervising the two ranks.
  • The README documents known limitations, such as memory constraints on the head Spark and the need to keep the context window at 524,288 tokens or less on a head that runs other software.

In short, this is a well-tested, performance-oriented setup for running a state-of-the-art language model on two powerful edge devices, with clear instructions and measured results.

Related

  • Project
    starbaser/ccproxy

    A transparent network interceptor for LLM tooling that allows for the inspection, transformation, and cross-provider routing of AI API traffic.

  • Project
    NVIDIA-AI-Blueprints/llm-router

    An experimental multimodal router that analyzes user prompts to recommend the optimal LLM or VLM based on trade-offs between accuracy, speed, and cost.

  • Project
    mlco2/ecologits

    EcoLogits is a Python library that estimates the energy consumption and CO₂‑equivalent emissions of generative‑AI API calls (OpenAI, Anthropic, Cohere, Google‑GenAI, HuggingFace, Mistral AI, etc.). It wraps the provider’s client, looks up model‑specific emission factors, and attaches an `impacts` field to the response containing mean energy (kWh) and GWP (kg CO₂‑eq). Install via `pip install ecologits` (optionally with `[openai]` etc.), call `EcoLogits.init`, then use the provider as usual; the library returns the impact metrics alongside the normal response.

  • Project
    xiaol/wkvm

    A state-native inference hypervisor for linear and hybrid-linear models that replaces paged KV caches with fixed-size state slots for efficient session management.