antirez/ds4

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

What it solves

DwarfStar is a native inference engine designed to run high-performance open-weight models—specifically DeepSeek V4 (Flash and PRO) and GLM 5.2—on consumer hardware. It addresses the challenge of running large models on machines with limited RAM by providing specialized optimizations, such as SSD streaming for MoE experts and support for aggressive quantization.

How it works

The engine is self-contained and optimized for a narrow set of models rather than being a general-purpose runner. It utilizes several key techniques to maximize performance:

  • Hardware Backends: Supports Metal (macOS), NVIDIA CUDA (including multi-GPU and DGX Spark), and ROCm (AMD Strix Halo).
  • SSD Streaming: For machines with insufficient RAM, it keeps non-routed weights resident while caching routed MoE experts from the SSD, allowing larger models to run at usable speeds.
  • Speculative Decoding: Implements "DSpark," an auxiliary draft model that proposes future tokens to accelerate generation speed.
  • Parallelism: Supports tensor parallelism (e.g., across two MacBooks) and pipeline parallelism to combine the RAM of multiple systems.
  • Quantization: Uses asymmetrical quantization where routed MoE experts are heavily compressed (e.g., 2-bit) while critical components remain high-precision to maintain quality.

Who it’s for

  • Power users and developers with high-end personal machines (e.g., 128GB+ MacBooks, DGX Spark, or Strix Halo systems).
  • Organizations looking to repurpose older CUDA hardware (Ada Lovelace) into multi-user LLM servers.
  • AI enthusiasts who are comfortable using coding agents to customize and optimize the software for their specific hardware.

Highlights

  • Optimized for DeepSeek V4 & GLM 5.2: Narrow focus ensures higher efficiency than general runners.
  • SSD Streaming: Enables the execution of massive models on hardware that would otherwise lack the RAM.
  • Multi-Backend Support: Native performance on Metal, CUDA, and ROCm.
  • DSpark Speculative Decoding: Increases generation throughput for predictable content like code.
  • Distributed Execution: Capability for tensor and pipeline parallelism across multiple machines.

Related

  • Dispatch
    DwarfStar 4: Bringing Frontier-Level AI to Local Hardware

    Antirez introduces DS4, a specialized inference engine for DeepSeek v4 Flash, enabling high-performance local AI experiences on high-end Mac and GPU hardware.

  • Project
    zhongkaifu/TensorSharp

    A native .NET LLM inference engine for GGUF models that supports multimodal generation (text, image, video, audio) and distributed inference across multiple GPUs and machines.

  • Project
    Entrpi/ds4-on-spark

    A high-performance serving stack for DeepSeek-V4-Flash on NVIDIA DGX Spark, featuring continuous batching, speculative decoding, and a memory governor for massive context windows.

  • Project
    ARahim3/mlx-dspark

    A high-performance inference engine for Apple Silicon that uses DSpark and DFlash speculative decoding to accelerate LLM generation speed losslessly.

  • Dispatch
    DeepSeek DSpark: Accelerating LLM Inference by 60-85%

    DeepSeek has open-sourced DSpark, an inference optimization technique that achieves 60-85% faster generation speeds by improving upon speculative decoding.