DwarfStar 4 (ds4) Local Inference Engine

DwarfStar 4 enables local execution of frontier MoE models through asymmetric quantization and SSD streaming

DwarfStar 4 (ds4), created by Salvatore Sanfilippo (antirez), is a narrow C inference engine optimized for high-memory machines. Unlike generic runners, ds4 is purpose-built for a specific set of model families—including DeepSeek V4/V4.1, GLM 5.x, and Qwen3.8 Flash Next—to ensure end-to-end validation of model layouts and performance.

Core Technical Architecture

Asymmetric 2-bit Quantization

To make massive mixture-of-experts (MoE) models practical on consumer hardware, ds4 employs asymmetric quantization. This technique compresses routed experts while maintaining high precision for critical shared paths, allowing models like DeepSeek V4 Flash (a 284-billion-parameter model) to fit within the memory constraints of high-end workstations without significant "lobotomization" of the model's capabilities.

KV Cache Disk Persistence

To optimize long-context workloads, ds4 treats the KV cache as a "disk citizen." The engine can save long prefixes to an SSD and resume sessions using a prompt hash. This prevents the need for a full re-prefill when restarting a session, significantly reducing latency for returning to long conversations or complex coding tasks.

Hardware Support and Backend

Ds4 is written in C and supports three primary backends:

• Metal: Optimized for Apple Silicon (specifically M-series Max chips).
• CUDA: For NVIDIA GPUs.
• ROCm: For AMD GPUs.

Performance and Hardware Requirements

Memory and Execution Modes

Hardware fit depends on the model size and quantization level. While the project targets high-end consumer hardware (such as DGX Spark or M5 Max), it offers different execution modes based on available RAM:

• Resident Mode: Models run entirely in memory for maximum speed. For example, V4 Flash Q2 is the baseline for high-memory machines.
• SSD Streaming: For models that exceed available RAM (such as V4.1 Q2 on certain configurations), ds4 can stream weights from the SSD.

Benchmarks

On an M5 Max with 128 GB of RAM, ds4 demonstrates high throughput for the Q2 quantization of V4 Flash:

• Short Context (2,048 tokens): 790.2 t/s prefill and 39.4 t/s generation.
• Long Context (65,536 tokens): 398.5 t/s prefill and 27.6 t/s generation.

Integrated Tooling and API

Ds4 provides a unified stack that shares the same model state and cache across three distinct interfaces:

1. CLI (./ds4): For direct chat interactions.
2. Local Server (./ds4-server): Provides OpenAI and Anthropic-style APIs, allowing integration with external coding agents like Codex, Claude Code, and OpenCode.
3. Native Agent (./ds4-agent): Designed for persistent coding sessions.

Community Insights and Extensions

Users and contributors have expanded the utility of ds4 through various forks and pull requests. Notable community developments include:

• Language Bindings: A fork by @neomantra provides shared libraries and Go bindings (ds4go), enabling the engine to be used via FFI in other languages.
• Context Extensions: A community PR has implemented fused TQ to enable context lengths up to 1 million tokens on 128 GB M5 Max machines when using Qwen 3.8 Flash Next.
• Alternative Implementations: Some users have noted that while ds4 is highly optimized for specific frontier models, other tools like llama.cpp or ollama remain more generic, whereas ds4 acts as a "working template" for the biggest current use cases.

"It must be more a working template for the biggest use cases, without trying to cover every possible setup." — Project documentation via GitHub.