antirez/ds4

What it solves

DwarfStar provides a way to run high-performance, large language models (LLMs) on consumer-grade hardware, such as MacBooks, NVIDIA GPUs, and Strix Halo systems. It specifically addresses the challenge of running very large models that exceed available system RAM by implementing SSD streaming and distributed inference across multiple machines.

How it works

It is a native, self-contained inference engine optimized for a narrow set of high-quality models (including DeepSeek V4, GLM 5.x, and Qwen3.8 Flash Next) rather than being a general-purpose runner. It leverages GGUF files and incorporates techniques like tensor parallelism via RDMA for multi-Mac setups, pipeline parallelism to combine RAM across systems, and speculative decoding to improve generation speed. The project is built upon the engineering foundations of llama.cpp and GGML.

Who it’s for

Users with high-end personal hardware (e.g., 96GB+ Macs, DGX Spark, or multi-GPU NVIDIA setups) who want to run capable open-weight models locally. It is also designed for developers who use AI coding agents to customize and optimize the software for their specific hardware configurations.

Highlights

• Hardware Optimization: Native support for Metal (Apple Silicon), NVIDIA CUDA (including Ada Lovelace/L40S), and ROCm.
• Memory Management: SSD streaming allows running models larger than available RAM.
• Distributed Inference: Supports tensor parallelism across Macs using RDMA and pipeline parallelism across multiple systems.
• Integrated Tooling: Includes a native coding agent (ds4-agent), an HTTP server for API access, and a capability evaluation tool (ds4-eval).
• Advanced LLM Features: Supports vision encoders for multimodal input, directional steering for output control, and adjustable "thinking" levels for reasoning effort.