Deltafin runs Kimi K3 (2.8T) at 1 token/s on a MacBook Pro using four SSDs

Quick takeaway

Deltafin streams the unpruned 2.8‑trillion‑parameter Kimi K3 model from four SSDs on an M1 Max MacBook Pro and reaches roughly 1 token / second (0.29 tok/s steady‑state) without sacrificing any of the model’s expert routing or output quality.

---

What Deltafin does

Deltafin is a single‑binary Rust runtime that runs the full, never‑pruned Kimi K3 MoE model (2.78 T parameters, ~1.45 TB of expert weights). It streams each expert weight from disk on demand, keeping the attention trunk resident in int8. The binary enforces that K3 itself decides every token; no draft model shortcuts are allowed to alter the final output.

• Exact quality – token‑identical output against the author’s reference prompt.
• No model compression – weights are used at the released MXFP4 precision; only the attention trunk is int8, which upstream already treats as non‑bit‑exact.
• Streaming architecture – one 17.5 MiB file per (layer, expert) is read with pread + F_NOCACHE across four Thunderbolt 5 SSD enclosures.

---

Benchmark results on an M1 Max MacBook Pro

Metric	Value	Interpretation
Steady‑state decode speed	0.2901 token/s (3.447 s/token)	Slight improvement over the previous 0.2847 token/s release (≈1.9 % higher).
Short‑run speed (128‑token)	1.13 token/s	Higher than the long‑run average because the cache is warm.
Median speed on 17‑token benchmark	0.96 token/s	Close to the author’s reference of 0.684 token/s reported upstream.
Time to first token (512‑token prompt)	~6.3 minutes	Prefill is the dominant cost; read‑amplification factor ≈6.2×.

Historical progression shows rapid gains from storage‑path optimisations (e.g., separating demand and prefetch queues, balancing reads across devices). The author notes four specific fixes that together contributed ~43 % of the total speedup.

---

How the storage subsystem works

• Four SSDs are attached via Thunderbolt 5 enclosures; each stores a subset of the expert files.
• Read path: a dedicated thread pool (default 16 threads) issues pread calls with F_NOCACHE to avoid OS page cache pollution.
• Prefetching: the original implementation walked a fixed directory order, causing all reads to hit a single enclosure. Deltafin now dispatches prefetches to the least‑expected‑completion device, improving parallelism.
• Replication: hot experts can be duplicated across two drives; splitting reads across replicas adds ~10 % throughput.
• Instrumentation: the companion ARGODRIVE repo provides a 10 ms per‑device read monitor and a barrier trace that records which drive served each expert.

---

Installation and usage

1. Install the binary

git clone https://github.com/gavamedia/deltafin.git
cd deltafin
cargo build --locked --release

2. Prepare the model

• Full resident model (fastest)

  ./target/release/deltafin setup --full   # downloads ~1.7 TB of weights
  

• Streaming mode (lower disk footprint)

  ./target/release/deltafin setup --stream   # starts with ~215 GB, fetches experts on demand
  

3. Run a prompt

./target/release/deltafin run --chat \
  --prompt "What are the three largest moons of Saturn?"

Add --stats to see per‑token timing, or --max-new N to limit the generated token count.

4. Serve via OpenAI‑compatible API

./target/release/deltafin serve --host 127.0.0.1 --port 8000

The server implements /v1/chat/completions, /v1/completions, and /v1/models with strict input validation and streaming SSE output.

---

Optional Qwen add‑on

Deltafin can load a small Qwen model (0.6 B/1.7 B) that drafts tokens for raw continuations. K3 then verifies the drafts, yielding a 2.7× speedup on a 17‑token completion while preserving identical output IDs. Install with:

./target/release/deltafin setup-qwen

This adds ~4.3 GiB on disk and does not accelerate chat mode.

---

Community insights from Hacker News comments

• Skepticism about practicality – several commenters noted that a 6‑minute prefill makes the system unsuitable for most real‑time use cases.
• Hardware curiosity – users asked whether faster SSDs (e.g., Intel Optane) would improve throughput; the author’s measurements already show a clear drive‑count ladder (57 % → 78 % → 92 % → 100 % of the four‑drive rate when adding drives).
• Potential extensions – a comment suggested using the approach for other MoE models such as GLM‑Flash, indicating broader relevance of the streaming‑expert technique.
• Critique of README – multiple users found the documentation low‑signal, but the author later added detailed performance notes and a catalogue of failed configurations that can help others reproduce the results.

---

What the author still needs help with

1. Expert‑major prefill scheduling – reading each expert once per layer and processing all routed rows together could reduce the 6.2× prefill amplification toward 1×.
2. Cross‑hardware validation – confirming that the observed drive‑ladder scaling holds on non‑Apple silicon platforms.

---

Why this matters

Running a 2.8 T‑parameter MoE model on consumer hardware demonstrates that storage‑centric streaming can bridge the gap between frontier AI research and personal‑compute accessibility. Even though raw decode speed is modest, the experiment uncovers concrete engineering bottlenecks (read‑path contention, prefetch balancing, expert replication) that are applicable to any large‑scale MoE deployment, whether on servers or edge devices.

---

License and provenance

• Deltafin code: MIT license.
• Kimi K3 weights, DSpark checkpoint, and optional Qwen checkpoints retain their upstream licenses.
• The project is independent of Moonshot AI and has no corporate affiliation.