DeepSeek V4 Flash runs on a single AMD MI300X – performance, fixes, and deployment guide

TL;DR

DeepSeek V4 Flash runs on a single AMD MI300X GPU at 168 tokens/s median decode (single stream) and ≈ 8 K tokens/s prefill, using the full 304‑billion‑parameter checkpoint with no weight quantization or offload. The performance is achieved by applying a collection of ROCm patches, AITER GEMM tuning tables, and a hybrid GPU‑CPU KV cache strategy.

---

Why the MI300X matters

The Instinct MI300X provides 192 GB of HBM3 and 5.3 TB/s memory bandwidth, roughly 2.4× the HBM capacity of an NVIDIA H100 SXM5. This memory headroom lets the entire 156.7 GB DeepSeek V4 Flash checkpoint reside in GPU memory, eliminating PCIe weight streaming or layer offload. The large KV pool (20 GB on‑GPU + 96 GB CPU tier) supports 2–8 typical concurrent streams and burst loads of up to 64 streams.

"The MI300X has 192 GB of HBM3 and 5.3 TB/s bandwidth, costing roughly half the list price of comparable NVIDIA hardware." – AMD product page

What the repository adds

The GitHub repo ryanzhou/deepseek-v4-flash-mi300x supplies four essential contributions beyond prior work (Fergus Finn’s MI300X bring‑up and Doubleword’s demo):

1. Correctness overlays for the ROCm nightly (vLLM ROCm 0.26.1rc1.dev229+g124154a88.rocm723) that fix FP8 format handling, MoE routing, speculative verification, and CPU‑KV synchronization.
2. Validated serving configuration with DSpark‑7 speculative drafting, block rejection, static K=7, a 2 048‑token scheduler budget, and a 1 024‑token long‑prefill cap.
3. AITER GEMM tuning tables for the gfx942 (MI300X) shapes that were missing from upstream, plus an OGS geometry override for MXFP4 experts.
4. Hybrid KV strategy: 20 GB GPU fp8_ds_mla cache + 96 GB native CPU offload, together with a load‑path fencing fix documented in vLLM issue #47282.

Repository layout (self‑contained overview)

.
├─ compose.yaml            # Production Docker‑Compose stack (vLLM ROCm + Caddy)
├─ Caddyfile.example      # HTTPS proxy template
├─ vllm-entrypoint.sh      # Cleans stale CPU‑KV mmap files before start
├─ SHA256SUMS              # SHA‑256 pins for all runtime artifacts
├─ patches/
│  ├─ *.py                # Full‑file overlays mounted read‑only
│  ├─ diffs/*.patch       # Unified diffs vs. upstream base
│  └─ README.md           # Provenance and regeneration instructions
└─ tuning/
   └─ *.csv               # AITER A8W8 block‑scale tuning tables for gfx942

Key performance numbers (vLLM ROCm nightly 0.26.1rc1.dev229+g124154a88.rocm723, AITER 0.1.19)

Metric	Result
Single‑stream decode (median)	168.6 tok/s
Prefill with tuned kernels	≈ 7.9–8.5 K tok/s (6 988–7 019 tok/s on fresh prompts)
8 concurrent streams	542 tok/s aggregate, 90.3 tok/s median per stream
64‑stream burst	830 tok/s aggregate, no OOM or engine errors
Context window	256 K tokens validated (architecture supports up to 1 M)
Weights in HBM	156.67 GiB (no extra quantization or offload)

Two critical correctness fixes

MXFP4 routing bug

The MoE bitmatrix kernel padded block columns to a Triton block size but masked against the global tensor bound, corrupting routing under load. The overlay replaces the mask with:

mask = (offs_local < BLOCK_SIZE) & (offs_global < nonzero_indx_size)

This also adds fused‑SiLU and fast DeepSeek routing for grouped MXFP4 experts.

FP8 format mismatch

DeepSeek V4 Flash’s Lightning Indexer cache writes FP8 in AMD’s FNUZ E4M3 layout (16×16 tile‑shuffled), while upstream AITER expects the OCP E4M3 layout. The overlay forces float8e4b8 with FP8_MAX=224.0 and applies the required shuffling, preventing a factor‑two scale error on MI300X.

Speculative decoding configuration

The stack uses probabilistic drafting with block rejection (DSpark‑7). Two Gumbel‑noise overlays keep draft‑proposal noise independent from rejection/recovery noise, required only when draft_sample_method=probabilistic.

Production‑grade tuning details

Optimization	Observed effect
Tune 21 recurring A8W8 GEMM shapes for gfx942	+42–62 % decode throughput (single/double stream)
Fused SiLU + fast routing + batch‑sensitive expert tiles	Native C1 decode 34.5 → 56.6 tok/s (+64 %)
BLOCK_H=64 sparse‑prefill tile	Prefill 7.9–8.5 K tok/s; sparse‑attention trace 317 → 142 ms per request
Static K=7 + block rejection + causal verify	119.5 tok/s single‑stream with correct output
2 048‑token budget + 1 024‑token long‑prefill cap	TTFT for a short request behind a 52 K cold prefill drops from 8.2 s to 0.5 s
Hybrid KV (20 GB GPU + 96 GB CPU)	1.93 M‑token length‑equivalent capacity, admits seven 256 K requests

Concurrency sweep (distinct ~400‑word prompts, temperature=1.0, top_p=0.95)

Streams	Aggregate tok/s	Median per‑stream decode	p50 TTFT
1	126.2	168.6 tok/s	1.026 s
2	145.4	152.7 tok/s	0.939 s
4	316.8	108.6 tok/s	0.369 s
8	542.3	90.3 tok/s	1.027 s
64	830.2	16.4 tok/s	2.190 s

Note: DSpark acceptance varies with prompt content; these numbers reflect this specific Docker image and configuration, not universal model benchmarks.

Deployment checklist

1. Hardware – One MI300X (gfx942, 304 CUs, ~192 GiB HBM), AMD driver, Docker Compose, ~235 GiB RAM, ~500 GB disk.
2. Pull pinned runtime – Use the digest‑pinned image vllm/vllm-openai-rocm@sha256:e68d18b2… and download the model revision 7872f01b1d….
3. Verify overlays – Run sha256sum -c SHA256SUMS before first start.
4. Start stack – docker compose up -d; watch logs for successful startup messages (model loading, KV cache size, CUDA graph capture).
5. Warm kernels – Execute an uncached prefill (~8 K tokens) once to initialize kernels; subsequent requests will be faster.
6. Smoke test – Issue a simple completion request via the Caddy‑proxied /v1/completions endpoint.

Production caveats

• HBM headroom is tight – Warm high‑water mark reaches ~204.5 GB of 205.8 GB. Raising --kv-cache-memory-bytes beyond 20 GB can trigger HSA_STATUS_ERROR_OUT_OF_RESOURCES during graph capture.
• CPU KV tier stores only cache entries – --kv-offloading-size 96 --kv-offloading-backend native maps ~103 GB in /dev/shm for evicted prefix‑cache entries; the entrypoint script cleans stale mappings after crashes.
• Scheduler warning – A 1 664‑token scheduler warning is expected because DSpark‑7 reserves draft slots from the 2 048‑token budget.
• Warm‑up latency – First prefill after restart takes ~5.3 s for 8.9 K tokens; subsequent prefills drop to ~1.7 s.
• Correctness testing – The repo includes a validation suite covering tool‑calling, schema checks, and a 380 K‑token needle recall for both native and DSpark paths.

License and provenance

The stack, documentation, and vLLM‑derived overlays are released under Apache‑2.0; the AITER overlay retains its MIT header. The DeepSeek V4 Flash model itself is MIT‑licensed on Hugging Face.

---

Community insights from Hacker News

• A user noted that DwarfStar can run the same model with less memory, likely using quantization, but the repo deliberately avoids quantization to preserve full‑weight inference.
• Another comment clarified that MI300X is typically sold as an 8‑GPU OAM chassis (~250 k EUR), not as a single card.
• Some participants compared the throughput to NVIDIA H800 results (≈ 15 K tok/s/gpu) and suggested further optimizations are possible.
• One contributor mentioned the MI350P PCIe variant (144 GB HBM) can also host DeepSeek V4 Flash because the model fits within 144 GB when using MXFP4 quantization.
• A broader question was raised about the feasibility of inference for frontier models with several‑trillion parameters without quantization; DeepSeek V4 Flash (304 B) demonstrates that a single high‑HBM GPU can handle sub‑trillion‑parameter models.