DeepSeek V4 Flash-0731 on a Single AMD MI300X: Performance and Configuration

Introduction

The DeepSeek V4 Flash-0731 model runs successfully on a single AMD MI300X GPU without weight quantization or offload, achieving 168.6 tokens per second in single-stream decode and supporting up to 64 concurrent streams.

Key Results

The production configuration delivers:

  • Single-stream decode: 168.6 tok/s (median per-stream, DSpark-7)
  • Prefill throughput: 7.9–8.5K tok/s with tuned kernels
  • 8 concurrent streams: 542 tok/s aggregate (90.3 tok/s median per stream)
  • 64-stream burst: 830 tok/s aggregate without OOM or engine errors
  • Validated context length: 256K tokens (architecture supports 1M)
  • Model weights occupy 156.67 GiB of HBM, leaving room for a 20 GB GPU KV cache and 96 GiB CPU tier.

Why MI300X?

The AMD MI300X's 192 GB HBM3 capacity and 5.3 TB/s memory bandwidth enable a single-GPU deployment of the 304B-parameter DeepSeek V4 Flash-0731, avoiding the need for PCIe weight streaming or layer offload. This configuration leverages the MI300X's approximately half the list price of an H100 SXM5 while providing 2.4× the HBM capacity.

Technical Adjustments for MI300X

Running DeepSeek V4 Flash-0731 on MI300X required correctness fixes for its FP8 format and MoE routing, followed by performance tuning. Key adjustments include:

  • Correcting the MXFP4 bitmatrix routing kernel to mask padding lanes by logical block size (not global tensor bound) to prevent routing corruption.
  • Adapting the FP8 cache writer for MI300X's FNUZ format (vs. OCP standard) via a 16×16 preshuffle and float8e4b8 with FP8_MAX=224.0.
  • Adding AITER GEMM tuning tables for gfx942 shapes and OGS geometry overrides for MXFP4 experts.
  • Implementing a hybrid KV strategy with 20 GB GPU cache (fp8_ds_mla) and 96 GiB CPU offload, including a load-path fencing fix for CPU→GPU KV restores.

Repository Contents and Deployment

The repository provides a digest-pinned production stack comprising:

  • A Docker Compose file (compose.yaml) for vLLM ROCm nightly and Caddy proxy.
  • SHA-256-pinned file overlays and unified diffs for correctness patches.
  • AITER tuning tables (CSV) for recurring gfx942 GEMM shapes.
  • An entrypoint script to clear stale CPU-KV mmaps. Deployment requires one MI300X (gfx942), ~235 GiB RAM, and ~500 GB disk. After verifying overlays with SHA256SUMS, starting the stack with docker compose up -d initializes the model and captures CUDA graphs, with a healthy startup showing 156.67 GiB model load and a warmed HBM high-water mark of ~204.5 GB.

Performance Analysis

Optimizations in the configuration yield:

  • Tuning 21 recurring A8W8 GEMM shapes for gfx942: +42–62% single/double-stream decode and +10–35% at 8–64 streams.
  • Fused SiLU and fast DeepSeek routing: native C1 decode increased from 34.5 to 56.6 tok/s (+64%); routing kernel latency reduced from 42.6 to 11.9 µs/layer.
  • BLOCK_H=64 sparse-prefill tile: prefill throughput reached 7.9–8.5K tok/s, reducing sparse-attention trace from 317 to 142 ms per request.
  • Static K=7, probabilistic drafting with block rejection, and causal verification: 119.5 tok/s single-stream with correct output.
  • A 2,048-token scheduler budget and 1,024-token long-prefill cap: reduced TTFT for short requests behind a 52K prefill from 8.2 s to 0.5 s.
  • The 20 GB GPU KV cache and 96 GiB CPU tier provide 1.93M-token length-equivalent capacity, admitting seven 256K requests concurrently.

Final concurrency sweep with ~400-word prompts (temperature=1.0, top_p=0.95):

  • 1 stream: 126.2 tok/s aggregate, 168.6 tok/s median per-stream, TTFT p50 1.026 s
  • 2 streams: 145.4 tok/s aggregate, 152.7 tok/s median per-stream, TTFT p50 0.939 s
  • 4 streams: 316.8 tok/s aggregate, 108.6 tok/s median per-stream, TTFT p50 0.369 s
  • 8 streams: 542.3 tok/s aggregate, 90.3 tok/s median per-stream, TTFT p50 1.027 s
  • 64 streams: 830.2 tok/s aggregate, 16.4 tok/s median per-stream, TTFT p50 2.190 s

Production Considerations

  • HBM headroom is critical: the warmed high-water mark is 204.5 GB of 205.8 GB total. Increasing the KV pool beyond 20 GB risks out-of-resources errors during graph capture.
  • The CPU KV tier stores evicted prefix-cache entries (not weights), using ~103 GiB in /dev/shm via native offloading; the entrypoint removes stale mappings after crashes.
  • The 1,664-token scheduler warning is expected due to DSpark-7 reserving draft slots; increasing the budget reduces usable KV capacity.
  • Kernels require warmup: the first prefill takes 5.3 s for 8.9K tokens, while subsequent runs take 1.7 s. An uncached prefill should run before admitting traffic.
  • Correctness validation includes two-turn tool-calling, BFCL subset, OpenCode tool-schema checks, and 380K-token needle recall on both native and DSpark paths.

Community Insights

Hacker News discussion highlighted:

  • The MI300X is typically available only in OAM modules or as part of 8-GPU boxes, not as a single standalone unit (per @majke).
  • Performance (168.6 tok/s single-stream) is below the 15k tok/s/gpu reported in DeepSeek's DSpark paper for H800, suggesting further optimization potential (per @xorfish).
  • The MI350P PCIe card (144 GB HBM) is an alternative for single-card use, though with less memory (per @Tepix).
  • The context window reduction from 1M to 256K tokens is a practical tradeoff for single-GPU deployment, with quality remaining acceptable for many use cases (per @WhitneyLand).
  • Positive feedback noted the repository's usefulness and referenced related work on 2xMI300X setups (per @fergusfinn).

Conclusion

This configuration demonstrates that the full-precision DeepSeek V4 Flash-0731 can be served efficiently on a single AMD MI300X, leveraging its high memory capacity to avoid quantization. While single-stream throughput trails NVIDIA H800 results, the MI300X offers a cost-effective option for multi-stream workloads, with validated stability up to 64 concurrent streams and a 256K context window.

Sources