DeepSeek V4 Flash on AMD MI300X: Production Deployment and Optimization
The DeepSeek V4 Flash (0731 checkpoint) can be deployed on a single AMD MI300X GPU in production, achieving a median single-stream decode speed of 168.6 tok/s and prefill speeds up to 8.5K tok/s. This deployment is significant because it allows a 304B-parameter model to run on one card without additional weight quantization or PCIe weight streaming, leveraging the MI300X's 192 GB of HBM3 memory.
Hardware Advantages of the MI300X
The AMD MI300X is uniquely suited for single-GPU deployment of large MoE models like DeepSeek V4 Flash due to its high memory capacity and bandwidth.
- Memory Capacity: With 192 GB of HBM3, the MI300X provides 2.4× the HBM capacity of an NVIDIA H100 SXM5. This allows the entire 156.67 GiB model to fit in HBM, eliminating the need for layer offloading.
- Memory Bandwidth: The card offers 5.3 TB/s of memory bandwidth.
- Resource Allocation: The configuration allocates a 20 GB GPU KV pool and a 96 GiB CPU tier for evicted prefix-cache entries, enabling the system to handle bursts of up to 64 concurrent streams.
Critical Correctness Fixes for MI300X
Running DeepSeek V4 Flash reliably on the MI300X requires specific patches to address architectural differences between AMD hardware generations and software bugs in the vLLM/AITER stack.
FP8 Format Incompatibility
DeepSeek V4's Lightning Indexer cache uses FP8. While newer AMD GPUs (MI325X/MI355X) use OCP-standard FP8, the MI300X implements the AMD/Graphcore fnuz variant of E4M3. Using OCP semantics on MI300X can result in scale errors by a factor of two. The production stack implements a fused_compress_quant_cache.fnuz-shuffle.py overlay to ensure correct float8e4b8 bytes and shuffled write offsets.
MXFP4 Routing Corruption
Under high load, the MoE bitmatrix kernel's padding lanes were being masked against the global tensor bound rather than the logical block size. This corrupted the routing matrix, leading to errors in tool-calling and schema adherence on long prompts. The fix involves masking padding lanes by the logical block size: mask = (offs_local < BLOCK_SIZE) & (offs_global < nonzero_indx_size).
Performance Optimizations
The deployment utilizes a variety of kernel tuning and speculative decoding strategies to maximize throughput and minimize latency.
Kernel Tuning and Throughput
| Optimization | Impact |
|---|---|
| A8W8 GEMM Tuning | Tuning 21 recurring shapes for gfx942 increased single/double-stream decode by 42–62%. |
| Fused SiLU & Fast Routing | Native C1 decode improved from 34.5 to 56.6 tok/s (+64%); routing kernel latency dropped from 42.6 to 11.9 µs/layer. |
| Sparse-Prefill Tile | Using BLOCK_H=64 allows prefill to reach 7.9–8.5K tok/s. |
Speculative Decoding
The stack employs DSpark-7 speculative decoding with probabilistic drafting and block rejection. This configuration achieves 119.5 tok/s single-stream output with correct results. To maintain independence between draft-proposal noise and rejection/recovery noise, specific Gumbel overlays are used.
Latency Isolation
To prevent large cold prompts from stalling the system, the configuration implements a 2,048-token scheduler budget and a 1,024-token long-prefill cap. This reduces the Time to First Token (TTFT) for short requests queued behind a 52K prefill from 8.2 seconds to 0.5 seconds.
Benchmarks and Concurrency
Decode Performance
Testing with ~400-word prompts at temperature=1.0, top_p=0.95 shows strong scaling:
| Streams | Aggregate tok/s | Median per-stream decode | TTFT p50 |
|---|---|---|---|
| 1 | 126.2 | 168.6 tok/s | 1.026 s |
| 2 | 145.4 | 152.7 | 0.939 s |
| 4 | 316.8 | 108.6 | 0.369 s |
| 8 | 542.3 | 90.3 | 1.027 s |
| 64 | 830.2 | 16.4 | 2.190 s |
Prefill Performance
Uncached prefill reaches 7.9–8.5K tok/s. Warm recall of 380K cached tokens takes 0.64–2.65 seconds after an initial cold prefill of 120–125 seconds.
Production Deployment Notes
- HBM Constraints: Memory headroom is extremely tight. The warmed high-water mark is 204.5 GB of 205.8 GB. Increasing the KV cache memory bytes may lead to
HSA_STATUS_ERROR_OUT_OF_RESOURCESduring graph capture. - KV Offloading: The system uses a native CPU KV offloading backend, mapping ~103 GB in
/dev/shmfor evicted prefix-cache entries. This is not for weight offloading, but for cache management. - Context Window: While the architecture supports 1M tokens, this production configuration is validated for 256K tokens.
- Kernel Warming: The first prefill after a restart initializes kernels and is slower (e.g., 5.3s vs 1.7s for 8.9K tokens). A warm-up prefill is recommended before admitting traffic.