Kimi K3 Deployment on AMD MI355X: Performance and Cost Analysis
The AMD MI355X provides a superior performance-per-dollar ratio for serving the 2.8T parameter Kimi K3 model, achieving 952 tokens per second (tok/s) per node. While NVIDIA's B300 maintains a lead in absolute aggregate throughput, the MI355X's lower cost per GPU-hour makes it a more cost-efficient alternative for frontier-scale models that exceed the memory capacity of a single B200 node.
Hardware Comparison and Throughput
The Kimi K3 model's size (2.8T parameters) requires over 1.5TB of VRAM before allocating KV cache for 1M tokens of context. This memory requirement makes the MI355X and B300 viable options due to their 288GB VRAM per GPU, whereas a single B200 node (8 GPUs) cannot fit the model weights and KV pool, necessitating a multi-node TP16 deployment.
Performance benchmarks on a 1,024-token input and 400-token output workload show the following results:
| Metric | 8× MI355X (TP8) | 2×8 B200 (TP16) | B300 (TP8+DCP8) |
|---|---|---|---|
| Decode tok/s per stream | 118 tok/s | 90 tok/s | 172 tok/s |
| Peak aggregate | 952 tok/s | 498 tok/s | 1,568 tok/s |
| Peak aggregate per GPU | 119 tok/s | 31 tok/s | 196 tok/s |
| Peak aggregate per $/GPU-hr | 48 tok/s/$ | 7 tok/s/$ | 33 tok/s/$ |
Pricing based on $2.50/GPU-hr for MI355X, $6.00 for B300, and $4.25 for B200.
While the B300 leads in absolute performance (approximately 1.65× the aggregate throughput of the MI355X), the MI355X is approximately 2.4× cheaper per GPU, resulting in a significantly higher performance-per-dollar metric.
Software Optimizations for ROCm
Deploying Kimi K3 on AMD hardware required two primary engineering interventions to reach peak throughput and reduce latency.
Speculative Decoding Fix
To implement speculative decoding using RadixArk’s Kimi-K3-DSpark, Wafer encountered a NameError in the sglang ROCm build. The error occurred because the ROCm build lacked the definition for top_k_renorm_prob, a function used in the dense path of the accept-sampling verifier.
Wafer resolved this by implementing a PyTorch function to handle top-k renormalization (sorting, masked filling, and rescaling). This fix enabled speculative decoding, resulting in a 2.2× performance gain for single-stream and an 18% increase in peak aggregate throughput.
Prefill Optimization
Initial cold prefill performance on the MI355X was significantly slower than the B300 (51s vs 23s for a 172k-token prefill). This was caused by the system falling back to a slow generic Triton attention kernel because the fast AITER MLA prefill kernel failed to load due to a shape mismatch (K3 at TP8 provides 12 attention heads per rank, while AITER expected 4, 8, or multiples of 16).
By zero-padding the head count from 12 to 16 and extracting the real 12 heads from the output, Wafer achieved a 2 - 3× speedup in prefill, bringing the AITER MLA prefill ASM to approximately 13k tok/s steady-state.
Community Critique and Counterpoints
Following the publication of these results, the technical community raised several concerns regarding the methodology and the framing of the data:
- TCO and Pricing: Critics argue that using cloud-rental prices per hour does not reflect the Total Cost of Ownership (TCO) for organizations owning the hardware, nor does it account for power costs.
- Benchmark Validity: Some users questioned whether the 1,024-token input length is a relevant benchmark for modern frontier models.
- Comparison Fairness: Commenters noted that the B300 outperforms the MI355X in every raw performance metric, and that the B200 comparison is disadvantaged by the requirement of a multi-node setup which introduces cross-node all-reduce overhead.
- Model Correctness: There were questions regarding whether the zero-padding optimization for the prefill kernel affected the model's coherence or correctness, though no evidence of degradation was provided.
"B300 is about 46% faster for one stream and 65% faster in aggregate. AMD wins only after Wafer divides throughput // selected cloud-rental prices... Benchmarks are unreproducible, power costs are missing, ROCm was patched..."