Kimi K3 on AMD Instinct MI355X: vLLM Serving Matches NVIDIA HGX B300 Throughput per GPU on AgentX
An eight-GPU AMD Instinct MI355X system serving Kimi K3 with vLLM delivers 24.4% more total tokens per second per GPU than NVIDIA HGX B300 at 50 tokens/s/user P90 interactivity on the AgentX benchmark, and 87.4% more tokens per dollar under the benchmark's modeled ownership-cost assumptions. The result matters because it is a software story: MI355X has a lower published dense low-precision matrix peak than B300, so the advantage comes from serving-stack engineering — cache partitioning, MoE collective restructuring, kernel fusion, an AITER assembly attention path, KDA parallelization and draft-depth tuning — rather than from raw hardware throughput.
The headline numbers on AgentX
At 50 tokens/s/user P90 interactivity, AMD Instinct MI355X serves approximately 7,567 total tokens/s/GPU, compared with 6,083 for NVIDIA HGX B300 — a 24.4% throughput advantage. Using SemiAnalysis's modeled TCO of $1.50 versus $2.26 per GPU-hour, that becomes 18.16 million versus 9.69 million total tokens per dollar of TCO, or 87.4% more tokens per dollar. At the 50-token interactivity target the comparison gives 46.6% lower modeled cost per token.
Interactivity in this benchmark is reciprocal P90 response inter-token latency and excludes time to first token. Total tokens include both reused input and generated output.
| Published specification, per GPU | AMD Instinct MI355X | NVIDIA B300 GPU |
|---|---|---|
| Dense low-precision matrix peak | 10.1 PFLOP/s MXFP4 | 13.5 PFLOP/s FP4 |
| GPU memory | 288 GB HBM3E | 288 GB HBM3E |
| Peak memory bandwidth | 8 TB/s | 8 TB/s |
The two accelerators have identical published memory capacity and bandwidth, but B300 has the higher published dense low-precision matrix peak. Reaching this serving performance reflects sustained software engineering by AMD and collaborators across the vLLM community.
What AgentX measures
The public AgentX benchmark replays real coding-agent session traces with synthetic tokens, retaining request lengths, timing and prefix-sharing patterns. The workload offers over 96% theoretical prefix reuse with unlimited cache. Speculative runs use a fixed synthetic acceptance reference.
Because agent turns reuse most of their prefix, the benchmark rewards systems that keep conversation state resident and avoid recomputing it. The public curves compare complete serving stacks; the measurements reported beside each individual optimization come from separate tests under their own stated configurations.
Inside Kimi K3: MLA, KDA and MoE
Kimi K3 combines two kinds of attention and a sparse MoE. A returning agent turn can reuse most of its prefix, but processing a short continuation still touches three kinds of data.
- 24 MLA layers retain a compressed KV history that grows with the conversation. A new query still consults that history.
- 69 KDA layers carry forward state matrices with fixed dimensions, updating them as new tokens arrive.
- MoE layers route each token through a router that selects expert networks. Kimi K3 selects 16 of 896 experts per token, and expert weights are shared across requests.
MLA retains more history as the conversation grows. KDA updates a fixed-dimension state, so retaining earlier checkpoints means keeping additional copies of that state. The optimization work splits into three scopes: across turns (cache reuse and offload), within a model step (expert projections, kernel handoffs, MLA/KDA scheduling), and choosing draft depth.
1. Cache: keep conversation state available for reuse
Decode context parallelism (DCP) reduces duplicate history in GPU memory, and CPU offload preserves reusable state outside it.
Under TP8, each GPU handles 12 of Kimi K3's 96 MLA query heads but stores the entire compressed KV history — tensor parallelism divides the heads without dividing the cache. With eight-way decode context parallelism (DCP8), each GPU stores one eighth of the cached positions. During decode, the GPUs exchange queries, compute against their local history and merge their partial answers using softmax weights. Each partial answer carries normalization information so the merge can weight shards correctly. PR #51705 extends the ROCm AITER DCP path to multi-token verification.
For a 142,000-token conversation, the estimated MLA payload is 24 layers × 142,000 positions × 576 FP8 bytes ≈ 1.96 GB per GPU. DCP8 reduces this to about 245 MB, before allocation overhead. Each GPU processes eight times as many heads against one eighth of the history, leaving the leading attention arithmetic roughly unchanged. This extra division applies to MLA history; KDA state keeps its existing partition across heads.
CPU offload keeps state that no longer fits in HBM
When state no longer fits in GPU memory, the AMD recipe's native vLLM CPU offload preserves MLA history and KDA checkpoints in host DRAM.
A cache hit needs matching resume points. If MLA history reaches token 3,072 but the latest compatible KDA checkpoint is at 2,048, the request must resume at 2,048 and recompute the remaining prefix. Saving the attention-block tail alongside the KDA checkpoint aligns their resume positions (PR #54736). Restoring that state trades a transfer back to the GPU for avoided prefix recomputation.
2. MoE: remove duplicate work across GPUs
On one GPU, the routed-expert output projection returns the complete vector its consumer expects. In the original ROCm path, an earlier reduction and normalization gave all eight GPUs the same latent input, and each held the full weights — so the same projection ran eight times.
The output coordinates can be divided among GPUs, but every rank still needs the complete result. Kimi K3 already has an all-reduce for its shared-expert partials. Each GPU adds only the routed outputs it owns to its shared-expert partial, and the existing all-reduce assembles the complete result with no new collective (PR #51253). The ROCm implementation builds on the corresponding CUDA strategy. Each output coordinate receives its routed contribution exactly once, and the full projection weights remain replicated.
Output throughput rose 3.05–4.91% in the PR's test on eight AMD Instinct MI355X GPUs at TP8, across concurrency 8–128 with random 8K-input/1K-output requests and prefix caching disabled.
Freeing GPU memory to keep more cached state
Loading only the rows each rank uses frees about 3.85 GiB per GPU across 92 MoE layers at TP8 (PR #59591). The fused path's calculation stays the same; the extra space can hold MLA history and KDA checkpoints.
This change landed after the image used for the headline public result. Its separate AgentX experiment found less than 1% higher throughput at concurrency 32 and 48, but 28.9% at 70, where the reported prefix-cache hit rate rose from 74.8% to 80.1%. The larger gain at high concurrency is consistent with cache pressure. The measurements used eight AMD Instinct MI355X GPUs, TP8/DCP8, FP8 KV, 30-minute runs, no CPU offload or speculative decoding; repeated-run uncertainty was not reported.
3. Memory traffic: fuse operations and remove unnecessary copies
Within each GPU, the next place to look is the data passed between kernels.
Kimi K3's single-token KDA path ran three compute kernels and two output copies. Convolution ran before a kernel that already fused Q/K normalization, decay and beta gates, and the recurrent update; output normalization ran afterward. The fused HIP path performs these operations in one launch and writes directly to the destination buffer (PR #50654). Intermediate handoffs stay on chip, while convolution history and recurrent state still go to GPU memory for the next token.
AttnRes mixes earlier layer-block outputs with the running residual. Combining that mix with RMSNorm keeps the temporary mixture in registers (PR #50593). The running residual saved for later layers retains its BF16 rounding; the mixture stays in FP32 through normalization. With AttnRes fusion, output throughput rose 14.2% on eight AMD Instinct MI355X GPUs at TP8, with 100K-input/1K-output requests and prefix caching disabled. Long-prefill latency was effectively unchanged.
Copy inputs only for kernels that use them
The ROCm matrix-multiplication dispatcher packed inputs before choosing a kernel. When it selected the fallback, that kernel read the original input and left the copy unused. Moving packing into the specialized branches removed these copies (PR #58566).
The profile made the waste visible: 69 unnecessary copies per decode step, one per KDA layer. Each moved only 1,792 bytes but took about 4.8 microseconds — launch overhead dominated, so moving those bytes faster would barely help. In the PR's TP8, concurrency-one, six-draft, 384K-context profile, the removed copies account for 0.339 ms of copy-kernel time per step.
4. MLA: faster long-context decoding with AMD AITER assembly
Kimi K3 has twelve query heads per GPU at TP8. Padding to sixteen made AMD AITER's assembly decode path available. In the reported long-context test, time per output token fell from 43.8 to 23.6 milliseconds.
The comparison used TP8 without DCP, at concurrency one. The adapter in PR #50578 copies four query heads, packs them into the sixteen-head layout and discards their outputs. DCP8 presents a different, 96-head shape. This preserves the model's operation because each query head attends independently over cached positions — adding another head does not change the original head's softmax. Only the query is padded in this decode path; the stored compressed KV history stays the same size.
| Input / output tokens | Original Gluon path | Padded assembly path |
|---|---|---|
| 1K / 1K | 22.9 ms/token | 22.8 ms/token |
| 32K / 1K | 43.8 ms/token | 23.6 ms/token |
These are PR-reported mean times per output token on eight AMD Instinct MI355X GPUs, TP8, concurrency one, MXFP4 model weights, BF16 attention and ordinary single-token decode. The short-input case barely changes; the long-input case nearly halves time per output token.
The mechanism is scheduling. In the historical code inspected, a conservative scheduling hint restricted the Gluon path to one workgroup per request, and that workgroup still processed the entire conversation. Padding made AMD AITER's assembly path available, with a different policy for dividing the history into parallel pieces and combining the results — the inspected assembly policy selected sixteen partitions. History partitions are work assigned to workgroups within one GPU; KV storage is unchanged.
5. KDA: parallelize prefill and speculative verification
MLA can split a retained history among workgroups. KDA presents a harder scheduling problem: each token depends on the preceding state. Fused prefill in PR #54038 prepares each token group's combined transformation before its incoming state is known.
KDA does this with matrix transformations. An ordered scan of the group summaries determines each group's starting state; the groups then replay their tokens in parallel. Token order stays intact, but each group no longer waits for the preceding group's replay. Grouping adds a second walk and expands the fused chunk path from two launches to four. It helps when few sequences or heads expose too little parallel work; otherwise, the extra passes can cost more than they save. Within each group, the recurrent walk keeps state in matrix-accumulator registers between chunks, avoiding a global-memory write and reload.
Fused KDA delivered 5.3–5.6% more input throughput than Triton at 4K and 16K input lengths in a separate three-backend comparison: eight AMD Instinct MI355X GPUs, TP8, concurrency 32, prefix caching disabled and one-token outputs. This measures the complete backend.
Smaller tiles for speculative verification
Short verification batches need a different split. At TP8, each GPU has 12 KDA heads with 128 value columns. Splitting each head into four 32-column tiles launches only 48 workgroups on 256 compute units. PR #58045 switches to four-column tiles, raising the count to 384. Each tile owns different value columns and walks the same token sequence in order.
In a separate graph-replay kernel test on one AMD Instinct MI355X GPU (TP8 dimensions, one sequence, four verification positions), time fell from 16.20 to 5.61 microseconds per layer (2.89×). This includes narrower tiles, single-wave workgroups, uniform-length specialization and hoisted token-invariant work. Ragged batches retain a dynamic path; tests compare outputs and recurrent state.
6. Speculative decoding: choose how many tokens to draft
Faster verification can change how many draft tokens are worth checking. The useful rate is committed output divided by the full draft–verify–commit cycle time.
Under AgentX's fixed acceptance reference, three drafts give 3.00 expected committed tokens per cycle and four drafts give 3.36, including the recovery or bonus token — 12% more committed output per cycle. Four drafts therefore improve throughput only if the complete cycle takes less than 12% longer than three drafts. That 12% threshold is a cost budget; complete cycle growth was not measured in the post.
The public AMD recipe uses three to seven drafts with DCP1 and disables speculation at the larger DCP8 concurrency settings.
Technical note: precision and on-chip memory in the KDA prologue
The KDA prologue computes the inverse of the 64×64 within-chunk dependency matrix, and later matrix products use this transformation to account for token dependencies. The inverse's intermediate values can grow and cancel, so rounding them too early loses information the final result needs. The implementation uses AMD's native FP32 matrix instructions, retaining FP32 inputs and accumulation for the sensitive inverse, and rounds only the completed inverse to BF16 for subsequent products.
FP32 accumulation cannot recover information lost before it starts. Subtracting 256 from 256.5 gives 0.5 when the original FP32 inputs are kept, but 0 when the inputs are rounded to BF16 first (256.5 rounds to 256 at that magnitude). These toy values illustrate why the inverse uses native 16×16×4 FP32 matrix instructions, retaining both input and accumulation precision.
The prologue also has a local-memory budget. AMD calls its on-chip shared scratch LDS. Q and K tiles reuse storage because they are needed at different times, and the triangular input stores only its ten occupied 16×16 blocks. The workgroup requests 79.5 KiB, which rounds to 80 KiB — half of an AMD CDNA 4 compute unit's 160 KiB LDS budget. Allocation rounds to 1,280-byte units, so two 80 KiB allocations fit while two 81.25 KiB allocations do not. This is a capacity calculation; registers and other resources can impose additional limits.
On the MoE side, a separate technical note covers precision: the A4W4 integration in PR #53940 changes activations from eight bits to four while weights remain four-bit. Total traffic also includes BF16 intermediate outputs and expert-tile padding, since a substantial server batch can still leave few rows per expert.
Benchmark configuration and caveats
Serving configurations at the comparison targets:
| P90 interactivity (tokens/s/user) | AMD Instinct MI355X | NVIDIA HGX B300 |
|---|---|---|
| 30 and 50 | SimpleCPU offload; DCP8 / no drafts to DCP1 / three drafts | Mooncake offload; DCP8 |
| 150 | GPU-only; DCP1; five to seven drafts | Mooncake offload; DCP8 |
Both systems use eight GPUs, TP8, FP4 weights and FP8 KV. Values interpolate within each published offload or GPU-only series at matching interactivity; concurrency, DCP and draft depth can vary between the bracketing runs.
The submissions use vLLM 36768d1bf (AMD) and 5894fdf98 (NVIDIA). Configured host-cache budgets are 224.875 and 281 GB per GPU, respectively. Replay uses the cc-traces-weka-062126 dataset and a pinned AIPerf harness, with warmup excluded and one hour of profiling under fixed synthetic acceptance curves. The downloaded dataset revision is unrecorded, and closed-loop replay can produce different completed request mixes.
The cost model uses SemiAnalysis's large-hyperscaler ownership model with published GPU cost constants of $1.50 per GPU-hour for AMD Instinct MI355X and $2.26 for NVIDIA HGX B300. Total tokens/$ = total tokens/s/GPU × 3,600 ÷ modeled $/GPU-hour. Some capital and utilization assumptions remain unpublished.
The snapshot is SemiAnalysis InferenceX data retrieved October 9, 2026, with submissions dated September 30, 2026 (AMD) and August 16, 2026 (NVIDIA). PR benchmarks use their stated workloads, and PR #59591 is a separate experiment run after the headline image.
What to optimize next
Ongoing prefill/decode disaggregation work adds another place to look: worker balance and whether transfers finish before their data is needed.
For another model or workload, the post recommends starting with the deployed request shapes and making three checks:
- Compare ranks. Are several GPUs computing the same result before a collective that could assemble it?
- Follow buffers. Does each temporary tensor have a consumer on the selected path? Can the data stay on chip?
- Count independent work. Do the actual heads, tokens and tiles produce enough workgroups to spread the load?
Measure the complete request at the response speed you need, and check both outputs and saved state. A faster kernel matters when it shortens the work holding up the next token.
The Kimi K3 effort spans more than 50 merged AMD/ROCm-related vLLM PRs from AMD, Embedded LLM and the wider community, alongside AITER development, integration, review and benchmarking. Contributions cover attention, recurrence, MoE, quantization, graph execution, cache support and platform integration.