GLM5.2 Inference on AMD MI355X: Achieving 2626 tok/s/node

Wafer has demonstrated that the AMD MI355X GPU can serve the GLM5.2 model at an aggregate throughput of 2626 tok/s/node, achieving a cost-to-performance ratio over twice as efficient as NVIDIA's Blackwell architecture. While NVIDIA typically holds a software advantage with "day-0" support for new models, Wafer's optimizations show that the gap in inference performance is closing as automated kernel and model optimization tools improve.

Performance Benchmarks

The AMD MI355X achieves high throughput and cost efficiency, particularly in prefill-bound workloads. In a test scenario with 20k input tokens, 1k output tokens, and a 60% cache hit rate, the MI355X reached an aggregate throughput of 2626 tok/s/node at 2.4 requests per second (RPS), with a Time to First Token (TTFT) p95 of 2.22s.

This performance represents approximately 80% of the throughput measured on a B200 (which hit 3192 tok/s/node at 3.0 RPS), but at a hardware cost approximately 2.75x lower per GPU.

Throughput Metrics

Sustained RPS Aggregate tok/s/node TTFT p50 / p95 Success
0.5 449 0.59s / 0.60s 100%
1.0 974 0.60s / 0.81s 100%
1.5 1913 0.62s / 1.03s 100%
2.0 1944 0.62s / 1.05s 100%
2.25 2089 0.63s / 1.23s 100%
2.4 (saturation) 2626 0.81s / 2.22s 100%

For single-stream decode, the MI355X achieved 213 tok/s on GLM5.2 using 10k input and 1.5k output tokens, served via TensorWave capacity.

Optimization Strategy

Achieving these results required specific quantization and framework selections to overcome the lack of native "out-of-the-box" support for GLM5.2 on the ROCm stack.

Quantization and Framework

Wafer used AMD Quark to quantize the base bf16 GLM-5.2 model to MXFP4. Benchmarks indicate this quantization is nearly lossless compared to the official FP8 baseline:

Eval FP8 baseline MXFP4 Δ (MXFP4 − FP8)
GSM8K 0.965 ± 0.013 0.955 ± 0.014 −0.010
GPQA-Diamond 0.9217 ± 0.027 0.9026 ± 0.029 −0.019
tau2 macro 0.819 0.834 +0.015

sglang was selected as the inference engine because it provided the least friction for native support and could leverage MXFP4 quantization while maintaining coherence, unlike vLLM (which lacked a working MXFP4 + GlmMoeDsa path) or ATOM (which suffered from output degradation at long contexts).

Technical Fixes for Speculative Decode

To enable speculative decode on sglang for ROCm, Wafer implemented two primary fixes:

  1. Weight Shape Mismatch: The MTP (Multi-Token Prediction) head's shared expert was stored in bf16, but sglang's quantization lookup failed due to a module prefix mismatch (model.decoder.* vs model.layers.78.mlp.shared_experts.*). Wafer resolved this by duplicating the layer 78 entries in the Quark quantization list under the correct decoder name, preventing the init crash.
  2. Cuda Runtime Dependency: Deep speculative decode (depth ≤ 4) was blocked by a fused multi-step metadata kernel that explicitly included <cuda_runtime.h>. This was resolved by adding a #ifdef USE_ROCM guard.

Prefill Optimization

To maximize aggregate throughput, Wafer shifted from a Tensor Parallelism (TP8) configuration to a TP4×DP2 configuration. They also manually tuned the MoE (Mixture of Experts) kernel selection for GLM-5.2's fp4 shapes (model_dim 6144, moe_inter 2048, E=256, topk=8), as the sglang image was defaulting to a slow FlyDSL heuristic fallback.

Community Insights and Counterpoints

While the technical achievement is significant, community discussion highlights several caveats regarding real-world deployment:

"While cool, quantization to FP4 is practically never lossless in actual use. A lot of providers are advertising high TPS on Kimi and GLM, but the models are functionally lobotomized and no longer close to frontier quality."

Other critics pointed out that the performance gains are heavily dependent on a 60% cache hit rate and the use of speculative decoding, which may not reflect all production workloads. There are also calls for more transparency regarding performance-per-watt metrics and the availability of MI355X hardware for rent in the open market.

Sources

Related

  • Dispatch
    Kimi K3 Deployment on AMD MI355X: Performance and Cost Analysis

    Wafer demonstrates that the AMD MI355X provides superior performance-per-dollar for serving the 2.8T parameter Kimi K3 model compared to NVIDIA B200 and B300 GPUs.

  • Dispatch
    vLLM Qwen3.5 Performance Optimization

    vLLM has achieved over 25,000 total tokens per second (TPS) per GPU for Qwen3.5 on GB200 NVL72 systems through Blackwell-optimized kernels, hybrid cache state transfer, and async scheduling.

  • Dispatch
    Serving GLM-5.2 on NVIDIA B300 GPUs with vLLM

    vLLM achieves production SLA compliance for GLM-5.2-NVFP4 on 24 NVIDIA B300 GPUs by implementing P/D disaggregation, speculative padding, and IndexerCache optimizations.

  • Dispatch
    vLLM speculative decoding on AMD GPUs: performance and methods

    vLLM adds speculative decoding to AMD Instinct MI300X/MI355X GPUs, letting a lightweight draft model propose multiple tokens that the target model verifies in a single pass, which can double or more the output-token throughput depending on the draft method, model family, and proposal length.

  • Dispatch
    vLLM AMD ROCm Attention Backends Optimization

    vLLM introduces optimized attention backends for AMD ROCm, delivering up to 4.4x higher throughput for MHA and 1.5x for MLA models on Instinct MI300X, MI325X, and MI355X GPUs.