vLLM AMD ROCm Attention Backends Optimization

vLLM has implemented a series of optimized attention backends for AMD ROCm, moving beyond simple porting to architectural co-design. By combining vLLM's orchestration layer with AMD's AITER primitives, the system achieves 1.2-4.4x higher throughput (TPS) for Multi-Head Attention (MHA) and Multi-Head Latent Attention (MLA) workloads on AMD Instinct MI300X, MI325X, and MI355X GPUs.

The Challenge of Mixed Inference Workloads

Production LLM serving involves continuous batching where prefill, extend, and decode tokens are processed simultaneously. Each phase has distinct performance bottlenecks:

  • Prefill: Compute-bound; requires large tile sizes and maximum ALU utilization for new prompts.
  • Extend: A hybrid workload; processes additional prompt-side tokens for requests with partially built KV caches.
  • Decode: Memory-bound; generates tokens one at a time, requiring coalesced memory access and minimal cache fetches.

Because a kernel tuned for one phase often underperforms in another, vLLM utilizes explicit routing to direct each request type to a specialized kernel.

ROCM_AITER_FA: Three-Path Orchestration for MHA

ROCM_AITER_FA is a sophisticated orchestration layer that routes requests through three specialized paths to maximize hardware utilization on CDNA architecture.

Technical Innovations

  • Three-Path Routing: Requests are dynamically categorized into Prefill (using flash_attn_varlen_func for matrix cores), Extend (using chunked attention with LSE merging for 100K+ contexts), and Decode (using AITER's highly optimized memory bandwidth kernels).
  • Batch Reordering: The vLLM Model Runner reorders requests into a [decode:extend:prefill] sequence. This ensures contiguous memory access and eliminates redundant KV cache fetches.
  • Hardware-Optimized KV Cache Layout: A preshuffled KV cache layout aligns memory access patterns with AMD's CDNA architecture. This allows the decode path to call AITER's pa_fwd_asm kernel with zero layout conversion overhead, improving decode throughput by 15-20%.
  • Chunked Context Processing: Long sequences are processed in fixed per-iteration token budgets (~32K tokens), using Log-Sum-Exp (LSE) based merging to maintain numerical stability.

MHA Backend Comparison

Beyond ROCM_AITER_FA, vLLM provides other MHA backends:

  • Unified Backends (TRITON_ATTN, ROCM_AITER_UNIFIED_ATTN): Process all tokens through a single kernel path.
  • Legacy 2-Path (ROCM_ATTN): Uses separate kernels for prefill (Triton) and decode (HIP paged attention). This backend supports Radeon GPUs but may fall back to slower Triton decode kernels for unsupported KV head sizes.

AITER MLA Backends: Optimized for DeepSeek

Multi-Head Latent Attention (MLA), used in DeepSeek and Kimi, compresses the KV cache to 576 dimensions, shifting the performance bottleneck. vLLM provides specialized AITER-based MLA backends to handle this compression.

The Hybrid Processing Strategy

MLA backends use a split strategy based on the processing phase:

  • Prefill/Extend (Non-Absorbed): Attention is computed using standard MHA kernels on the uncompressed representation.
  • Decode (Absorbed): Specialized MLA kernels operate directly on the compressed 576-dim latent space.

Performance Gains from Assembly

The primary performance increase in ROCM_AITER_MLA and ROCM_AITER_TRITON_MLA comes from the mla_decode_fwd assembly kernel. This hand-tuned kernel maximizes HBM3 bandwidth, delivering 1.2-1.6x faster Time Per Output Token (TPOT) compared to the TRITON_MLA baseline.

Performance Benchmarks

Benchmarks were conducted using ROCm 7.0.0 on Qwen3-235B (MHA) and DeepSeek-R1 (MLA) models.

MHA Results (Qwen3-235B)

ROCM_AITER_FA significantly outperforms legacy backends in output throughput (TPS) across all tested hardware:

Hardware ROCM_AITER_FA ROCM_AITER_UNIFIED_ATTN TRITON_ATTN ROCM_ATTN
MI300X (64 req) 1.00x 1.05x 1.30x 3.82x
MI325X (64 req) 1.00x 1.02x 1.19x 4.36x
MI355X (64 req) 1.00x 0.95x 1.08x 3.61x

ROCM_AITER_FA achieves 2.7-4.4x higher throughput than ROCM_ATTN for this model, partly because ROCM_ATTN falls back to Triton decode kernels for unsupported head sizes.

MLA Results (DeepSeek-R1)

AITER MLA backends provide up to 1.5x higher throughput than TRITON_MLA:

Hardware ROCM_AITER_MLA ROCM_AITER_TRITON_MLA TRITON_MLA
MI300X (64 req) 1.00x 0.98x 1.33x
MI325X (64 req) 1.00x 0.98x 1.41x
MI355X (64 req) 1.00x 1.03x 1.52x

ROCM_AITER_MLA is recommended as the default for MLA workloads, particularly on MI355X where it achieves the best Time To First Token (TTFT).

Implementation and Deployment

To enable optimized AITER backends, users should set the following environment variable:

export VLLM_ROCM_USE_AITER=1

With this setting, vLLM automatically selects ROCM_AITER_FA for MHA models (e.g., Llama, Qwen, Mistral) and ROCM_AITER_MLA for MLA models (e.g., DeepSeek, Kimi).

Hardware Support Matrix

GPU Memory Architecture
MI300X 192GB HBM3 gfx942
MI325X 256GB HBM3e gfx942
MI355X 288GB HBM3e gfx950

Sources