vLLM Triton Attention Backend Deep Dive

vLLM has introduced a Triton-based attention backend to provide high-performance, portable attention kernels across diverse GPU hardware. This backend allows vLLM to maintain a single codebase for attention operations that runs efficiently on NVIDIA, AMD, and Intel GPUs, reducing the maintenance burden of hardware-specific kernels while matching the performance of specialized implementations.

Performance Portability via Triton

Maintaining hundreds of specialized kernels for every GPU architecture (such as NVIDIA Hopper, Blackwell, and AMD MI300) is impractical. vLLM utilizes Triton, a domain-specific language that allows GPU kernels to be written in Python and compiled into efficient code for multiple platforms.

Triton uses a tiled programming model where developers define logical tiles of computation. The Triton compiler and autotuner then map these tiles to hardware-specific execution layouts. This abstraction enables the backend to be hardware-agnostic while still allowing for low-level optimizations.

The Triton Attention Backend Architecture

To isolate attention implementations from other components like linear layers, vLLM uses an attention backend abstraction layer. The Triton attention backend is native to vLLM, depends only on PyTorch and Triton, and is implemented entirely in Triton.

Usage Scenarios

The Triton backend is deployed in the following scenarios:

  • Default for AMD GPUs: It is the primary backend for AMD GPUs running on ROCm.
  • Intel XPU: Used for float32 operations, as FlashAttention does not support fp32 on this platform.
  • Specific Model Features: Supports ALiBi sqrt (used by StepFun audio models), sink tokens, and GPT-OSS behavior, particularly on pre-Hopper NVIDIA GPUs (e.g., A100).
  • Specialized Requirements: Handles models with small head sizes, encoder/decoder attention, and multimodal prefix attention.
  • Fallback Mechanism: Serves as a general fallback if FlashAttention, FlashInfer, or other dependencies are unavailable.

Technical Implementation of Paged Attention

Paged attention optimizes memory by paging the KV cache. The kernel processes query tokens, iterates over query and KV heads, and traverses the paged KV cache to compute attention scores.

Q Block Optimization

To maximize the utilization of tl.dot (Triton's matrix multiplication), the backend uses "Q blocks." Because KV cache page sizes constrain tile sizes on the KV side, the kernel groups multiple query tokens and heads into a single work item (a Q block) to increase parallelism and cache reuse, particularly for Group Query Attention (GQA).

Parallel Tiled Softmax (3D Kernel)

While Q blocks benefit prefill workloads, decode workloads process only a single query token. To optimize this, vLLM implements a "3D kernel" using parallel tiled softmax. This approach splits the KV cache traversal across multiple kernel instances. Each instance computes partial results, which are then reduced via a second kernel launch to produce the final output.

Persistent Kernels and CUDA Graphs

CUDA graphs reduce launch overhead by recording fixed execution graphs. However, standard attention kernels often use variable launch grids based on batch size and sequence length, which leads to inefficiencies when replayed via CUDA graphs (e.g., wasted work or underutilization of streaming multiprocessors).

To solve this, vLLM developed persistent kernels. Instead of variable grids, a fixed number of kernel instances—equal to the available compute resources—are launched. These instances dynamically read metadata from GPU memory to determine their workload, ensuring constant launch grids and efficient CUDA graph reuse.

Benchmarking and Results

Benchmarks from late 2025 demonstrate that the Triton backend achieves high efficiency with significantly less code complexity than specialized alternatives. The Triton paged attention implementation consists of approximately 800 lines of code, compared to roughly 70,000 lines for FlashAttention-3.

Performance Metrics (Llama 3.1 8B, 500 input tokens, batch size 1):

  • NVIDIA H100: Achieved 100.7% of the performance of FlashAttention-3 for long decode requests.
  • AMD MI300: Achieved a speedup of approximately 5.8× over previous implementations.

Future Directions: Helion

Experimental work has begun with Helion, a new domain-specific language from the PyTorch team described as a higher-level Triton or tiled PyTorch. A simplified paged attention kernel implemented in Helion has shown promising early results and is currently available as a draft pull request in the vLLM repository.

Sources