vLLM Optimizations for Arm CPUs

vLLM has collaborated with PyTorch, oneDNN, and KleidiAI to optimize its serving stack for Arm Neoverse-based servers. These updates significantly improve usability, feature coverage, and performance, reducing the cost and complexity of deploying large language models (LLMs) on CPU-based infrastructure.

Deployment and Feature Enablement

vLLM on Arm CPUs now features improved out-of-the-box usability and broader model support. Key enablement updates include:

  • Simplified Installation: Availability of pre-built wheels and Docker images.
  • Expanded Model Support: Support for GPT-OSS, Whisper, and Qwen 3.5 / 3.6.
  • New Features: Support for chunked prefill, prefix caching, and INT8 W8A8 and INT8 W4A8 inference.
  • Stability: Bug fixes addressing crashes, accuracy, threading, and CPU utilization, alongside better integration with the PyTorch and UXL ecosystems.

Core Performance Optimizations

Performance gains were achieved by addressing bottlenecks across the entire inference stack rather than focusing solely on GEMM kernels.

Memory Allocation with mimalloc

PyTorch's default use of glibc malloc caused high page faults and contention during the repeated tensor allocations required for KV-cache management and scheduling. vLLM now enables mimalloc as the default allocator on Arm-based CPUs in PyTorch. This caching allocator scales better under multi-threaded pressure, improving Llama 3.1 8B offline throughput by 2.3× and providing up to 7× gains in low-concurrency serving scenarios.

Synchronization and Arm LSE Atomics

At high core counts, performance previously regressed due to contention in OpenMP dynamic scheduling. Profiling revealed that gomp_iter_dynamic_next relied on a load-linked/store-conditional retry loop, which caused repeated failures under high thread contention.

vLLM addressed this by building a libgomp runtime in PyTorch that utilizes Arm Large System Extensions (LSE). By using hardware atomic instructions like LDADDAL, the system eliminates the inefficient retry loop, improving Llama 3.1 8B offline throughput by 9% and reducing Time Per Output Token (TPOT) latency by 15% in low-concurrency scenarios.

Dense-Layer Weight Prepacking

To eliminate the overhead of transforming weights from framework layout to kernel-friendly formats during every call, vLLM enabled a fast oneDNN path accelerated by the Compute Library for Arm Architecture. This allows BF16 weights to be packed during model warmup and reused during inference, reducing TPOT latency by 60% and increasing Llama 3.1 8B offline throughput by 16% in low-concurrency scenarios.

Optimized Paged Attention

Previously, the CPU paged attention kernel relied on reference implementations for QK and PV matrix multiplications and softmax exponentials. vLLM optimized these paths using:

  • BFMMLA Advanced SIMD instructions for QK and PV paths.
  • Vectorized third-degree polynomial approximation for the softmax exponential.

These optimizations made paged attention up to 4× faster and increased Llama 3.1 8B offline throughput by 12%, while also unlocking support for chunked prefill and prefix caching on Arm CPUs.

Quantization Performance

INT8 W8A8 (8-bit Weights and Activations)

Using oneDNN JIT kernels that leverage the SMMLA (signed INT8 matrix multiply-accumulate) instruction on SVE128 and SVE256, vLLM now supports efficient W8A8 quantization. Compared to the optimized BF16 baseline, W8A8 delivers up to 88% higher throughput, 45% lower TPOT, and 54% lower TTFT.

INT8 W4A8 (4-bit Weights, 8-bit Activations)

Accelerated via KleidiAI INT4 micro-kernels, W4A8 further reduces memory bandwidth pressure, which is particularly effective in low-concurrency, memory-bound scenarios. Compared to the W8A8 baseline, W4A8 provides up to 29% higher throughput, 26% lower TPOT, and 18% lower TTFT.

Summary of Performance Gains

When compared to the October 2025 BF16 baseline, the cumulative impact of these optimizations is substantial:

Configuration Max Throughput Gain Max TPOT Speedup Max TTFT Speedup
Optimized BF16 2.7× - -
INT8 W8A8 4.8× 5.7× -
INT8 W4A8 6.2× 7.8× 2.6×

These improvements establish vLLM as a production-ready inference stack for Arm Neoverse-based servers by optimizing the full path from memory allocation and runtime synchronization to kernel-level execution.

Sources