vLLM Optimizations for Arm CPUs
vLLM has implemented a series of full-stack optimizations for Arm Neoverse-based servers, achieving up to 6.2x throughput gains through improvements in memory allocation, synchronization, and quantization.
vLLM Speculators: Parallel Drafting for Speculative Decoding
vLLM and the Speculators project introduce open-source support for P-EAGLE, DFlash, and DSpark, moving beyond autoregressive drafting to generate candidate token blocks in parallel for faster LLM inference.
vLLM Kimi K3 Support
vLLM has released day-0 support for Kimi K3, a 2.8-trillion-parameter multimodal MoE model, featuring optimizations for Kimi Delta Attention and DSpark speculative decoding to achieve up to 370 tok/s.
vLLM AFD Plugin: Disaggregating Attention and FFN for MoE Serving
The vLLM AFD Plugin introduces Attention-FFN Disaggregation (AFD), allowing Attention and FFN paths in Mixture-of-Experts (MoE) models to scale and execute independently to optimize serving throughput and latency.
Serving GLM-5.2 on NVIDIA B300 GPUs with vLLM
vLLM achieves production SLA compliance for GLM-5.2-NVFP4 on 24 NVIDIA B300 GPUs by implementing P/D disaggregation, speculative padding, and IndexerCache optimizations.
vLLM Kimi K3 Support Preview
vLLM is preparing day-0 open-source serving support for Moonshot AI's Kimi K3, a 2.8-trillion-parameter model featuring a hybrid KDA/full-attention architecture and native vision support.
Beyond a Single Model: Building Mixture-of-Models Systems with vLLM Semantic Router
vLLM Semantic Router now enables building, versioning, and deploying Mixture-of-Models systems that coordinate multiple independent models through a single model interface.
vLLM Production Quality: CI, Benchmarking, and Release Process Overview
vLLM utilizes a three-layer quality assurance framework—comprising continuous integration, performance benchmarking, and a strict release process—to maintain stability across a massive variety of hardware and model architectures.
vLLM TML Inkling Support
vLLM has announced Day-0 support for TML Inkling, a 1T-parameter multimodal model, delivering up to 380 tok/s/user on 4 GB200 GPUs through specialized architectural optimizations.
vLLM and TileRT Integration for Latency-Critical Serving
vLLM has integrated TileRT 0.1.5 as a pluggable decode engine to provide native per-user decode speed for latency-critical workloads while maintaining vLLM's standard prefill and serving infrastructure.
EAGLE-3 Speculative Decoding on AMD Instinct GPUs
vLLM and AMD Quark have implemented an end-to-end pipeline for EAGLE-3 speculative decoding on AMD Instinct GPUs, achieving throughput speedups of up to 2.00x for Kimi-K2.5 and 1.79x for MiniMax-M2.5.
vime ROCm Support for AMD Instinct GPUs
vLLM has announced ROCm support for vime, enabling end-to-end reinforcement learning post-training workflows to run natively on AMD Instinct MI300X and MI355X GPUs.
vLLM × HPC-Ops: High-Performance Attention and MoE Backends from Tencent Hunyuan
vLLM now includes HPC-Ops attention and MoE backends optimized for NVIDIA Hopper H20, delivering up to 2.95× attention speedup and 1.59× MoE speedup, cutting TTFT by about 24% and TPOT by about 17% on Hy3.
vLLM-Omni Optimizations for Qwen3-Omni-30B-A3B-Instruct Serving
vLLM-Omni serves Qwen3-Omni-30B-A3B-Instruct via a three-stage pipeline (Thinker, Talker, Code2Wav) and improves throughput and latency using stage decomposition, CUDA Graphs, async chunk handoffs, async output, stage replicas, and hot‑path cleanup.
vLLM Semantic Router: Enhancing Model Performance via Micro-Agent Collaboration
vLLM introduces the Semantic Router, a serving-layer primitive that transforms a single model API call into a bounded collaboration of micro-agents to outperform frontier models on complex benchmarks.
vLLM-Omni TTS Inference Engineering
vLLM-Omni engineered TTS inference for models like Qwen3-TTS, VoxCPM2, Higgs Audio V3, and Fish Speech S2 Pro by applying model-specific optimizations that decouple latency and throughput bottlenecks, significantly improving audio throughput and reducing end-to-end latency.
vLLM Semantic Router Fusion primitive enables programmable multi‑model serving
vLLM introduced the Fusion primitive for its Semantic Router, enabling programmable multi‑model panels, judging, and synthesis as a first‑class serving pattern.
MiniMax M3 vLLM Support: Day-0 Serving for 1M-Token Multimodal Reasoning
vLLM has released day-0 support for the MiniMax M3 model family, enabling efficient serving of 1M-token context and native multimodal reasoning using MiniMax Sparse Attention (MSA).
DiffusionGemma: The First Diffusion LLM Natively Supported in vLLM
vLLM has integrated DiffusionGemma, a 26B-parameter discrete diffusion language model, enabling high-throughput, low-latency generation by iteratively denoising token blocks instead of sequential autoregressive decoding.
vime RL Framework Release
vLLM introduces vime, an open-source RL post-training framework that integrates Megatron training with vLLM inference to provide a stable and efficient pipeline for LLM reinforcement learning.
vLLM Semantic Router v0.3 Themis release notes
vLLM Semantic Router v0.3 Themis introduces stateful production routing, featuring Session-Aware Agentic Routing (SAAR), a canonical configuration contract, and expanded hardware support for AMD ROCm and Intel OpenVINO.
NVIDIA Nemotron 3 Ultra Support on vLLM
vLLM announces Day-0 support for NVIDIA Nemotron 3 Ultra, a 550B parameter model optimized for long-running autonomous agent workflows through a hybrid Transformer-Mamba MoE architecture.
Fast & Efficient LLM Inference with vLLM Course Launch
vLLM, in collaboration with DeepLearning.AI and Red Hat, has launched a free intermediate course titled Fast & Efficient LLM Inference with vLLM to teach the full AI deployment lifecycle.
vLLM Session-Aware Agentic Routing (SAAR) Release
vLLM introduces Session-Aware Agentic Routing (SAAR), a model selection policy that preserves session continuity and reduces costs for long-horizon LLM agents by preventing unsafe model switches during tool loops and provider-state transitions.
vLLM-Omni Accelerates Inference with AutoRound Quantization
vLLM-Omni now integrates Intel's AutoRound post-training quantization, enabling W4A16 quantization that cuts model size up to 62% while preserving accuracy and unlocking performance gains on Intel XPU and NVIDIA GPUs.
vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation
vLLM provides a high-performance local inference endpoint for NVIDIA DGX Spark, enabling the deployment of large NVFP4 models like Nemotron-3-Super using a unified-memory architecture and OpenAI-compatible API.
Speculators v0.5.0 release notes / what's new
Speculators v0.5.0 introduces DFlash algorithm support for single-pass draft token generation, unified online and offline training via vLLM's native hidden states extraction, and updated documentation.
vLLM Semantic Router Multimodal Routing and Vision Encoder Hardening
vLLM has introduced multimodal routing to the Semantic Router (VSR), enabling the system to use visual evidence as a first-class signal for request-level policy decisions while resolving critical implementation drifts between Rust/Candle and PyTorch paths.
Laguna XS.2 Inference Optimization with vLLM, Speculators, and LLM Compressor
Poolside and Red Hat AI have optimized the Laguna XS.2 33B-A3B MoE model for agentic coding tasks using vLLM integration, DFlash speculative decoding, and LLM Compressor quantization.
vLLM Native RL APIs Release
vLLM has introduced native weight syncing APIs and improved asynchronous RL support to standardize weight transfer between training and inference and eliminate deadlocks in large-scale DPEP deployments.
EAGLE 3.1 Release Notes: Enhancing Speculative Decoding Robustness and Efficiency
EAGLE 3.1 introduces architectural improvements to solve attention drift, doubling acceptance length in long-context workloads and significantly increasing throughput via vLLM and TorchSpec integration.
vLLM x Novita AI: PegaFlow for Production-Grade External KV Cache
vLLM and Novita AI announce PegaFlow, an external KV cache service that decouples KV cache from vLLM workers, enabling faster startups, higher throughput via cache sharing, and RDMA-based cross-node access.
VeRL-Omni: RL Training Framework for Diffusion and Omni-Modality Models
vLLM has announced the pre-release of VeRL-Omni, a general reinforcement learning post-training framework designed specifically for multimodal generative models, including diffusion and omni-modality architectures.
vLLM Elastic Expert Parallelism
vLLM introduces Elastic Expert Parallelism (Elastic EP), enabling Mixture-of-Experts (MoE) deployments to scale the number of workers up or down at runtime without requiring a server restart.
vLLM Performance Benchmarks: Topping Artificial Analysis Leaderboard
vLLM has achieved top rankings on the Artificial Analysis leaderboard for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B by implementing aggressive kernel fusion and speculative decoding optimizations.
vLLM TurboQuant Study: Accuracy and Performance Analysis
A comprehensive study by vLLM reveals that FP8 remains the superior default for KV-cache quantization, while TurboQuant variants trade significant throughput and latency for additional memory capacity.
Serving Agentic Workloads at Scale with vLLM x Mooncake
vLLM integrates Mooncake's distributed KV cache store to boost agentic LLM serving, delivering 3.8× higher throughput, 46× lower TTFT, and 8.6× lower end‑to‑end latency on realistic traces while scaling to 60 GB200 GPUs.
NVIDIA Nemotron 3 Nano Omni Support in vLLM
vLLM now supports NVIDIA Nemotron 3 Nano Omni, a highly efficient 30B MoE multimodal model that unifies vision, audio, and language reasoning in a single loop to power agentic AI.
vLLM DeepSeek V4 Support: Efficient Long-context Attention
vLLM now supports DeepSeek V4-Pro and V4-Flash, implementing a new attention mechanism that enables context lengths up to one million tokens with significant KV cache memory savings.
vLLM FP8 KV-Cache and Attention Quantization Update
vLLM has optimized FP8 KV-cache and attention quantization to reduce decode latency and memory usage while maintaining near-baseline accuracy across Hopper and Blackwell architectures.
vLLM v0.20.0 adds disaggregated serving for hybrid SSM models
vLLM v0.20.0 introduces disaggregated prefill/decode serving for hybrid SSM-FA models, enabling efficient KV transfer via dual descriptor views and a 3-descriptor conv state transfer.
vLLM Korea Meetup 2026 Wrap-Up
The vLLM Korea Meetup 2026 in Seoul highlighted vLLM's evolution into a unified inference infrastructure, showcasing community growth, hardware integration advances, production‑stack features, and real‑world deployment strategies across open‑source and enterprise tracks.
vLLM Prefill-Decode Disaggregation with MORI-IO
vLLM implements Prefill-Decode disaggregation on single-node 8-GPU MI300X setups using AMD's MORI-IO connector, achieving 2.5x higher goodput by eliminating Inter-Token Latency spikes.
Gemma 4 on vLLM: Advanced Reasoning and Multimodal Capabilities
vLLM introduces Day 0 support for Gemma 4, a family of open models from Google featuring advanced reasoning, multimodal inputs, and broad hardware compatibility across TPUs, GPUs, and XPUs.
vLLM Hidden States Extraction System
vLLM v0.18.0 introduces a native hidden states extraction system that enables high-performance retrieval of internal model representations for training speculative decoding draft models.
vLLM Model Runner V2 release notes / what's new
vLLM has introduced Model Runner V2 (MRV2), a ground-up re-implementation of the model runner that improves throughput and reduces latency through a GPU-native, async-first, and modular architecture.
P-EAGLE: Parallel Speculative Decoding in vLLM
vLLM introduces P-EAGLE, a parallel speculative decoding method that generates all draft tokens in a single forward pass, delivering up to 1.69x speedup over vanilla EAGLE-3 on NVIDIA B200 GPUs.
NVIDIA Nemotron 3 Super Support in vLLM
vLLM now supports NVIDIA Nemotron 3 Super, a 120B parameter hybrid MoE model optimized for multi-agent AI with a 1 million token context window and high inference efficiency.
vLLM Semantic Router v0.2 Athena release notes / what's new
vLLM Semantic Router v0.2 Athena introduces a rebuilt model stack, the experimental ClawOS orchestration layer, and advanced model selection primitives to transform semantic routing into a strategic system brain for multi-agent deployments.
vLLM Triton Attention Backend Deep Dive
vLLM has implemented a portable Triton-based attention backend that achieves state-of-the-art performance across NVIDIA, AMD, and Intel GPUs using a single source code implementation.