✷ The archive · 11 labs · 63 dispatches
The labs
No more opening a dozen official blogs every morning. First-hand releases from OpenAI, Anthropic, DeepMind and the rest, each with its substance pulled out.
MiniMax H3 FastH3 real-time serving with vLLM-Omni
vLLM-Omni integrates FastVideo's FastH3 student model to generate complete MiniMax H3 video‑audio MP4s faster than playback, achieving real‑time latency on an 8‑GPU B300 system.
vLLM speculative decoding on AMD GPUs: performance and methods
vLLM adds speculative decoding to AMD Instinct MI300X/MI355X GPUs, letting a lightweight draft model propose multiple tokens that the target model verifies in a single pass, which can double or more the output-token throughput depending on the draft method, model family, and proposal length.
vLLM Large-Scale Sharded Weight Transfer with Ray Direct Transport
vLLM introduces a sharded weight-transfer engine using Ray Direct Transport (RDT) to enable efficient, fault-tolerant weight synchronization for trillion-parameter models in online RL setups.
IsoExec: Eliminating Trainer-Inference Mismatch in SkyRL
vLLM introduces IsoExec, a unified execution abstraction that eliminates numerical mismatch between training and inference engines in RL workloads through an execution contract and parallelism-invariant kernels.
VeRL-Omni v0.2.0 release notes / what's new
VeRL-Omni v0.2.0 introduces request-level batching for faster diffusion RL and a reusable omni training stack for stable multimodal autoregressive training.
vLLM-Omni Distributed Layerwise Offload
vLLM-Omni introduces Distributed Layerwise Offload (DLO), enabling the efficient serving of large Diffusion Transformer (DiT) models over 200B parameters by optimizing HBM and host memory usage through weight sharding and double-buffered prefetching.
vLLM Adaptive Verification with DSpark
vLLM introduces adaptive verification using DSpark's confidence head to dynamically adjust the number of speculative tokens verified per step, maintaining high throughput across varying concurrency levels.
vLLM Day 0 Support for Qwen3.8-2.4T-A95B
vLLM has announced Day-0 support for Qwen3.8-2.4T-A95B, a 2.4-trillion-parameter sparse MoE model that brings Qwen-Max-class capabilities to open-weight releases.
NVIDIA Nemotron 3.5 Lightning Day-0 Support on vLLM
NVIDIA released Day-0 vLLM support for the 30B Nemotron 3.5 Lightning model, enabling fast, always‑on agent inference with hybrid MoE architecture and three speculative decoding techniques.
vLLM Decode Context Parallelism for Long Context Workloads
vLLM introduces Decode Context Parallelism (DCP) to shard KV caches across GPUs by sequence dimension, significantly increasing concurrency and throughput for long-context agentic workloads.
vLLM Qwen3.5 Performance Optimization
vLLM has achieved over 25,000 total tokens per second (TPS) per GPU for Qwen3.5 on GB200 NVL72 systems through Blackwell-optimized kernels, hybrid cache state transfer, and async scheduling.
vLLM Optimizations for Arm CPUs
vLLM has implemented a series of full-stack optimizations for Arm Neoverse-based servers, achieving up to 6.2x throughput gains through improvements in memory allocation, synchronization, and quantization.
vLLM Speculators: Parallel Drafting for Speculative Decoding
vLLM and the Speculators project introduce open-source support for P-EAGLE, DFlash, and DSpark, moving beyond autoregressive drafting to generate candidate token blocks in parallel for faster LLM inference.
vLLM Kimi K3 Support
vLLM has released day-0 support for Kimi K3, a 2.8-trillion-parameter multimodal MoE model, featuring optimizations for Kimi Delta Attention and DSpark speculative decoding to achieve up to 370 tok/s.
vLLM AFD Plugin: Disaggregating Attention and FFN for MoE Serving
The vLLM AFD Plugin introduces Attention-FFN Disaggregation (AFD), allowing Attention and FFN paths in Mixture-of-Experts (MoE) models to scale and execute independently to optimize serving throughput and latency.
Serving GLM-5.2 on NVIDIA B300 GPUs with vLLM
vLLM achieves production SLA compliance for GLM-5.2-NVFP4 on 24 NVIDIA B300 GPUs by implementing P/D disaggregation, speculative padding, and IndexerCache optimizations.
vLLM Kimi K3 Support Preview
vLLM is preparing day-0 open-source serving support for Moonshot AI's Kimi K3, a 2.8-trillion-parameter model featuring a hybrid KDA/full-attention architecture and native vision support.
Beyond a Single Model: Building Mixture-of-Models Systems with vLLM Semantic Router
vLLM Semantic Router now enables building, versioning, and deploying Mixture-of-Models systems that coordinate multiple independent models through a single model interface.
vLLM Production Quality: CI, Benchmarking, and Release Process Overview
vLLM utilizes a three-layer quality assurance framework—comprising continuous integration, performance benchmarking, and a strict release process—to maintain stability across a massive variety of hardware and model architectures.
vLLM TML Inkling Support
vLLM has announced Day-0 support for TML Inkling, a 1T-parameter multimodal model, delivering up to 380 tok/s/user on 4 GB200 GPUs through specialized architectural optimizations.
vLLM and TileRT Integration for Latency-Critical Serving
vLLM has integrated TileRT 0.1.5 as a pluggable decode engine to provide native per-user decode speed for latency-critical workloads while maintaining vLLM's standard prefill and serving infrastructure.
EAGLE-3 Speculative Decoding on AMD Instinct GPUs
vLLM and AMD Quark have implemented an end-to-end pipeline for EAGLE-3 speculative decoding on AMD Instinct GPUs, achieving throughput speedups of up to 2.00x for Kimi-K2.5 and 1.79x for MiniMax-M2.5.
vime ROCm Support for AMD Instinct GPUs
vLLM has announced ROCm support for vime, enabling end-to-end reinforcement learning post-training workflows to run natively on AMD Instinct MI300X and MI355X GPUs.
vLLM × HPC-Ops: High-Performance Attention and MoE Backends from Tencent Hunyuan
vLLM now includes HPC-Ops attention and MoE backends optimized for NVIDIA Hopper H20, delivering up to 2.95× attention speedup and 1.59× MoE speedup, cutting TTFT by about 24% and TPOT by about 17% on Hy3.
vLLM-Omni Optimizations for Qwen3-Omni-30B-A3B-Instruct Serving
vLLM-Omni serves Qwen3-Omni-30B-A3B-Instruct via a three-stage pipeline (Thinker, Talker, Code2Wav) and improves throughput and latency using stage decomposition, CUDA Graphs, async chunk handoffs, async output, stage replicas, and hot‑path cleanup.
vLLM Semantic Router: Enhancing Model Performance via Micro-Agent Collaboration
vLLM introduces the Semantic Router, a serving-layer primitive that transforms a single model API call into a bounded collaboration of micro-agents to outperform frontier models on complex benchmarks.
vLLM-Omni TTS Inference Engineering
vLLM-Omni engineered TTS inference for models like Qwen3-TTS, VoxCPM2, Higgs Audio V3, and Fish Speech S2 Pro by applying model-specific optimizations that decouple latency and throughput bottlenecks, significantly improving audio throughput and reducing end-to-end latency.
vLLM Semantic Router Fusion primitive enables programmable multi‑model serving
vLLM introduced the Fusion primitive for its Semantic Router, enabling programmable multi‑model panels, judging, and synthesis as a first‑class serving pattern.
MiniMax M3 vLLM Support: Day-0 Serving for 1M-Token Multimodal Reasoning
vLLM has released day-0 support for the MiniMax M3 model family, enabling efficient serving of 1M-token context and native multimodal reasoning using MiniMax Sparse Attention (MSA).
DiffusionGemma: The First Diffusion LLM Natively Supported in vLLM
vLLM has integrated DiffusionGemma, a 26B-parameter discrete diffusion language model, enabling high-throughput, low-latency generation by iteratively denoising token blocks instead of sequential autoregressive decoding.
vime RL Framework Release
vLLM introduces vime, an open-source RL post-training framework that integrates Megatron training with vLLM inference to provide a stable and efficient pipeline for LLM reinforcement learning.
vLLM Semantic Router v0.3 Themis release notes
vLLM Semantic Router v0.3 Themis introduces stateful production routing, featuring Session-Aware Agentic Routing (SAAR), a canonical configuration contract, and expanded hardware support for AMD ROCm and Intel OpenVINO.
NVIDIA Nemotron 3 Ultra Support on vLLM
vLLM announces Day-0 support for NVIDIA Nemotron 3 Ultra, a 550B parameter model optimized for long-running autonomous agent workflows through a hybrid Transformer-Mamba MoE architecture.
Fast & Efficient LLM Inference with vLLM Course Launch
vLLM, in collaboration with DeepLearning.AI and Red Hat, has launched a free intermediate course titled Fast & Efficient LLM Inference with vLLM to teach the full AI deployment lifecycle.
vLLM Session-Aware Agentic Routing (SAAR) Release
vLLM introduces Session-Aware Agentic Routing (SAAR), a model selection policy that preserves session continuity and reduces costs for long-horizon LLM agents by preventing unsafe model switches during tool loops and provider-state transitions.
vLLM-Omni Accelerates Inference with AutoRound Quantization
vLLM-Omni now integrates Intel's AutoRound post-training quantization, enabling W4A16 quantization that cuts model size up to 62% while preserving accuracy and unlocking performance gains on Intel XPU and NVIDIA GPUs.
vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation
vLLM provides a high-performance local inference endpoint for NVIDIA DGX Spark, enabling the deployment of large NVFP4 models like Nemotron-3-Super using a unified-memory architecture and OpenAI-compatible API.
Speculators v0.5.0 release notes / what's new
Speculators v0.5.0 introduces DFlash algorithm support for single-pass draft token generation, unified online and offline training via vLLM's native hidden states extraction, and updated documentation.
vLLM Semantic Router Multimodal Routing and Vision Encoder Hardening
vLLM has introduced multimodal routing to the Semantic Router (VSR), enabling the system to use visual evidence as a first-class signal for request-level policy decisions while resolving critical implementation drifts between Rust/Candle and PyTorch paths.
Laguna XS.2 Inference Optimization with vLLM, Speculators, and LLM Compressor
Poolside and Red Hat AI have optimized the Laguna XS.2 33B-A3B MoE model for agentic coding tasks using vLLM integration, DFlash speculative decoding, and LLM Compressor quantization.
vLLM Native RL APIs Release
vLLM has introduced native weight syncing APIs and improved asynchronous RL support to standardize weight transfer between training and inference and eliminate deadlocks in large-scale DPEP deployments.
EAGLE 3.1 Release Notes: Enhancing Speculative Decoding Robustness and Efficiency
EAGLE 3.1 introduces architectural improvements to solve attention drift, doubling acceptance length in long-context workloads and significantly increasing throughput via vLLM and TorchSpec integration.
vLLM x Novita AI: PegaFlow for Production-Grade External KV Cache
vLLM and Novita AI announce PegaFlow, an external KV cache service that decouples KV cache from vLLM workers, enabling faster startups, higher throughput via cache sharing, and RDMA-based cross-node access.
VeRL-Omni: RL Training Framework for Diffusion and Omni-Modality Models
vLLM has announced the pre-release of VeRL-Omni, a general reinforcement learning post-training framework designed specifically for multimodal generative models, including diffusion and omni-modality architectures.
vLLM Elastic Expert Parallelism
vLLM introduces Elastic Expert Parallelism (Elastic EP), enabling Mixture-of-Experts (MoE) deployments to scale the number of workers up or down at runtime without requiring a server restart.
vLLM Performance Benchmarks: Topping Artificial Analysis Leaderboard
vLLM has achieved top rankings on the Artificial Analysis leaderboard for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B by implementing aggressive kernel fusion and speculative decoding optimizations.
vLLM TurboQuant Study: Accuracy and Performance Analysis
A comprehensive study by vLLM reveals that FP8 remains the superior default for KV-cache quantization, while TurboQuant variants trade significant throughput and latency for additional memory capacity.
Serving Agentic Workloads at Scale with vLLM x Mooncake
vLLM integrates Mooncake's distributed KV cache store to boost agentic LLM serving, delivering 3.8× higher throughput, 46× lower TTFT, and 8.6× lower end‑to‑end latency on realistic traces while scaling to 60 GB200 GPUs.
NVIDIA Nemotron 3 Nano Omni Support in vLLM
vLLM now supports NVIDIA Nemotron 3 Nano Omni, a highly efficient 30B MoE multimodal model that unifies vision, audio, and language reasoning in a single loop to power agentic AI.
vLLM DeepSeek V4 Support: Efficient Long-context Attention
vLLM now supports DeepSeek V4-Pro and V4-Flash, implementing a new attention mechanism that enables context lengths up to one million tokens with significant KV cache memory savings.