✷ The archive · 11 labs · 73 dispatches
The labs
No more opening a dozen official blogs every morning. First-hand releases from OpenAI, Anthropic, DeepMind and the rest, each with its substance pulled out.
vLLM Scaling Multi-GPU Video Captioning with PyNvVideoCodec
vLLM now supports NVIDIA hardware video decoding via PyNvVideoCodec, removing CPU bottlenecks and more than doubling throughput for multi-GPU video captioning tasks on H100 GPUs.
vLLM Kimi-K3 DSpark Speculative Decoding Implementation
vLLM has implemented a DSpark speculator for the 2.8T-parameter Kimi K3 model, increasing math reasoning interactivity from 110 to 435 tok/s/user and delivering up to 3.5x higher output throughput.
vLLM and Novita AI Release Chord: Faster INT4 MoE Kernels for Kimi K2.x
Novita AI open‑sourced Chord, a W4A16 MoE CUDA operator that speeds up INT4‑quantized Kimi K2.x serving by up to 2.15× over public Humming kernels.
Kimi K3 Performance Optimizations in vLLM
vLLM has implemented a series of stack-wide optimizations for Kimi K3, resulting in up to 2.8x throughput increases and up to 85% lower Time to First Token (TTFT).
Optimizing MiniMax M3 on AMD Instinct MI355X
vLLM has achieved significant throughput gains for MiniMax M3 on AMD Instinct MI355X, increasing MXFP8 output tokens/s/GPU from 109.1 to 342.4 at concurrency 32 through a systematic bottleneck-driven optimization process.
vLLM Tiered KV Cache Offloading
vLLM introduced tiered KV cache offloading, which preserves evicted key‑value data across host memory, storage, and remote peers to avoid recomputation, cut latency, and boost serving capacity.
vLLM GLM 5.3 Optimizations: Hybrid HiSparse Offloading
vLLM introduces Hybrid HiSparse offloading for GLM 5.3, enabling full 1 million context length and higher concurrency on a single 8x H200 node by dynamically offloading KV cache to CPU memory under pressure.
vLLM AgentX release: Optimizing real‑world agentic serving
vLLM announced a suite of KV‑cache, parallelism, and scheduling optimizations that deliver up to 130K tokens per GPU‑second and 14.6×–106× cost advantage on agentic workloads such as DeepSeek V4 Pro, MiniMax M3, and Kimi K3.
vLLM TT Plugin brings Tenstorrent accelerators to LLM serving
The vLLM TT Plugin enables OpenAI‑compatible LLM serving on Tenstorrent mesh hardware using a phase‑based scheduler, on‑device sampling, and in‑process lane data parallelism.
vLLM GLM 5.3 Optimizations: Hybrid HiSparse Offloading
vLLM introduces Hybrid HiSparse offloading for GLM 5.3, enabling full 1 million context length and higher concurrency on memory-constrained hardware like 8x H200 nodes.
MiniMax H3 FastH3 real-time serving with vLLM-Omni
vLLM-Omni integrates FastVideo's FastH3 student model to generate complete MiniMax H3 video‑audio MP4s faster than playback, achieving real‑time latency on an 8‑GPU B300 system.
vLLM speculative decoding on AMD GPUs: performance and methods
vLLM adds speculative decoding to AMD Instinct MI300X/MI355X GPUs, letting a lightweight draft model propose multiple tokens that the target model verifies in a single pass, which can double or more the output-token throughput depending on the draft method, model family, and proposal length.
vLLM Large-Scale Sharded Weight Transfer with Ray Direct Transport
vLLM introduces a sharded weight-transfer engine using Ray Direct Transport (RDT) to enable efficient, fault-tolerant weight synchronization for trillion-parameter models in online RL setups.
IsoExec: Eliminating Trainer-Inference Mismatch in SkyRL
vLLM introduces IsoExec, a unified execution abstraction that eliminates numerical mismatch between training and inference engines in RL workloads through an execution contract and parallelism-invariant kernels.
VeRL-Omni v0.2.0 release notes / what's new
VeRL-Omni v0.2.0 introduces request-level batching for faster diffusion RL and a reusable omni training stack for stable multimodal autoregressive training.
vLLM-Omni Distributed Layerwise Offload
vLLM-Omni introduces Distributed Layerwise Offload (DLO), enabling the efficient serving of large Diffusion Transformer (DiT) models over 200B parameters by optimizing HBM and host memory usage through weight sharding and double-buffered prefetching.
vLLM Adaptive Verification with DSpark
vLLM introduces adaptive verification using DSpark's confidence head to dynamically adjust the number of speculative tokens verified per step, maintaining high throughput across varying concurrency levels.
vLLM Day 0 Support for Qwen3.8-2.4T-A95B
vLLM has announced Day-0 support for Qwen3.8-2.4T-A95B, a 2.4-trillion-parameter sparse MoE model that brings Qwen-Max-class capabilities to open-weight releases.
NVIDIA Nemotron 3.5 Lightning Day-0 Support on vLLM
NVIDIA released Day-0 vLLM support for the 30B Nemotron 3.5 Lightning model, enabling fast, always‑on agent inference with hybrid MoE architecture and three speculative decoding techniques.
vLLM Decode Context Parallelism for Long Context Workloads
vLLM introduces Decode Context Parallelism (DCP) to shard KV caches across GPUs by sequence dimension, significantly increasing concurrency and throughput for long-context agentic workloads.
vLLM Qwen3.5 Performance Optimization
vLLM has achieved over 25,000 total tokens per second (TPS) per GPU for Qwen3.5 on GB200 NVL72 systems through Blackwell-optimized kernels, hybrid cache state transfer, and async scheduling.
vLLM Optimizations for Arm CPUs
vLLM has implemented a series of full-stack optimizations for Arm Neoverse-based servers, achieving up to 6.2x throughput gains through improvements in memory allocation, synchronization, and quantization.
vLLM Speculators: Parallel Drafting for Speculative Decoding
vLLM and the Speculators project introduce open-source support for P-EAGLE, DFlash, and DSpark, moving beyond autoregressive drafting to generate candidate token blocks in parallel for faster LLM inference.
vLLM Kimi K3 Support
vLLM has released day-0 support for Kimi K3, a 2.8-trillion-parameter multimodal MoE model, featuring optimizations for Kimi Delta Attention and DSpark speculative decoding to achieve up to 370 tok/s.
vLLM AFD Plugin: Disaggregating Attention and FFN for MoE Serving
The vLLM AFD Plugin introduces Attention-FFN Disaggregation (AFD), allowing Attention and FFN paths in Mixture-of-Experts (MoE) models to scale and execute independently to optimize serving throughput and latency.
Serving GLM-5.2 on NVIDIA B300 GPUs with vLLM
vLLM achieves production SLA compliance for GLM-5.2-NVFP4 on 24 NVIDIA B300 GPUs by implementing P/D disaggregation, speculative padding, and IndexerCache optimizations.
vLLM Kimi K3 Support Preview
vLLM is preparing day-0 open-source serving support for Moonshot AI's Kimi K3, a 2.8-trillion-parameter model featuring a hybrid KDA/full-attention architecture and native vision support.
Beyond a Single Model: Building Mixture-of-Models Systems with vLLM Semantic Router
vLLM Semantic Router now enables building, versioning, and deploying Mixture-of-Models systems that coordinate multiple independent models through a single model interface.
vLLM Production Quality: CI, Benchmarking, and Release Process Overview
vLLM utilizes a three-layer quality assurance framework—comprising continuous integration, performance benchmarking, and a strict release process—to maintain stability across a massive variety of hardware and model architectures.
vLLM TML Inkling Support
vLLM has announced Day-0 support for TML Inkling, a 1T-parameter multimodal model, delivering up to 380 tok/s/user on 4 GB200 GPUs through specialized architectural optimizations.
vLLM and TileRT Integration for Latency-Critical Serving
vLLM has integrated TileRT 0.1.5 as a pluggable decode engine to provide native per-user decode speed for latency-critical workloads while maintaining vLLM's standard prefill and serving infrastructure.
EAGLE-3 Speculative Decoding on AMD Instinct GPUs
vLLM and AMD Quark have implemented an end-to-end pipeline for EAGLE-3 speculative decoding on AMD Instinct GPUs, achieving throughput speedups of up to 2.00x for Kimi-K2.5 and 1.79x for MiniMax-M2.5.
vime ROCm Support for AMD Instinct GPUs
vLLM has announced ROCm support for vime, enabling end-to-end reinforcement learning post-training workflows to run natively on AMD Instinct MI300X and MI355X GPUs.
vLLM × HPC-Ops: High-Performance Attention and MoE Backends from Tencent Hunyuan
vLLM now includes HPC-Ops attention and MoE backends optimized for NVIDIA Hopper H20, delivering up to 2.95× attention speedup and 1.59× MoE speedup, cutting TTFT by about 24% and TPOT by about 17% on Hy3.
vLLM-Omni Optimizations for Qwen3-Omni-30B-A3B-Instruct Serving
vLLM-Omni serves Qwen3-Omni-30B-A3B-Instruct via a three-stage pipeline (Thinker, Talker, Code2Wav) and improves throughput and latency using stage decomposition, CUDA Graphs, async chunk handoffs, async output, stage replicas, and hot‑path cleanup.
vLLM Semantic Router: Enhancing Model Performance via Micro-Agent Collaboration
vLLM introduces the Semantic Router, a serving-layer primitive that transforms a single model API call into a bounded collaboration of micro-agents to outperform frontier models on complex benchmarks.
vLLM-Omni TTS Inference Engineering
vLLM-Omni engineered TTS inference for models like Qwen3-TTS, VoxCPM2, Higgs Audio V3, and Fish Speech S2 Pro by applying model-specific optimizations that decouple latency and throughput bottlenecks, significantly improving audio throughput and reducing end-to-end latency.
vLLM Semantic Router Fusion primitive enables programmable multi‑model serving
vLLM introduced the Fusion primitive for its Semantic Router, enabling programmable multi‑model panels, judging, and synthesis as a first‑class serving pattern.
MiniMax M3 vLLM Support: Day-0 Serving for 1M-Token Multimodal Reasoning
vLLM has released day-0 support for the MiniMax M3 model family, enabling efficient serving of 1M-token context and native multimodal reasoning using MiniMax Sparse Attention (MSA).
DiffusionGemma: The First Diffusion LLM Natively Supported in vLLM
vLLM has integrated DiffusionGemma, a 26B-parameter discrete diffusion language model, enabling high-throughput, low-latency generation by iteratively denoising token blocks instead of sequential autoregressive decoding.
vime RL Framework Release
vLLM introduces vime, an open-source RL post-training framework that integrates Megatron training with vLLM inference to provide a stable and efficient pipeline for LLM reinforcement learning.
vLLM Semantic Router v0.3 Themis release notes
vLLM Semantic Router v0.3 Themis introduces stateful production routing, featuring Session-Aware Agentic Routing (SAAR), a canonical configuration contract, and expanded hardware support for AMD ROCm and Intel OpenVINO.
NVIDIA Nemotron 3 Ultra Support on vLLM
vLLM announces Day-0 support for NVIDIA Nemotron 3 Ultra, a 550B parameter model optimized for long-running autonomous agent workflows through a hybrid Transformer-Mamba MoE architecture.
Fast & Efficient LLM Inference with vLLM Course Launch
vLLM, in collaboration with DeepLearning.AI and Red Hat, has launched a free intermediate course titled Fast & Efficient LLM Inference with vLLM to teach the full AI deployment lifecycle.
vLLM Session-Aware Agentic Routing (SAAR) Release
vLLM introduces Session-Aware Agentic Routing (SAAR), a model selection policy that preserves session continuity and reduces costs for long-horizon LLM agents by preventing unsafe model switches during tool loops and provider-state transitions.
vLLM-Omni Accelerates Inference with AutoRound Quantization
vLLM-Omni now integrates Intel's AutoRound post-training quantization, enabling W4A16 quantization that cuts model size up to 62% while preserving accuracy and unlocking performance gains on Intel XPU and NVIDIA GPUs.
vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation
vLLM provides a high-performance local inference endpoint for NVIDIA DGX Spark, enabling the deployment of large NVFP4 models like Nemotron-3-Super using a unified-memory architecture and OpenAI-compatible API.
Speculators v0.5.0 release notes / what's new
Speculators v0.5.0 introduces DFlash algorithm support for single-pass draft token generation, unified online and offline training via vLLM's native hidden states extraction, and updated documentation.
vLLM Semantic Router Multimodal Routing and Vision Encoder Hardening
vLLM has introduced multimodal routing to the Semantic Router (VSR), enabling the system to use visual evidence as a first-class signal for request-level policy decisions while resolving critical implementation drifts between Rust/Candle and PyTorch paths.
Laguna XS.2 Inference Optimization with vLLM, Speculators, and LLM Compressor
Poolside and Red Hat AI have optimized the Laguna XS.2 33B-A3B MoE model for agentic coding tasks using vLLM integration, DFlash speculative decoding, and LLM Compressor quantization.