✷ The archive · 11 labs · 73 dispatches
The labs
No more opening a dozen official blogs every morning. First-hand releases from OpenAI, Anthropic, DeepMind and the rest, each with its substance pulled out.
vLLM Native RL APIs Release
vLLM has introduced native weight syncing APIs and improved asynchronous RL support to standardize weight transfer between training and inference and eliminate deadlocks in large-scale DPEP deployments.
EAGLE 3.1 Release Notes: Enhancing Speculative Decoding Robustness and Efficiency
EAGLE 3.1 introduces architectural improvements to solve attention drift, doubling acceptance length in long-context workloads and significantly increasing throughput via vLLM and TorchSpec integration.
vLLM x Novita AI: PegaFlow for Production-Grade External KV Cache
vLLM and Novita AI announce PegaFlow, an external KV cache service that decouples KV cache from vLLM workers, enabling faster startups, higher throughput via cache sharing, and RDMA-based cross-node access.
VeRL-Omni: RL Training Framework for Diffusion and Omni-Modality Models
vLLM has announced the pre-release of VeRL-Omni, a general reinforcement learning post-training framework designed specifically for multimodal generative models, including diffusion and omni-modality architectures.
vLLM Elastic Expert Parallelism
vLLM introduces Elastic Expert Parallelism (Elastic EP), enabling Mixture-of-Experts (MoE) deployments to scale the number of workers up or down at runtime without requiring a server restart.
vLLM Performance Benchmarks: Topping Artificial Analysis Leaderboard
vLLM has achieved top rankings on the Artificial Analysis leaderboard for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B by implementing aggressive kernel fusion and speculative decoding optimizations.
vLLM TurboQuant Study: Accuracy and Performance Analysis
A comprehensive study by vLLM reveals that FP8 remains the superior default for KV-cache quantization, while TurboQuant variants trade significant throughput and latency for additional memory capacity.
Serving Agentic Workloads at Scale with vLLM x Mooncake
vLLM integrates Mooncake's distributed KV cache store to boost agentic LLM serving, delivering 3.8× higher throughput, 46× lower TTFT, and 8.6× lower end‑to‑end latency on realistic traces while scaling to 60 GB200 GPUs.
NVIDIA Nemotron 3 Nano Omni Support in vLLM
vLLM now supports NVIDIA Nemotron 3 Nano Omni, a highly efficient 30B MoE multimodal model that unifies vision, audio, and language reasoning in a single loop to power agentic AI.
vLLM DeepSeek V4 Support: Efficient Long-context Attention
vLLM now supports DeepSeek V4-Pro and V4-Flash, implementing a new attention mechanism that enables context lengths up to one million tokens with significant KV cache memory savings.
vLLM FP8 KV-Cache and Attention Quantization Update
vLLM has optimized FP8 KV-cache and attention quantization to reduce decode latency and memory usage while maintaining near-baseline accuracy across Hopper and Blackwell architectures.
vLLM v0.20.0 adds disaggregated serving for hybrid SSM models
vLLM v0.20.0 introduces disaggregated prefill/decode serving for hybrid SSM-FA models, enabling efficient KV transfer via dual descriptor views and a 3-descriptor conv state transfer.
vLLM Korea Meetup 2026 Wrap-Up
The vLLM Korea Meetup 2026 in Seoul highlighted vLLM's evolution into a unified inference infrastructure, showcasing community growth, hardware integration advances, production‑stack features, and real‑world deployment strategies across open‑source and enterprise tracks.
vLLM Prefill-Decode Disaggregation with MORI-IO
vLLM implements Prefill-Decode disaggregation on single-node 8-GPU MI300X setups using AMD's MORI-IO connector, achieving 2.5x higher goodput by eliminating Inter-Token Latency spikes.
Gemma 4 on vLLM: Advanced Reasoning and Multimodal Capabilities
vLLM introduces Day 0 support for Gemma 4, a family of open models from Google featuring advanced reasoning, multimodal inputs, and broad hardware compatibility across TPUs, GPUs, and XPUs.
vLLM Hidden States Extraction System
vLLM v0.18.0 introduces a native hidden states extraction system that enables high-performance retrieval of internal model representations for training speculative decoding draft models.
vLLM Model Runner V2 release notes / what's new
vLLM has introduced Model Runner V2 (MRV2), a ground-up re-implementation of the model runner that improves throughput and reduces latency through a GPU-native, async-first, and modular architecture.
P-EAGLE: Parallel Speculative Decoding in vLLM
vLLM introduces P-EAGLE, a parallel speculative decoding method that generates all draft tokens in a single forward pass, delivering up to 1.69x speedup over vanilla EAGLE-3 on NVIDIA B200 GPUs.
NVIDIA Nemotron 3 Super Support in vLLM
vLLM now supports NVIDIA Nemotron 3 Super, a 120B parameter hybrid MoE model optimized for multi-agent AI with a 1 million token context window and high inference efficiency.
vLLM Semantic Router v0.2 Athena release notes / what's new
vLLM Semantic Router v0.2 Athena introduces a rebuilt model stack, the experimental ClawOS orchestration layer, and advanced model selection primitives to transform semantic routing into a strategic system brain for multi-agent deployments.
vLLM Triton Attention Backend Deep Dive
vLLM has implemented a portable Triton-based attention backend that achieves state-of-the-art performance across NVIDIA, AMD, and Intel GPUs using a single source code implementation.
vLLM AMD ROCm Attention Backends Optimization
vLLM introduces optimized attention backends for AMD ROCm, delivering up to 4.4x higher throughput for MHA and 1.5x for MLA models on Instinct MI300X, MI325X, and MI355X GPUs.
vLLM Multi-LoRA Serving for MoE Models
vLLM version 0.15.0 introduces optimized Multi-LoRA serving for Mixture of Experts (MoE) models, enabling multiple fine-tuned adapters to share a single GPU to reduce idle compute capacity.