vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
What it solves
vLLM is designed to make Large Language Model (LLM) inference and serving fast, easy, and cost-effective. It addresses the bottleneck of memory management for attention keys and values, which often limits the throughput of LLM serving systems.
How it works
The library uses a technique called PagedAttention, which efficiently manages the attention key-value (KV) cache memory. It further optimizes performance through continuous batching of requests, chunked prefill, and prefix caching. To maximize hardware utilization, it employs optimized attention kernels (like FlashAttention) and supports a wide range of of quantization methods (FP8, INT8, INT4, etc.) and distributed inference techniques (tensor, pipeline, and expert parallelism).
Who it’s for
Developers and researchers who need to deploy LLMs at scale with high throughput and low latency, supporting a variety of hardware (NVIDIA, AMD, Intel GPUs, and CPUs) and over 200 Hugging Face model architectures.
Highlights
- High Throughput: State-of-the-art serving throughput using PagedAttention and continuous batching.
- Broad Model Support: Supports 200+ architectures, including decoder-only, Mixture-of-Experts (MoE), multi-modal, and embedding models.
- Hardware Flexibility: Compatible with NVIDIA, AMD, and Intel GPUs, as well as x86/ARM/PowerPC CPUs and various NPUs/TPUs.
- API Compatibility: Provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support.
- Advanced Decoding: Includes speculative decoding and support for structured outputs using xgrammar or guidance.