vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

vLLM – Fast, flexible LLM serving

What it is – vLLM is an open‑source Python library that lets you run (serve) large language models for inference at high speed. It focuses on squeezing out maximum throughput on GPUs (and other accelerators) while keeping the API simple.

Key capabilities

  • PagedAttention – a custom attention memory manager that swaps KV‑cache pages in and out, letting you serve many concurrent requests without blowing GPU memory.
  • Continuous batching & chunked pre‑fill – new prompts are added to the same batch on‑the‑fly, reducing idle time.
  • Quantization support – FP8, INT8/4, GPTQ, AWQ, GGUF, and more, so you can run models with lower‑precision weights for speed/size gains.
  • Optimised kernels – integrates FlashAttention, FlashInfer, TRT‑LLM, FlashMLA and Triton kernels, plus custom GEMM/MoE kernels via CUTLASS, TRT‑LLM‑GEN, CuTeDSL.
  • Speculative decoding – n‑gram, suffix, EAGLE, DFlash methods that generate tokens ahead of time to cut latency.
  • Distributed inference – tensor, pipeline, data, expert and context parallelism for multi‑GPU/CPU setups.
  • Broad model coverage – works out‑of‑the‑box with 200+ Hugging‑Face models: decoder‑only LLMs (Llama, Qwen), MoE models (Mixtral), hybrid state‑space models (Mamba), multimodal (LLaVA), embedding/retrieval models, etc.
  • API layers – provides an OpenAI‑compatible REST endpoint, Anthropic‑style messages API, and gRPC, plus streaming token output and tool‑calling support.
  • Hardware flexibility – runs on NVIDIA, AMD, Intel GPUs, CPUs, and via plugins on TPUs, Gaudi, Huawei Ascend, Apple Silicon, etc.

Typical workflow

  1. Install with uv pip install vllm (or pip) and optionally vllm download‑kernels for pre‑compiled kernels.
  2. Load a model from Hugging Face using vllm.Engine or the CLI vllm serve <model>.
  3. Start the server (python -m vllm.entrypoints.openai.api_server) and send requests via the OpenAI‑compatible HTTP API, gRPC, or the Python client.
  4. Adjust settings (batch size, quantisation, speculative decoding, LoRA adapters) to hit your desired latency‑throughput trade‑off.

Why it matters – Serving LLMs at production scale is often bottlenecked by memory‑intensive KV‑cache and per‑token kernel overhead. vLLM’s paging, continuous batching and specialised kernels dramatically raise the number of tokens per second per GPU, lowering cost and enabling real‑time applications.

Resources – Documentation site, quick‑start guide, model list, and a research paper on the PagedAttention technique are linked in the README.

Related

  • Project
    Wenyueh/MinivLLM

    A custom vLLM-style inference engine implementation that benchmarks Flash Attention and Paged Attention to optimize LLM prefilling and decoding.

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Dispatch
    Optimizing LLMs in Production: Precision, Attention, and Architecture

    Hugging Face outlines key techniques for efficient LLM deployment, focusing on lower precision quantization, Flash Attention for memory efficiency, and architectural optimizations like RoPE, ALiBi, MQA, and GQA.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.