vllm-project/router

A high-performance and light-weight router for vLLM large scale deployment

What it solves

This project provides a high-performance, lightweight request forwarding system designed for large-scale vLLM deployments. It solves the problem of efficiently distributing incoming requests across multiple vLLM workers to maximize throughput and optimize resource utilization, particularly when using advanced deployment patterns like prefill-decode disaggregation.

How it works

The router acts as a gateway that receives requests and forwards them to backend vLLM workers based on configurable load balancing policies. It supports several routing strategies, including consistent hashing for session affinity (KV cache reuse) and cache-aware routing to optimize for prefix cache hits.

For advanced setups, it supports "Prefill-Decode Disaggregation," where the routing logic is specialized to handle the separate processing phases of LLM inference. It also integrates with Kubernetes for automatic worker discovery and health monitoring, and includes enterprise-grade reliability features like circuit breakers and retry logic with exponential backoff.

Who it’s for

MLOps engineers and developers deploying vLLM at scale who need a robust way to manage request distribution, ensure high availability, and optimize inference performance through intelligent routing.

Highlights

  • Diverse Load Balancing: Supports round robin, random, power of two, consistent hashing, and cache-aware routing.
  • Prefill-Decode Disaggregation: Specialized routing for separated processing phases to improve efficiency.
  • Kubernetes Native: Built-in service discovery and worker management for K8s environments.
  • Enterprise Reliability: Includes circuit breakers, retry logic, and Prometheus metrics for monitoring.
  • High Performance: Implemented in Rust for minimal overhead.

Related

  • Dispatch
    vLLM and TileRT Integration for Latency-Critical Serving

    vLLM has integrated TileRT 0.1.5 as a pluggable decode engine to provide native per-user decode speed for latency-critical workloads while maintaining vLLM's standard prefill and serving infrastructure.

  • Dispatch
    vLLM Prefill-Decode Disaggregation with MORI-IO

    vLLM implements Prefill-Decode disaggregation on single-node 8-GPU MI300X setups using AMD's MORI-IO connector, achieving 2.5x higher goodput by eliminating Inter-Token Latency spikes.

  • Dispatch
    Optimizing LLM Performance: Solving Long Prompt Blocking and Decode Slowdowns

    Hugging Face explores how long prompts block request queues and slow down token generation, proposing request-parallel prefills and disaggregated prefill as solutions to reduce latency.

  • Project
    vllm-project/production-stack

    A reference implementation for production vLLM deployment on Kubernetes, providing a scalable inference stack with request routing and observability.

  • Project
    llm-d/llm-d

    llm‑d is an open‑source, CNCF‑sandbox project that adds production‑grade routing, KV‑cache tiering, and expert‑parallelism to existing LLM model servers (vLLM, SGLang). Delivered as Helm charts for Kubernetes, it lets organizations serve large language models at scale with lower latency and higher throughput across many accelerator types.