vllm-project/router

A high-performance and light-weight router for vLLM large scale deployment

What it solves

When running vLLM at a large scale, many worker instances serve thousands of requests per second. Without a smart router, traffic can land unevenly, some workers get overloaded while others sit idle, and repeated prompts waste compute by recomputing the same prefixes. The vLLM Router solves this by acting as a lightweight forwarding layer that distributes requests intelligently, keeps multi-turn sessions on the same worker, and can separate the prefill phase (reading and analyzing the prompt) from the decode phase (generating tokens) to optimize hardware use.

How it works

The router sits in front of your vLLM workers and receives OpenAI‑compatible API calls. It uses one of several load‑balancing policies to pick the best worker for each request: round robin, random, consistent hashing (routes a session/user to the same worker), power of two (picks the least loaded of two random workers), or cache‑aware (routes requests to workers with the highest prefix cache reuse). For prefill/decode disaggregation, it maintains separate pools of prefill and decode workers and routes each phase accordingly. It also includes retry logic with exponential backoff, circuit breakers that automatically open and close to protect workers, Prometheus metrics, and optional Kubernetes service discovery. When enabled, it can forward requests to vLLM’s Rust gRPC workers (using grpc:// URLs) and can load a WASM plugin to run custom on‑request middleware (for example, input validation or transformation).

Who it’s for

This router is for teams deploying vLLM at scale—especially those running many replicas in Kubernetes or using specialised hardware for prefill vs. decode. It suits workloads with heavy multi‑turn chat (to keep sessions sticky), repeated prompts like few‑shot or agent loops (to exploit caching), or high‑throughput services that need fine‑grained control over retries, circuit breaking, and metrics.

Highlights

  • Multiple load‑balancing policies: round robin, random, consistent hash, power of two, and cache‑aware.
  • Prefill/decode disaggregation with support for NIXL, NCCL, and Mooncake connectors.
  • Kubernetes service discovery automatically finds worker pods via label selectors.
  • Enterprise‑grade resilience: retries with exponential backoff and jitter, circuit breakers with a configurable state machine.
  • gRPC backend support to route requests to vLLM’s Rust Inference workers.
  • Optional WASM middleware for custom request handling on protected routes.
  • Prometheus metrics endpoint for monitoring.
  • Lightweight and high‑performance, written in Rust with a Python launcher available.

Related

  • Dispatch
    vLLM and TileRT Integration for Latency-Critical Serving

    vLLM has integrated TileRT 0.1.5 as a pluggable decode engine to provide native per-user decode speed for latency-critical workloads while maintaining vLLM's standard prefill and serving infrastructure.

  • Dispatch
    vLLM Prefill-Decode Disaggregation with MORI-IO

    vLLM implements Prefill-Decode disaggregation on single-node 8-GPU MI300X setups using AMD's MORI-IO connector, achieving 2.5x higher goodput by eliminating Inter-Token Latency spikes.

  • Dispatch
    vLLM Disaggregated Serving Guide

    vLLM introduces disaggregated serving to separate prefill and decode phases and offload CPU-intensive frontend tasks, reducing tail latency and increasing goodput for high-concurrency LLM workloads.

  • Project
    llm-d/llm-d-router

    An intelligent inference router for LLMs that optimizes request placement based on load, priority, and KV-cache locality to improve serving performance.

  • Dispatch
    vLLM Semantic Router: Enhancing Model Performance via Micro-Agent Collaboration

    vLLM introduces the Semantic Router, a serving-layer primitive that transforms a single model API call into a bounded collaboration of micro-agents to outperform frontier models on complex benchmarks.