llm-d/llm-d-router

llm-d Router: The intelligent entry point for inference requests

What it solves

llm-d Router provides an intelligent entry point for LLM inference traffic. It solves the problem of inefficient request placement by using real-time signals to route requests to the best possible model server, considering factors like current load, request priority, and KV-cache locality to optimize performance and resource utilization.

How it works

The system consists of a proxy (such as Envoy) and an Endpoint Picker (EPP), which acts as the intelligence engine. The EPP integrates with the proxy via the ext-proc protocol to inject routing decisions into the data plane. It can be deployed in Standalone Mode (using a self-managed Envoy proxy) or Gateway Mode (integrating with the Kubernetes Gateway API).

Additionally, a Disaggregation Sidecar is used to coordinate complex multi-stage inference lifecycles (like Prefill/Decode or Encode/Prefill/Decode) by managing the transfer of KV-caches and embeddings between specialized workers.

Who it’s for

It is designed for teams deploying LLMs at scale in production environments who need advanced traffic management, request prioritization, and support for disaggregated inference architectures.

Highlights

  • Prefix-cache aware routing: Optimizes request placement based on KV-cache locality.
  • Flexible Deployment: Supports both standalone Envoy setups and Kubernetes Gateway API integration.
  • Traffic Management: Includes capabilities for model name rewriting for A/B testing and canary rollouts.
  • Disaggregation Support: Orchestrates multi-stage inference lifecycles across specialized workers.

Related

  • Project
    llm-d/llm-d

    A high-performance distributed inference serving stack for Kubernetes that optimizes LLM production deployments across various hardware accelerators.

  • Project
    vllm-project/router

    vLLM Router is a high-performance, lightweight request-forwarding system for vLLM large-scale deployments. It provides advanced load balancing methods and prefill/decode disaggregation to keep LLM inference fast and efficient.

  • Project
    aurelio-labs/semantic-router

    A high-speed decision-making layer for LLMs and agents that uses semantic routing to classify user intent and trigger workflows in milliseconds, avoiding the latency of standard LLM calls.

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Dispatch
    vLLM Semantic Router Fusion primitive enables programmable multi‑model serving

    vLLM introduced the Fusion primitive for its Semantic Router, enabling programmable multi‑model panels, judging, and synthesis as a first‑class serving pattern.