llm-d/llm-d-router

What it solves

It addresses the challenge of efficiently routing inference traffic for Large Language Models (LLMs) by optimizing request placement based on real-time signals like load, priority, and KV-cache locality, which helps reduce latency and improve resource utilization.

How it works

The system consists of an intelligent entry point called the llm-d Router, which combines a proxy (like Envoy) with an Endpoint Picker (EPP). The EPP acts as the "brain," evaluating incoming requests against the state of the inference pool to make routing decisions. It integrates with proxies via the ext-proc protocol. It can be deployed in Standalone Mode (Envoy and EPP in one pod) or Gateway Mode (integrating with the Kubernetes Gateway API for production-scale traffic management).

Who it’s for

It is designed for engineers and operators managing LLM inference infrastructure who need advanced load balancing, request prioritization, and support for complex inference lifecycles (such as Prefill/Decode disaggregation).

Highlights

• KV-Cache Aware Routing: Optimizes request placement based on cache locality.
• Flexible Deployment: Supports both a simple standalone setup and a production-grade Kubernetes Gateway API integration.
• Request Management: Includes APIs for setting scheduling goals (InferenceObjective) and performing model name rewriting (InferenceModelRewrite) for A/B testing.
• Disaggregation Support: Features a sidecar component to coordinate multi-stage inference lifecycles (Encode/Prefill/Decode).