kubernetes-sigs/gateway-api-inference-extension

Gateway API Inference Extension

What it solves

This project addresses the challenges of self-hosting generative AI models on Kubernetes by optimizing how requests are routed to model servers. It prevents issues like high tail latency and throughput drops caused by inefficient load balancing, especially when dealing with KV-cache evictions or queuing as load increases.

How it works

It transforms a standard Kubernetes Gateway API-compatible proxy (like Envoy) into an "Inference Gateway" by using Envoy's External Processing (ext-proc) filter. This allows the gateway to intercept inference requests and use an Endpoint Picker (EPP) and an Inference Scheduler to route requests to the most optimal model server replica based on real-time metrics and capabilities (such as Prefix Cache status or LoRA adapter availability).

Who it’s for

It is designed for inference platform teams who self-host generative models (primarily LLMs) on Kubernetes and need to manage access, optimize performance, and maintain operational guardrails across multiple AI workloads.

Highlights

  • KV-Cache Aware Routing: Improves throughput and reduces latency by using scheduling algorithms that consider request cost and cache status.
  • LoRA Adapter Management: Provides Kubernetes-native APIs to route requests to specific LoRA adapters and manage their rollout, A/B testing, and blue-green upgrades.
  • Pluggable Architecture: Works with any gateway supporting ext-proc and Gateway API, and integrates with model servers like vLLM via the llm-d Router.
  • Operational Guardrails: Allows multiple GenAI workloads to share a pool of foundation model servers securely and efficiently.

Related

  • Project
    llm-d/llm-d-router

    An intelligent inference router for LLMs that optimizes request placement using load and prefix-cache awareness to improve serving efficiency.

  • Project
    volcano-sh/kthena

    A Kubernetes-native LLM inference platform that provides declarative model lifecycle management and intelligent routing to make AI infrastructure scalable and cost-efficient.

  • Project
    kubeai-project/kubeai

    A Kubernetes operator and proxy that simplifies deploying and scaling AI models with prefix-aware load balancing to optimize LLM performance.

  • Project
    vllm-project/aibrix

    A cloud-native, Kubernetes-native control plane for deploying and managing scalable and cost-effective LLM inference infrastructure.

  • Project
    agentic-in/inferoa

    An inference-native agent harness designed for loop engineering, optimizing token usage and cache reuse to maintain efficient, long-horizon recursive AI workflows.