theagentrouter/agent-router

Agent Router – Quick Overview

What it is – Agent Router (formerly Envoy AI Gateway) is an open‑source control‑plane that sits in front of LLM and tool APIs. It presents a single, OpenAI‑compatible HTTP endpoint (/v1) and lets you route requests to many back‑ends – hosted services (OpenAI, Azure, Anthropic, etc.) or self‑hosted inference clusters – while handling auth, quotas, fail‑over and usage attribution.

Why it matters – In large organisations the same application may need to call several LLM providers, each with its own keys, rate limits and pricing. Agent Router centralises that plumbing so developers can keep using the familiar OpenAI client libraries and let the router decide which model to hit, enforce policies, and collect metrics.

Core components

• Envoy – the high‑performance proxy that actually forwards traffic.
• Envoy Gateway – adds Kubernetes‑native CRDs and control‑plane logic.
• CLI (aigw) – lets you run the router locally (aigw run) or install it on a cluster via Helm.
• CRDs – AIGatewayRoute, AIServiceBackend, BackendSecurityPolicy (all under aigateway.envoyproxy.io).

How you use it

1. Install the CLI or Helm chart.
2. Define back‑ends for each provider (API keys, endpoints, quotas) using the CRDs.
3. Create routing rules that map incoming model names or paths to a specific back‑end.
4. Point your application at http://localhost:1975/v1 (local) or the cluster service URL. The router rewrites the request, adds the right auth header, and forwards it.

Supported providers – A long list of commercial LLM services is built‑in, including OpenAI, Azure OpenAI, Anthropic, Google Gemini, AWS Bedrock, Mistral, Cohere, Groq, Together AI, DeepInfra, DeepSeek, Hunyuan, SambaNova, Grok, Tetrate’s own service, BitDeer, Eden AI, Lanseq, Tencent LLM Knowledge Engine, TypeSafe AI, etc.

Deployment options

• Standalone – run OPENAI_API_KEY=… aigw run on a laptop for quick testing.
• Kubernetes – deploy the two‑tier gateway pattern: a Tier‑One gateway for auth/rate‑limiting and a Tier‑Two gateway that talks to a self‑hosted model serving cluster.

Community & governance – Part of the Agentic AI Foundation under the Linux Foundation Projects umbrella. Contributions follow an LF Code of Conduct; there’s a Discord, weekly community meetings, and public docs.

---

Bottom line – Agent Router is a real, production‑grade open‑source project that gives you a unified, OpenAI‑style front‑door to many LLM providers, backed by Envoy’s performance and Kubernetes‑native management.