theagentrouter/agent-router

What it solves

Agent Router provides a centralized control plane for managing AI and agent traffic. It eliminates the need for application teams to manage multiple different APIs for various models and tools, while allowing platform teams to centrally enforce security, quotas, and routing policies across hosted providers and self-hosted inference servers.

How it works

Powered by Envoy and Envoy Gateway, the project implements a two-tier gateway pattern:

• Tier One Gateway: Acts as the primary entry point to handle authentication, global rate limiting, and top-level routing.
• Tier Two Gateway: Manages ingress traffic to self-hosted model clusters, providing fine-grained control and endpoint picker support to optimize LLM inference.

It offers a single, OpenAI-compatible API that abstracts away the differences between various AI providers (such as OpenAI, Azure OpenAI, Google Gemini, Anthropic, and others) and MCP servers.

Who it’s for

• Application Teams: Who want a consistent API to interact with multiple AI models and tools without changing code for every provider.
• Platform Teams: Who need a centralized way to manage credentials, failover, usage attribution, and quotas for AI traffic.

Highlights

• OpenAI-Compatible API: Provides a unified interface for a wide range of hosted and self-hosted AI providers.
• Envoy-Powered: Leverages the robust traffic management capabilities of Envoy Proxy.
• Two-Tier Architecture: Separates global entry point concerns from specific model cluster optimization.
• Broad Provider Support: Integrates with major providers including OpenAI, Azure, Google, AWS Bedrock, Mistral, Cohere, and DeepSeek.