envoyproxy/ai-gateway
Manages Unified Access to Generative AI Services built on Envoy Gateway
What it solves
It provides a centralized way to manage request traffic from application clients to various Generative AI services, simplifying the integration and management of LLM providers.
How it works
It uses a two-tier gateway pattern:
- Tier One Gateway: Acts as a centralized entry point handling authentication, top-level routing, and global rate limiting.
- Tier Two Gateway: Manages ingress traffic to self-hosted model serving clusters, providing fine-grained control over access and using endpoint picker support to optimize LLM inference.
Who it’s for
Developers and platform engineers who need to manage traffic to multiple AI providers (such as OpenAI, Google Gemini, Anthropic, and others) or self-hosted model clusters.
Highlights
- Supports a wide range of AI providers including OpenAI, Azure OpenAI, Google Gemini, Vertex AI, AWS Bedrock, Mistral, Cohere, Groq, Together AI, DeepInfra, DeepSeek, Hunyuan, SambaNova, Grok, and Tetrate Agent Router Service.
- Implements a two-tier architecture for centralized control and optimized inference routing.
- Built on top of Envoy Gateway.