labring/aiproxy

What it solves

AI Proxy is a production-ready AI gateway that acts as middleware between client applications and various AI providers (like OpenAI, Anthropic, and Gemini). It solves the problem of managing multiple AI APIs in a production environment by providing a single, unified interface for routing, monitoring, and access control, eliminating the the need to handle provider-specific logic in every application.

How it works

The gateway uses an OpenAI-compatible protocol to receive requests and routes them to the appropriate AI provider based on configurable rules. It includes a plugin system to extend functionality and a management panel for configuration. Key mechanisms include:

• Protocol Conversion: Translates requests between OpenAI, Claude, and Gemini formats.
• Intelligent Routing: Uses priority-based channel selection and load balancing to distribute traffic.
• Multi-tenant Architecture: Isolates organizations with specific resource quotas (RPM/TPM) and custom pricing.
• MCP Support: Integrates Model Context Protocol servers to extend AI capabilities through public, private, or automatically generated tools from OpenAPI specs.

Who it’s for

• Developers building AI applications who need a unified API for multiple providers.
• DevOps/Platform Engineers managing AI infrastructure for teams or organizations requiring monitoring, billing, and quota management.
• Enterprise Teams needing secure, multi-tenant access to LLMs with audit logs and subnet restrictions.

Highlights

• Smart Request Management: Features automatic retry logic and load balancing across providers.
• Comprehensive Monitoring: Real-time alerts for balance and error rates, along with detailed request/response audit trails.
• Plugin Ecosystem: Includes built-in plugins for Redis/memory caching, real-time web search (Google, Bing, Arxiv), and handling reasoning model <think> tags.
• Protocol Flexibility: Supports multi-format processing (text, image, audio, documents) and transparent protocol conversion for responses-only models.
• Built-in Tokenizer: Operates without external tiktoken dependencies.