tbphp/gpt-load

What it solves

GPT-Load is a high-performance transparent proxy service designed for enterprises and developers who need to manage and integrate multiple AI services. It solves the problem of managing numerous API keys across different providers (like OpenAI, Gemini, and Claude) and ensures service availability through load balancing and automatic failure recovery.

How it works

Built with Go, the service acts as a gateway between the user and AI providers. It preserves native API formats so that existing integrations don't need to be changed. It manages a pool of API keys and uses weighted load balancing to distribute requests across upstream endpoints. If a key fails, the system automatically blacklists it and attempts to rotate to a different key to ensure the request succeeds.

Who it’s for

• Enterprises requiring high-concurrency production environments and horizontal scaling.
• Developers who want a unified way to manage multiple AI API keys without changing their application code.
• AI Teams needing real-time monitoring, health checks, and detailed request logging.

Highlights

• Transparent Proxying: Supports OpenAI, Google Gemini, and Anthropic Claude formats natively.
• Intelligent Key Management: Features automatic rotation, failure recovery, and group-based management.
• Enterprise Architecture: Supports distributed leader-follower deployment for high availability.
• High Performance: Utilizes zero-copy streaming and connection pool reuse for efficiency.
• Management Interface: Includes a Vue 3-based web UI for intuitive configuration and monitoring.