tbphp/gpt-load

GPT‑Load – a self‑hosted gateway for LLM APIs

What it does

• Provides a single HTTP endpoint that mimics the native OpenAI, Anthropic, Gemini, etc. APIs.
• Behind that endpoint you can configure many channels (different providers, accounts, or subscription services) and let the gateway route requests based on weights, health, or cost.
• Handles API‑key management, OAuth flows, retries, cooldowns, black‑listing, and per‑client access‑keys.
• Stores logs, usage statistics and cost estimates in an embedded UI backed by SQLite, MySQL or PostgreSQL, with credentials encrypted on‑disk.

Why you might need it

• When an app needs to talk to several LLM providers (OpenAI, Claude, Gemini, Bedrock, etc.) without changing its code for each provider.
• To balance traffic across multiple API keys, avoid rate‑limit hits, and automatically fall back to a healthy key if one fails.
• To keep a unified audit trail and cost dashboard for all LLM calls in one place.

Key components

Component	Role
Gateway server (written in Go)	Exposes OpenAI‑compatible routes, forwards requests to the selected upstream channel, applies scheduling and retry logic.
Management UI	Web console (port 3001) for adding channels, groups, and AccessKeys; shows health, logs, and usage charts.
Channel	Configuration for a specific upstream service (e.g., OpenAI, Anthropic, a custom OpenAI‑compatible relay) and one or more API credentials.
Group	A collection of channels together with model whitelist and routing policy (weights, cooldowns, session affinity).
AccessKey	Token given to client applications; it determines which groups and protocols the client may use.

Supported protocols & models

• OpenAI chat, completions, embeddings, images, rerank, OCR/audio, etc.
• Anthropic messages, Gemini models endpoints, Mistral OCR/audio, and a generic OpenAI‑compatible relay.
• Over 30 built‑in providers: OpenAI, Azure OpenAI, AWS Bedrock, Google Vertex AI, Claude, Gemini, DeepSeek, Moonshot, Hugging Face, Cohere, Groq, and many more, plus subscription‑based services like Codex and Grok.

Deployment options

• Docker Compose (default) – runs the service in a container, stores data in a Docker volume (gpt-load-data).
• Native binary – download a pre‑built executable for Linux/macOS/Windows; Windows also offers an installer that creates a service.
• Database – SQLite out‑of‑the‑box; can be switched to MySQL or PostgreSQL via DATABASE_DSN.

Configuration highlights

• .env file controls host, port, timeouts, logging, proxy settings, and encryption keys.
• AUTH_KEY protects the management UI; ENCRYPTION_KEY encrypts stored API secrets (must be backed up with the DB).
• CLIENT_IP_HEADER and TRUSTED_PROXIES let you run behind a reverse proxy and enforce IP‑based restrictions.

Typical workflow

1. Start the service (docker compose up -d).
2. Read the generated management key (cat /app/data/auth.key).
3. Log in to http://127.0.0.1:3001 and add a channel (e.g., OpenAI + your API key).
4. Create a group, select the channel(s) and the models you want to expose.
5. Generate an AccessKey for your application and use that key in the Authorization: Bearer … header when calling the gateway.

When to be careful

• Losing encryption.key makes all stored credentials unrecoverable.
• Version 2.0 is a complete rewrite; it cannot import data from the older 1.x line.
• The service binds to localhost by default – expose it only behind your own firewall or reverse‑proxy.

License & community

• MIT‑licensed core code.
• Relies on Apache‑2.0 Bifrost Core for provider adapters and MIT CLIProxyAPI for OAuth handling.
• Official Docker image: ghcr.io/tbphp/gpt-load:2.

---

All details are taken directly from the project's README.