theopenco/llmgateway

Route, manage, and analyze your LLM requests across multiple providers with a unified API interface.

LLM Gateway – What It Is

LLM Gateway is an open‑source API gateway that sits between your applications and any number of Large Language Model (LLM) providers (OpenAI, Anthropic, Google Vertex AI, etc.). It presents a single, OpenAI‑compatible endpoint, lets you store the different provider keys in one place, and records usage data so you can see how many tokens you’ve spent and how much each call costs.

Core Features (as described in the README)

Unified API – Calls follow the OpenAI /v1/chat/completions shape, so you can switch providers without changing your client code.
Multi‑provider routing – Configure which backend model to hit for each request; the gateway forwards the request to the appropriate provider.
Key management – Keep all API keys for the various providers centrally, protected by a secret you set at startup.
Usage analytics – The service logs request count, token usage, response latency and cost, enabling you to monitor and optimise spend.
Performance monitoring – Compare latency and price across models/providers from the same dashboard.
Self‑hosted or SaaS – Use the hosted service at llmgateway.io or run the whole stack yourself.

How to Run It

Hosted option

Sign up at llmgateway.io.
Obtain an API key and start sending requests to https://api.llmgateway.io/v1/....

Self‑hosted (Docker‑first)

Generate two secrets:

export LLM_GATEWAY_SECRET=$(openssl rand -base64 32 | tr -d '\n')
export GATEWAY_API_KEY_HASH_SECRET=$(openssl rand -base64 32 | tr -d '\n')

Run the provided script which creates PostgreSQL and Redis volumes and starts the unified container, or launch a container manually with named volumes (see README for the full docker run command).
The container exposes several ports (3002‑3007, 4001‑4002) for the UI, API, and internal services.

Once up, you can call the gateway just like the OpenAI API, e.g.:

curl -X POST https://api.llmgateway.io/v1/chat/completions \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
     -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Hello, how are you?"}]}'

Development Workflow

Dependencies – Managed with pnpm. Run pnpm i && pnpm run setup to install packages, start Docker services, sync the DB schema and seed data.
Dev servers – pnpm dev launches all apps (frontend dashboards, API, gateway, etc.) in watch mode.
Production build – pnpm build creates optimized bundles for deployment.
Monorepo layout – The repo is a monorepo with several Next.js apps (ui, playground, code, airside, docs) and shared packages (db, models, shared). The actual routing logic lives in apps/gateway (a Hono backend).

Licensing

Open‑source core – AGPL‑v3 (see LICENSE).
Enterprise add‑ons – The ee/ directory contains commercial features (advanced billing, unlimited data retention, multi‑org admin, etc.) under a separate enterprise license. Contact contact@llmgateway.io for details.

Who Might Use It?

Teams building products that need to switch between or combine multiple LLM providers without rewriting client code.
Organizations that want a single place to audit token usage and cost.
Developers who prefer to self‑host for data‑privacy or compliance reasons.

Bottom Line

LLM Gateway is a practical, production‑ready middleware that abstracts away the differences between LLM APIs, adds analytics, and offers both SaaS and self‑hosted deployment paths. It’s built as a TypeScript monorepo with Docker support and a clear separation between UI, API, and gateway components.