mozilla-ai/otari

Otari – Self‑hosted OpenAI‑compatible LLM gateway

What it is – Otari is a lightweight, open‑source server that sits between your applications and any LLM provider (OpenAI, Anthropic, Azure, etc.). It presents a single OpenAI‑compatible API endpoint while handling authentication, credential storage, budget enforcement, routing policies, usage tracking, and optional guard‑rail services.

Key capabilities

• Multi‑provider routing – One /v1/chat/completions endpoint can forward requests to 40+ back‑ends. You configure providers in a YAML file; Otari resolves the correct API key at request time.
• Access control – Revocable API keys scoped by user, workspace, and model. Keys are stored encrypted and can be managed from the web dashboard.
• Budget & spend limits – Define per‑user or per‑workspace budgets; Otari checks them before dispatch and records actual spend after the call.
• Routing policies – Simple fail‑over, weighted load‑balancing, or learned selection policies to choose the best provider for a request.
• Built‑in tools – Optional sandboxed code execution, web‑search, multi‑call‑plan (MCP), inference guardrails, and file‑understanding services that can be attached to a request.
• Deployments – Run as a single‑process Docker container with SQLite (ephemeral) or as a full stack with PostgreSQL, multiple optional services, and a multi‑tenant control plane (otari.ai).
• Developer friendliness – Swagger UI, OpenAPI spec, and a Python SDK (any‑llm) for easy integration. Existing OpenAI client libraries work by just pointing base_url at Otari.

Typical use‑cases

• Companies that want to keep provider API keys behind a firewall while exposing a unified endpoint to internal services.
• Teams needing per‑project budgets and audit logs for LLM usage.
• Developers building agents (e.g., Claude Code) who want automatic guardrails and tool integration without modifying each model’s client code.
• Anyone who wants to experiment with different LLM providers without rewriting application code.

Quick start

# Run a throw‑away gateway (SQLite) with Docker
docker run --rm -p 8000:8000 \
  -e OTARI_MASTER_KEY=SET_A_MASTER_KEY \
  -e OPENAI_API_KEY=$YOUR_OPENAI_KEY \
  -e OTARI_CONFIG_YAML='default_pricing: true
providers:
  openai:
    api_key: ${OPENAI_API_KEY}' \
  mzdotai/otari:latest otari serve

The container prints a bootstrap API key on first start. Use it like any OpenAI key:

curl http://localhost:8000/api/v1/chat/completions \
  -H "Authorization: Bearer tk-..." \
  -H "Content-Type: application/json" \
  -d '{"model":"openai:gpt-4o-mini","messages":[{"role":"user","content":"Say hello."}]}'

For persistent deployments replace the Docker run with the provided docker‑compose setup (PostgreSQL, optional code‑exec, web‑search, guardrails) and configure OTARI_MODE/OTARI_AI_TOKEN for hybrid or hosted modes.

Documentation & community – Full docs (quickstart, deployment, configuration, routing, access control, tools, agent guardrails) live under docs/. A public dashboard demo is at https://otari.ai, and the project has a Discord community for support.

License – Apache 2.0.

---

TL;DR – Otari lets you run your own OpenAI‑compatible gateway that aggregates many LLM providers, enforces budgets, stores credentials securely, and adds optional safety tools—all via a single API endpoint you can self‑host or connect to Mozilla’s hosted control plane.