respanai/respan-gateway

Respan Gateway is a single program (a Rust binary) that sits between your application and all the major AI providers. Instead of writing separate code for each AI vendor, you point your existing AI SDK at Respan Gateway, and it translates your requests to whatever provider you want — OpenAI, Anthropic, Gemini, and many others — and handles the messy details like switching formats, retrying failed calls, and routing to the cheapest or fastest option.

What it does

• Any client, any model — You can use the OpenAI, Anthropic, or Gemini SDKs and still call models from any provider. The gateway translates between the different API formats automatically.
• Multimodal — It supports not just text chat but also images, voice (realtime), speech, transcription, embeddings, and video, across vendors.
• Tools for every model — You can attach MCP servers (a standard way to give AI models extra tools like web search or code execution) and have them work with any model, even if that model wasn't originally designed for them.
• Traffic control — You can set up named routes, fallbacks, retries, and "hedged" requests (where a slow call is raced against a second provider to get the fastest response).
• Observability — It logs every request with cost, latency, and token usage, and can send traces to Respan's analytics platform.
• Performance — The README claims very low added latency (0.07 ms), high throughput (56,000+ requests per second), and low memory usage (31 MB) in their benchmarks.
• Plugins — Everything the gateway does is built as a plugin on a public Rust API. You can write your own plugin (e.g., for guardrails, PII masking, or billing) with just one trait and four hooks.

How to use it

1. Install — pip install respan-gateway (or via Cargo or a GitHub release).
2. Run — Set your API keys as environment variables (e.g., OPENAI_API_KEY) and run respan-gateway. No config file needed to start; it automatically picks up any provider whose key is set.
3. Call — Point your existing SDK at http://localhost:8080 (the default address). For example, with the Anthropic SDK you can set base_url to that address and then call any model, even one from OpenAI.

The web UI is available at http://localhost:8080/ui/ for monitoring.

Configuration

When you need more control, run respan-gateway init to generate a commented gateway.toml file. That file lets you define providers, routes, tools, and tracing. You can validate it with respan-gateway check and reload it without restarting using --watch or SIGHUP. Without client keys, the gateway only listens on localhost for security.

Supported API formats

The gateway accepts requests in several common formats (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, Gemini API, plus realtime voice, images, speech, embeddings, video, and decisions) and translates them to any provider that supports that capability. It also relays any vendor-specific API calls untouched.

Plugins and community

Everything is plugin-based, so you can extend it. The README lists 16 model providers, 12 tool vendors, and guides for integrating with popular frameworks like LangChain, Vercel AI SDK, and Pydantic AI. Contributions are welcome, and adding a new vendor is usually just one file.

For more details, the README points to documentation files for configuration, vendors, MCP tools, and plugin development.