OctaFuse/octafuse-gateway

What it solves

Octafuse Gateway is a self-hosted open-source AI gateway designed for AI Agents. It solves the problem of fragmented AI resources by consolidating multiple AI providers, models, and tools into a single, unified entry point. This allows individuals and teams to manage routing, API keys, budgets, and auditing across various text, image, and speech modalities without needing to manage separate endpoints for every service.

How it works

The gateway acts as a middle layer between the client and various AI providers. It provides a unified Base URL for all requests. It supports multiple protocols (OpenAI, Anthropic, Gemini, and native endpoints for DashScope, MiniMax, and Volcengine) and can perform protocol translation, allowing clients to use a familiar API (like OpenAI's) to call models from different providers.

Key operational mechanisms include:

• Provider and Model Integration: One-click import of pre-configured providers and models.
• Agent Tooling: Unified access to tools like web search, web fetching, and AI detection via /v1/tools/* endpoints.
• Routing Strategies: Four distinct strategies (hash affinity, weighted random, weight priority, and weighted round robin) to balance load, cost, and cache hit rates.
• User and Billing Management: A three-ledger system (catalog price, supply cost, and user price) to track costs and manage quotas across different users and teams.

Who it’s for

• AI Agent Developers: Those who need a single point of access for both models and tools.
• Enterprise AI Teams: Organizations needing centralized governance, cost tracking, and budget control over their AI resource usage.
• Self-hosters: Users who want to deploy their AI gateway on their own infrastructure (Docker or Cloudflare Workers).

Highlights

• Multi-modal Support: Handles text, image generation/editing, ASR, TTS, and real-time voice.
• Agent-Centric: Built-in tools for web search, deep search, and web scraping specifically for agents.
• Agentic Routing: Advanced routing strategies including Prompt Cache affinity to improve performance.
• Three-Ledger Billing: Detailed cost accounting including standard price, actual cost, and user-specific pricing.
• Flexible Deployment: Supports Cloudflare Workers + D1 for free/edge deployment or Docker with Postgres/MySQL.