atopos31/llmio

Unified LLM gateway with weighted load balancing, observability & cost tracking. 统一的 LLM 网关，提供权重负载均衡、可观测性与费用追踪。

What it solves

LLMIO acts as a centralized gateway for managing multiple LLM providers (OpenAI, Anthropic, Gemini, etc.). It solves the complexity of integrating different APIs by providing a unified REST interface, allowing developers to route requests across models, manage quotas, and monitor usage through a single service.

How it works

Built in Go, the project functions as a load-balancing proxy. It translates between different LLM protocols (e.g., converting Anthropic messages to OpenAI format), handles weighted request scheduling, and manages custom headers or session keys to ensure compatibility with various clients. It includes a React-based admin UI for monitoring, log inspection, and configuring advanced features like time-of-day pricing and database maintenance.

Who it’s for

Developers and teams who need to manage multiple LLM integrations, implement cost-tracking, or require a robust, self-hosted proxy to handle load balancing and observability across different AI model providers.

Highlights

  • Unified API: Standardizes disparate provider protocols into a single, consistent interface.
  • Advanced Routing: Supports weighted scheduling and automatic protocol translation between providers.
  • Observability: Provides detailed usage analytics, latency tracking, and request logging with session comparison tools.
  • Efficient Storage: Uses content-defined chunking and global deduplication to drastically reduce the storage footprint of request logs in SQLite.
  • Operational Control: Includes built-in circuit breakers, rate limiting, and sophisticated database maintenance tools for space reclamation.

Related

  • Project
    theopenco/llmgateway

    LLM Gateway is an open‑source API layer that lets you call many LLM providers through a single OpenAI‑compatible endpoint, with built‑in key management, cost tracking, and performance monitoring. It can be used as a hosted SaaS or self‑hosted via Docker.

  • Project
    yym68686/uni-api

    uni‑api is an open‑source, Docker‑based gateway that lets you expose a single OpenAI‑compatible API while routing requests to many different LLM back‑ends (OpenAI, Anthropic, Gemini, Vertex, Azure, AWS, etc.). It offers YAML‑only configuration, weighted/round‑robin load balancing, automatic retries, per‑model timeouts, rate limiting, moderation, and tool/image support.

  • Project
    ENTERPILOT/GoModel

    GoModel is an open‑source Go‑written AI gateway that provides a single OpenAI‑compatible endpoint for dozens of LLM providers. It adds caching, cost‑tracking, budgets, rate‑limits, fail‑over, observability (Prometheus/OpenTelemetry), a web dashboard, and extensible plugins, helping teams control spend and reliability while keeping existing SDKs unchanged.

  • Project
    tbphp/gpt-load

    GPT‑Load is an open‑source Go service that acts as a single, self‑hosted gateway for many LLM APIs. It lets you configure multiple providers, manage API keys, schedule traffic, handle retries, and view unified logs and cost dashboards—all behind a familiar OpenAI‑compatible HTTP interface.

  • Project
    router-for-me/CLIProxyAPI

    CLI Proxy API is an open‑source proxy server that presents a unified OpenAI‑compatible API for many LLM providers (OpenAI, Gemini, Claude, Grok, Kimi, Meta, etc.). It handles OAuth logins, multi‑account load‑balancing, streaming, function calls, and multimodal inputs, and offers optional dashboards for usage and cost tracking. Ideal for developers with multiple AI subscriptions who want a single local endpoint for their CLI tools.