atopos31/llmio

What it solves

LLMIO is a load-balancing gateway that simplifies the management of multiple LLM providers. It eliminates the need to integrate multiple different APIs (like OpenAI, Anthropic, and Gemini) into a client application by providing a single, unified REST API. It also solves the problem of unreliable provider connectivity and uneven load distribution across different model endpoints.

How it works

The system acts as a proxy between the LLM client and the various AI providers. It uses a Go-based backend with a SQLite database for configuration and logging. It provides two weighted scheduling strategies (random by weight or priority by weight) to route requests based on specific capabilities like tool calling or multimodal support. It includes built-in rate-limit fallbacks and connectivity checks to isolate faults.

Who it’s for

Developers and teams who use multiple LLM providers and want a centralized way to manage routing, observability, and cost tracking across different models and APIs.

Highlights

• Unified API: Supports OpenAI, Anthropic, and Gemini native formats, as well as compatible generic endpoints.
• Weighted Scheduling: Route requests based on weights and capabilities (e.g., structured output, tool calling).
• Observability: Detailed request logs with TraceID, latency breakdowns, token usage, and per-request cost calculation based on configurable pricing.
• Admin UI: A modern React-based console for managing providers, models, and monitoring metrics.
• Session Tracking: Ability to tag and filter logs using a session_id for better debugging and the ability to trace full IO logs.
• Local Persistence: Uses a pure Go SQLite implementation for out-of-the-box readiness.