BerriAI/litellm

The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]

What it solves

Managing LLM calls across different providers is complex because each has its own SDK, authentication method, and request format. This project provides a unified interface that allows developers to switch between 100+ LLM providers (such as OpenAI, Anthropic, Gemini, and Bedrock) without rewriting their code.

How it works

It functions as both a Python SDK and a self-hosted AI Gateway (proxy server). The Python SDK allows for direct library integration, while the proxy server acts as a centralized service. It translates various provider-specific APIs into a standard OpenAI-compatible format. The gateway includes production-ready features like virtual keys, spend tracking, load balancing, and guardrails.

Who it’s for

It is designed for developers and organizations that need to integrate multiple LLMs into their applications, manage API costs, or maintain a consistent interface across different AI services.

Highlights

  • Unified API for 100+ LLM providers, eliminating the need to juggle multiple SDKs.
  • Drop-in compatibility with OpenAI's API format.
  • Production-ready features including spend tracking, virtual keys, and load balancing.
  • High performance with 8ms P95 latency at 1k requests per second.
  • Support for advanced features like MCP (Model Context Protocol) tools, agent harnesses, and A2A (Agent-to-Agent) protocols.

Related

  • Project
    iimeta/fastapi

    An enterprise-grade LLM API integration system that provides a unified, OpenAI-compatible interface to access multiple AI model providers.

  • Project
    mozilla-ai/any-llm

    A unified Python SDK that provides a single interface to communicate with multiple LLM providers, allowing developers to switch models without changing their code.

  • Project
    theopenco/llmgateway

    LLM Gateway is an open‑source API layer that lets you call many LLM providers through a single OpenAI‑compatible endpoint, with built‑in key management, cost tracking, and performance monitoring. It can be used as a hosted SaaS or self‑hosted via Docker.

  • Project
    mnfst/llm-gateway

    Manifest LLM Gateway is an open-source tool that unifies access to hundreds of LLMs from various providers into a single, reliable, OpenAI-compatible endpoint with built-in routing, cost tracking, and automatic failover.

  • Project
    agentgateway/agentgateway

    An open-source proxy that provides security, observability, and governance for agent-to-LLM, agent-to-tool, and agent-to-agent communication.