BerriAI/litellm

The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]

What it solves

LiteLLM simplifies the complexity of managing multiple LLM providers. Instead of juggling different SDKs, authentication patterns, and request formats for every model, it provides a single, unified interface to call over 100 LLM providers using the OpenAI format.

How it works

LiteLLM can be implemented in two ways:

  1. Python SDK: A library for direct integration into your code, allowing you to call various models (e.g., GPT-4o, Claude Sonnet) using a consistent syntax.
  2. AI Gateway (Proxy Server): A centralized service that can be self-hosted. It acts as a proxy that translates requests into the format required by the specific provider, offering enterprise features like virtual keys, spend tracking, load balancing, and guardrails.

Additionally, it supports the A2A protocol for invoking agents and provides an MCP (Model Context Protocol) bridge to connect MCP servers to any LLM.

Who it’s for

Developers and organizations that use multiple LLM providers and want to avoid provider lock-in, reduce code rewriting when swapping models, and centralize the management of LLM calls for their teams.

Highlights

  • Unified API: One interface for 100+ LLMs, removing the need for provider-specific SDKs.
  • OpenAI Compatibility: Drop-in compatibility allows providers to be swapped without rewriting code.
  • Enterprise Gateway: Includes virtual keys, spend tracking, and an admin dashboard.
  • High Performance: 8ms P95 latency at 1k RPS.
  • Agent & Tool Support: Supports A2A Agents and MCP tools integration.

Related

  • Project
    mozilla-ai/any-llm

    A unified Python SDK that provides a single interface to communicate with multiple LLM providers, allowing developers to switch models without changing their code.

  • Project
    theopenco/llmgateway

    LLM Gateway is an open‑source API gateway that provides a single OpenAI‑compatible endpoint for routing requests to multiple LLM providers, centralising API‑key management and offering built‑in usage and performance analytics. It can be used as a hosted SaaS service or self‑hosted via Docker.

  • Project
    agentjido/req_llm

    ReqLLM is an Elixir library that unifies calls to dozens of LLM providers (OpenAI, Anthropic, Azure, Google Gemini, etc.) behind a single, high‑level API. It offers functions like `generate_text/3`, `stream_text/3`, and `generate_object/4` that work across 1,200+ models, handles streaming via Finch, validates structured outputs, tracks token usage and estimated cost, and provides built‑in Telemetry. Keys are managed automatically from env files or in‑memory storage, and the library includes extensive provider guides and a Discord community for support.

  • Project
    bentoml/OpenLLM

    OpenLLM is a tool for self-hosting open-source LLMs as OpenAI-compatible APIs, providing a simplified workflow for local serving and enterprise cloud deployment.