BerriAI/litellm

The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]

What it solves

LiteLLM simplifies the complexity of managing multiple Large Language Model (LLM) providers. Instead of juggling different SDKs, authentication patterns, and request formats for every model, LiteLLM provides a single, unified interface to call over 100 LLM providers using the OpenAI format.

How it works

LiteLLM can be implemented in two primary ways:

  1. Python SDK: A library for direct integration into your application code.
  2. AI Gateway (Proxy Server): A centralized, self-hosted service that acts as a middleman between your application and various LLM providers, allowing you to swap providers without rewriting code.

It also includes an A2A (Agent-to-Agent) protocol for invoking agents and an MCP (Model Context Protocol) bridge to connect MCP servers to any LLM.

Who it’s for

Developers and organizations that use multiple LLM providers (such as OpenAI, Anthropic, Gemini, and Bedrock) and want to avoid provider lock-in, centralize their API management, and maintain a consistent API format.

Highlights

  • Unified API: One interface for 100+ LLMs, eliminating the need for provider-specific SDKs.
  • OpenAI Compatibility: Drop-in compatibility allows for easy swapping of providers.
  • Enterprise Features: Includes virtual keys, spend tracking, load balancing, guardrails, and an admin dashboard.
  • High Performance: Low latency (8ms P95 at 1k RPS).
  • Agent & Tool Integration: Supports A2A agents and MCP tools.