litellm: a unified AI gateway and SDK to call 100+ LLMs using a single OpenAI-compatible API

What it solves

LiteLLM simplifies the complexity of managing multiple Large Language Model (LLM) providers. Instead of juggling different SDKs, authentication patterns, and request formats for every provider (such as OpenAI, Anthropic, and Gemini), it provides a single, unified interface to call over 100 LLMs using the OpenAI format.

How it works

LiteLLM can be implemented in two ways:

1. Python SDK: A library for direct integration into your code, allowing you to call different models using a consistent function call.
2. AI Gateway (Proxy Server): A centralized service that can be self-hosted. It acts as a proxy that translates requests into the correct provider-specific format, providing enterprise features like virtual keys, spend tracking, load balancing, and guardrails.

Additionally, it supports the A2A protocol for invoking agents and provides an MCP (Model Context Protocol) bridge to connect MCP servers to any LLM.

Who it’s for

Developers and organizations that use multiple LLM providers and want to avoid provider lock-in, reduce code rewriting when swapping models, and implement centralized management for LLM usage across a team.

Highlights

• Unified API: One interface for 100+ LLMs, eliminating the need for provider-specific SDKs.
• OpenAI Compatibility: Drop-in compatibility allows swapping providers without rewriting code.
• Enterprise Gateway: Includes virtual keys, spend tracking, and an admin dashboard.
• High Performance: 8ms P95 latency at 1k RPS.
• Agent and Tool Integration: Supports A2A agents and MCP tools.