Litelm: A Minimalist Alternative to LiteLLM

Litelm is a minimalist Python library designed to provide the core routing and translation capabilities of LiteLLM without the extensive feature set of the original project. It reduces the codebase to approximately 2,900 lines of code and relies on only two primary dependencies: openai and httpx.

Core Functionality and API

Litelm provides a streamlined path for model routing, message translation, streaming, tool use, and embeddings. It is designed as a drop-in replacement for LiteLLM, mirroring its function names, arguments, and response types. This allows developers to switch from LiteLLM to Litelm by simply changing the import statement.

Supported Features

  • Model Routing: Maps provider/model syntax to the correct API endpoint.
  • Message Translation: Handles format conversions for providers including Anthropic, Bedrock, Cloudflare, and Mistral.
  • Streaming: Supports streaming responses and includes a stream_chunk_builder.
  • Tool Use: Supports function calling and tool definitions.
  • Embeddings: Provides routing and translation for embedding models.
  • Text Completions: Supports standard text completion endpoints.
  • OpenAI Responses API: Maintains compatibility with OpenAI-style response objects.

Removed Features

To achieve its small footprint, Litelm removes several high-level management features found in LiteLLM:

  • Router Class: No built-in load balancing or automatic fallbacks.
  • Proxy Server: No integrated proxy capabilities.
  • Caching and Budgeting: No built-in caching, cost tracking, or spend management.
  • Token Counting: No integrated token counting utilities.
  • Advanced Modalities: No support for image generation, audio, OCR, or fine-tuning.
  • Agents and Guardrails: No integrated agent frameworks, schedulers, or guardrails.

Provider Support and Integration

Litelm supports 19 providers using the provider/model-name syntax. It also supports any OpenAI-compatible endpoint via the api_base parameter.

Provider Environment Variable Handler Type
OpenAI OPENAI_API_KEY OpenAI SDK
Anthropic ANTHROPIC_API_KEY Custom
Groq GROQ_API_KEY OpenAI-compat
Mistral MISTRAL_API_KEY Custom
xAI XAI_API_KEY OpenAI-compat
OpenRouter OPENROUTER_API_KEY OpenAI-compat
Azure AZURE_API_KEY OpenAI SDK (Azure)
Bedrock AWS_ACCESS_KEY_ID Custom
Cloudflare CLOUDFLARE_API_TOKEN Custom
Together TOGETHERAI_API_KEY OpenAI-compat
Fireworks FIREWORKS_API_KEY OpenAI-compat
DeepSeek DEEPSEEK_API_KEY OpenAI-compat
Perplexity PERPLEXITYAI_API_KEY OpenAI-compat
DeepInfra DEEPINFRA_API_TOKEN OpenAI-compat
Gemini GEMINI_API_KEY OpenAI-compat
Cohere COHERE_API_KEY OpenAI-compat
Ollama N/A OpenAI-compat
vLLM N/A OpenAI-compat
LM Studio N/A OpenAI-compat

Technical Implementation and Status

Litelm is currently in Alpha. The project maintainer has verified it as a drop-in replacement for DSPy, with all seven execution paths (Predict, CoT, typed signatures, streaming, embeddings, tool use, and multi-output) proven live.

Development Methodology

The software was developed using a hybrid human-AI approach. Code written before May 14, 2026, was assisted by Claude Code (Claude Opus 4.6/4.7), and subsequent code was written via Pi using GPT-5.5. The maintainer conducted a manual audit of 360 core-path commits from the upstream LiteLLM project to ensure compatibility.

Error Handling

Litelm maps provider-specific errors to a standardized exception hierarchy, including ContextWindowExceededError, RateLimitError, and AuthenticationError.

Community Feedback and Perspectives

Community discussion on Hacker News highlights a divide between those who value minimalism and those who who rely on the "bloat" Litelm removes.

"A lot of the features that have been removed (like cost tracking, streaming, caching) are... kind of the core value proposition of LiteLLM for many of their users."

Other users pointed out technical concerns regarding dependencies, specifically suggesting the migration from httpx to httpx2 due to maintenance concerns.

Some developers argued that with the advent of frontier LLMs, creating a custom router is a trivial task, suggesting that libraries for this purpose are becoming less necessary as AI coding tools make it easier to write bespoke clients.

"This is a 30 minute project with a frontier LLM. I don’t see why anyone would use anyone else’s router."

Finally, some users praised the high degree of interoperability among AI providers, noting that the "OpenAI-compatible" standard has become a rare example of successful industry-wide API interoperability.

Sources

Related

  • Project
    BerriAI/litellm

    An open-source AI gateway and Python SDK that provides a unified OpenAI-compatible interface for calling 100+ different LLM providers.

  • Project
    theopenco/llmgateway

    LLM Gateway is an open‑source API gateway that provides a single OpenAI‑compatible endpoint for routing requests to multiple LLM providers, centralising API‑key management and offering built‑in usage and performance analytics. It can be used as a hosted SaaS service or self‑hosted via Docker.

  • Project
    mozilla-ai/any-llm

    A unified Python SDK that provides a single interface to communicate with multiple LLM providers, allowing developers to switch models without changing their code.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.