thushan/olla

What it solves

Olla is a high-performance proxy and load balancer designed to manage self-hosted LLM infrastructure. It solves the problem of reliably routing requests across multiple inference nodes, providing a unified interface for different backends while ensuring high availability through failover and health monitoring.

How it works

Olla acts as an intermediary between the client and various LLM backends (such as vLLM, Ollama, and llama.cpp). It uses two proxy engines—Sherpa for simplicity and Olla for high performance—to route requests based on priority and health. It features KV-cache-aware affinity routing (sticky sessions) to keep multi-turn conversations on the same backend and can translate between Anthropic and OpenAI API formats to ensure compatibility across different providers.

Who it’s for

It is designed for developers and operators managing self-hosted LLM deployments, ranging from home labs to enterprise data centers, who need a reliable way to load balance and unify multiple inference endpoints.

Highlights

• Smart Load Balancing: Priority-based routing with automatic failover and connection retries.
• KV-Cache Affinity: Sticky sessions that pin conversations to specific backends to optimize performance.
• Broad Backend Support: Native integrations with vLLM, Ollama, llama.cpp, LM Studio, and others.
• API Translation: Automatic translation between Anthropic Messages API and OpenAI-compatible formats.
• Lightweight Footprint: Written in Go, running on less than 50MB of RAM.
• Integrated Dashboard: A built-in, read-only fleet overview for monitoring health and latency.