algorithmicsuperintelligence/optillm
Optimizing inference proxy for LLMs
What it solves
OptiLLM addresses the performance gap between smaller, faster LLMs and frontier models on complex reasoning tasks. It allows users to achieve significant accuracy improvements (2-10x) in math, coding, and logical reasoning without the need for expensive model training or fine-tuning.
How it works
OptiLLM acts as an OpenAI API-compatible inference proxy. Instead of sending a request directly to an LLM, you route it through OptiLLM, which applies one or more of over 20 state-of-the-art inference-time optimization techniques. These techniques include multi-agent reasoning, Monte Carlo Tree Search (MCTS), and Mixture of Agents (MoA), which use additional compute at inference time to refine and verify responses.
Who it’s for
It is designed for developers and researchers who want to boost the reasoning capabilities of their existing LLM deployments across various providers (OpenAI, Anthropic, Google, Cerebras) without changing their underlying models.
Highlights
- Zero Training: Improves accuracy instantly without fine-tuning.
- Drop-in Replacement: Works as a transparent proxy for any OpenAI-compatible API endpoint.
- Extensive Library: Implements 20+ techniques including MARS, CePO, and PlanSearch.
- Plugin Ecosystem: Includes plugins for memory, privacy (PII anonymization), web search, and code execution.
- Multi-Provider Support: Compatible with 100+ models via LiteLLM integration.