algorithmicsuperintelligence/optillm

OptiLLM – An OpenAI‑compatible inference proxy that boosts LLM reasoning

What it does – OptiLLM sits between your code and any OpenAI‑compatible LLM (OpenAI, Anthropic, Google, Cerebras, etc.). It intercepts the chat‑completion request, runs one or more optimisation techniques (best‑of‑N sampling, chain‑of‑thought prompting, self‑consistency, Monte‑Carlo‑Tree‑Search, Mixture‑of‑Agents, etc.) and returns the best answer. All of this happens at inference time; no model weights are changed, no fine‑tuning is required.

Why it matters – Reasoning‑heavy tasks (math, coding, logic) often benefit from multiple “thinking” passes. By automating those passes and selecting the most reliable output, OptiLLM can give 2‑10× higher accuracy on benchmark suites while using the same underlying model. The claim is that, with enough compute at inference, a cheaper model (e.g., GPT‑4o‑mini) can match or exceed the performance of a larger one (GPT‑4).

Core features (as described in the README)

Feature	What it means
Drop‑in OpenAI proxy	Run `optillm` locally, point your client’s `base_url` to `http://localhost:8000/v1`, and keep the same API calls you already use.
Multi‑provider support	Through LiteLLM it can forward calls to OpenAI, Anthropic, Google Gemini, Cerebras, Azure OpenAI, etc.
20+ optimisation techniques	Includes best‑of‑N, chain‑of‑thought with reflection, self‑consistency, Monte‑Carlo Tree Search, R* algorithm, Z3 theorem prover, planning (CePO), Mixture‑of‑Agents, and more.
Zero‑training	All techniques are applied at inference; you never need to fine‑tune the base model.
Plugin architecture	Plugins add capabilities such as memory, privacy (PII masking), URL fetching, code execution, structured JSON output, web search, and routing.
SSL & security controls	Options to disable SSL verification (dev only), supply custom CA certs, and guard against SSRF when fetching URLs.
Docker images	Full image, lightweight proxy‑only image, and an offline image with pre‑downloaded models.
Production‑ready hints	Logs, health‑check routing, and a note that the default server binds only to localhost for safety.

How you would use it

Install – pip install optillm (or pull the Docker image).
Start the proxy – set your API key (export OPENAI_API_KEY=…) and run optillm. The server prints a line like Starting server with approach: auto.
Call it from your code – point an OpenAI client at the proxy and optionally prepend a technique slug to the model name, e.g.:
```
client = OpenAI(base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="moa-gpt-4o-mini",   # Mixture‑of‑Agents on top of GPT‑4o‑mini
    messages=[{"role": "user", "content": "Solve: 2x+3=7"}]
)
```
You can also specify the technique via extra_body={"optillm_approach": "bon|mcts"} or embed <optillm_approach> tags in the prompt.

Reported performance

The README lists benchmark gains for several techniques:

MARS on Gemini Flash Lite: +30 points on AIME 2025 (43.3 → 73.3).
CePO on Llama‑3.3‑70B: +18.6 points on Math‑L5.
AutoThink on DeepSeek‑R1‑1.5B: +9.34 points on GPQA‑Diamond.
MOA on GPT‑4o‑mini: matches GPT‑4 on the Arena‑Hard‑Auto suite.
PlanSearch on GPT‑4o‑mini: +20 % pass@5 on LiveCodeBench. These numbers are presented as “full benchmark results” further down in the repo (not reproduced here).

Who might benefit?

Researchers who want to experiment with advanced inference tricks without training new models.
Product teams that already use an OpenAI‑compatible provider and need a quick accuracy boost for reasoning‑heavy endpoints.
Developers looking for a plug‑and‑play proxy that can add memory, privacy, or web‑search capabilities.

Limitations & considerations

Compute cost – Techniques like best‑of‑N, MCTS, or multiple planning passes multiply the number of API calls, so you’ll pay more per request.
Latency – More passes mean higher response times; the proxy is best suited for batch or non‑real‑time workloads.
Security – The proxy can fetch external URLs; the README warns about SSRF and DNS‑rebinding risks and advises network‑level egress controls for high‑security environments.
Reliance on external APIs – OptiLLM does not host models itself (except the optional offline image with spaCy); it still depends on the underlying provider’s uptime and rate limits.

Quick reference summary

Aspect	Detail
Package name	`optillm` (PyPI)
License	Open source (license file linked in badge)
Main entry point	`optillm` CLI starts a Flask server
Supported providers	OpenAI, Anthropic, Google Gemini, Cerebras, Azure OpenAI, any LiteLLM‑compatible model
Key techniques	Best‑of‑N, Chain‑of‑Thought, Self‑Consistency, MCTS, Mixture‑of‑Agents, CePO, R*, Z3, etc.
Plugins	Memory, privacy, URL fetch, code execution, JSON schema, web search, routing, etc.
Installation	`pip install optillm` or Docker (`ghcr.io/algorithmicsuperintelligence/optillm:latest`)
Typical use case	Boost reasoning accuracy on existing LLM APIs without retraining

All information above is taken directly from the repository’s README; no external claims have been added.