Experiential Open Source Model Gateway Enables Unified Access, Routing, and Optimization of LLMs

What Experiential Does and Why It Matters

Experiential provides a single OpenAI‑compatible API that can forward requests to any combination of hosted, bring‑your‑own‑key (BYOK), or locally‑run language models, while giving operators fine‑grained control over user access, spend limits, and model selection. By collecting production traffic as OpenTelemetry traces, the platform can automatically train a custom router or fine‑tune a model to improve quality, latency, and cost.

Core Features Delivered Out‑of‑the‑Box

Unified Control Plane

  • All models—OpenAI, Anthropic, Gemini, Azure, Bedrock, Fireworks, OpenRouter, or self‑hosted open‑source models—are reachable through the same /v1/chat/completions endpoint.
  • The gateway enforces per‑user, per‑agent, and per‑use‑case policies, including budget caps (e.g., a default $50 command budget).

Traffic‑Driven Optimization

  • OpenTelemetry traces are ingested from agents and stored anonymously; no prompts, credentials, or raw content are ever sent to telemetry services.
  • The exp build wizard creates a simulated project router based on these traces, and exp optimize can fine‑tune an open‑source model (via the Tinker tool) to better serve the observed workload.

Low‑Overhead Routing

  • Benchmarks show the gateway adds under 1 ms of latency for BYOK requests, making it suitable for high‑throughput applications.
  • Caching strategies are not detailed in the repository, but community members have asked for clarification on token‑level caching and its cost implications.

Getting Started Quickly

Local Development

pip install experiential
exp   # runs the setup wizard and prints a one‑time API key
export EXP_GATEWAY_KEY=...   # use the printed key
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $EXP_GATEWAY_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"opus-5","messages":[{"role":"user","content":"Help me"}]}'

The wizard automatically creates a public alias (e.g., opus-5) and a default spend budget.

Hosted Platform

  • The managed service lives at https://platform.experientiallabs.ai and exposes the same OpenAI‑compatible API at https://api.experientiallabs.ai/v1.
  • Onboarding steps include uploading LLM traces, connecting BYOK keys for any provider, and repointing existing coding agents (Claude Code, Cursor, Codex, Aider, etc.) to the new endpoint.

Optimizing From Real Traffic

  1. Collect Traces – Export an OTLP JSONL file from your agents or use the public terminal‑tasks dataset:
    curl -L -o traces.otel.jsonl \
  https://huggingface.co/datasets/experiential-labs/wmo-terminal-tasks-traces/resolve/540883e451dc13d34fb50fdd36b143cb0f1fb0db/traces.otel.jsonl
  2. Build a Project Router – Run the interactive builder:
    exp build support-agent   # selects providers, models, budget, and trace file
  3. Fine‑Tune a Model – After gathering router traces, invoke Tinker to create a custom model:
    exp optimize model support-agent

The resulting model can be served through the same gateway, closing the loop between observation and improvement.

Telemetry Policy

  • Experiential ships with anonymous aggregate telemetry via PostHog.
  • Telemetry never includes user prompts, model names, credentials, or raw content.
  • Users can toggle telemetry with:
    exp config telemetry status   # view current state
exp config telemetry disable  # turn off
exp config telemetry enable   # turn on

Settings are stored locally in .exp/settings.toml.

Community Feedback Highlights

"Finally an open source tool doing this!" – ashermania

"The gateway adds under 1 ms for BYOK requests. Amazing!" – ceroxylon

"Is it similar to LiteLLM? If so, what sets it apart?" – cheema33

"Caching is definitely one of the hardest parts to get right…" – foremerge

"Curious if you initially had this in Python and then rewrote in Rust?" – 23david

These comments underscore three recurring themes: (1) the novelty of an open‑source, model‑agnostic gateway, (2) the importance of ultra‑low latency, and (3) open questions around caching strategies and implementation language.

Comparison to Similar Projects

  • LiteLLM – Provides a thin wrapper for multiple providers but lacks a built‑in traffic‑driven router and optimization loop.
  • Enterpilot/GoModel, Maximhq/Bifrost, BerriAI/LiteLLM – Offer multi‑provider routing; Experiential distinguishes itself with its telemetry‑guided router builder and integrated fine‑tuning via Tinker.
  • vLLM Semantic Router – Focuses on semantic routing for open‑source inference; Experiential complements this by handling BYOK, budget enforcement, and a hosted SaaS offering.

Development and Contribution

uv sync --extra dev               # install dev dependencies
uv run ruff format --check .      # code formatting check
uv run ruff check .               # linting
uv run ty check                   # type checking
uv run pytest -q                  # test suite

Contribution guidelines, agent conventions, and API docs are maintained in the repository’s AGENTS.md and SETUP.md files.

Outlook

Experiential demonstrates that an open‑source gateway can serve as both a control plane for heterogeneous LLM fleets and a data collection point for continuous model improvement. As more organizations adopt multi‑provider strategies, tools that automate routing, enforce spend limits, and close the feedback loop between production traffic and model training will become essential infrastructure.

Sources

Related

  • Project
    experientiallabs/experiential

    An open-source gateway and router for agent workflows that provides a unified API for multiple LLM providers and optimizes model routing based on production traffic.

  • Project
    BerriAI/litellm

    LiteLLM is an open‑source Python SDK and self‑hosted proxy that provides a unified, OpenAI‑compatible API for calling over 100 LLM providers. It offers production features like virtual keys, spend tracking, guardrails, auto‑routing, and an admin dashboard, letting teams swap models or providers without code changes.

  • Dispatch
    World Model Optimizer: Distill and Serve Frontier Models at Half the Cost

    World Model Optimizer (wmo) is an open‑source tool that lets developers turn agent traces into continuously improving models, achieving frontier‑quality performance with routing and distillation that can cut inference costs by 40%+.

  • Project
    agentjido/req_llm

    ReqLLM is an Elixir library that unifies calls to dozens of LLM providers (OpenAI, Anthropic, Azure, Google Gemini, etc.) behind a single, high‑level API. It offers functions like `generate_text/3`, `stream_text/3`, and `generate_object/4` that work across 1,200+ models, handles streaming via Finch, validates structured outputs, tracks token usage and estimated cost, and provides built‑in Telemetry. Keys are managed automatically from env files or in‑memory storage, and the library includes extensive provider guides and a Discord community for support.

  • Project
    AISBench/benchmark

    A model evaluation tool based on OpenCompass that provides accuracy and performance benchmarking for both local and service-based AI models across text and multimodal tasks.