SomeOddCodeGuy/WilmerAI

WilmerAI is one of the oldest LLM semantic routers. It uses multi-layer prompt routing and complex workflows to allow you to not only create practical chatbots, but to extend any kind of application that connects to an LLM via REST API. Wilmer sits between your app and your many LLM APIs, so that you can manipulate prompts as needed.

What it solves

WilmerAI addresses the limitations of simple prompt routers that rely on keywords or only the most recent message. It solves the problem of accurately understanding user intent in complex conversations by using full conversation history and a structured, node-based orchestration system to route requests to the most appropriate LLM or sequence of tasks.

How it works

At its core, WilmerAI uses a node-based workflow engine where processes are defined in JSON files.

  • Contextual Routing: It analyzes the entire conversation history to select a specialized workflow (e.g., Coding or Creative) or to make conditional "if/then" decisions within a workflow.
  • Orchestration: Each node in a workflow can trigger a different LLM endpoint, call external tools, run custom Python scripts, or execute other nested workflows.
  • Memory System: It maintains state through a four-part system: chronological summaries, rolling summaries, a searchable vector database, and a state document.
  • API Gateway: It acts as a middleware that exposes OpenAI- and Ollama-compatible endpoints, allowing it to integrate with existing front-ends like Open WebUI or SillyTavern without requiring changes to the client.

Who it’s for

It is designed for users who want granular control over how LLMs are orchestrated, those who wish to distribute workloads across multiple local or proprietary LLM instances, and developers who want to build complex, semi-autonomous agents without relying on fully autonomous (and often unreliable) agent frameworks.

Highlights

  • Multi-LLM Coordination: Ability to use multiple different models (e.g., a small local model for summary and a large cloud model for reasoning) in a single request.
  • Distributed Inference: Support for routing tasks across multiple physical computers running different LLM APIs.
  • Structured Output: Grammar-enforced JSON Schema output for routing and extraction, ensuring reliability even with small local models.
  • Stateful Memory: Hybrid memory combining summaries and vector search for deep contextual awareness.
  • MCP Integration: Experimental support for the Model Context Protocol (MCP) for agentic tool discovery.
  • Privacy-Centric: Designed to minimize outbound calls, with optional encryption for stored conversation data.