vllm-project/semantic-router

An open, programmable decision layer for models and compute.

What it solves

It addresses the fragmentation of AI models and compute resources. Instead of relying on a single model, it provides a programmable decision layer that allows developers to select or combine the most appropriate models and backends based on the specific task, cost, latency, and location requirements.

How it works

It acts as an API gateway between an agent harness and various model backends. While the agent harness manages the loop and state, the Router uses a defined policy to route calls to configured backends across local, private, or cloud compute. It can select a single model or combine multiple models (Mixture-of-Models) to optimize performance.

Who it’s for

Developers building AI agents and LLM applications who need to optimize for quality, cost, and latency across a heterogeneous environment of models and hardware.

Highlights

Programmable Decision Layer: Set policies to automatically route requests to the best model for the task.
Multi-Backend Support: Seamlessly switch between local, private, and cloud compute.
Mixture-of-Models: Ability to combine multiple models for a single call.
Integrated Hallucination Detection: Includes capabilities like LettuceDetect for generative hallucination detection as an endpoint.

Project
ulab-uiuc/LLMRouter
An open-source library for intelligent LLM routing that dynamically selects the most cost-effective and performant model for each query using over 16 routing strategies.
Dispatch
vLLM Semantic Router: Enhancing Model Performance via Micro-Agent Collaboration
vLLM introduces the Semantic Router, a serving-layer primitive that transforms a single model API call into a bounded collaboration of micro-agents to outperform frontier models on complex benchmarks.
Project
NVIDIA-AI-Blueprints/llm-router
An experimental multimodal router that analyzes user prompts to recommend the optimal LLM or VLM based on trade-offs between accuracy, speed, and cost.
Dispatch
Beyond a Single Model: Building Mixture-of-Models Systems with vLLM Semantic Router
vLLM Semantic Router now enables building, versioning, and deploying Mixture-of-Models systems that coordinate multiple independent models through a single model interface.
Dispatch
vLLM Semantic Router Fusion primitive enables programmable multi‑model serving
vLLM introduced the Fusion primitive for its Semantic Router, enabling programmable multi‑model panels, judging, and synthesis as a first‑class serving pattern.

vllm-project/semantic-router

What it solves

How it works

Who it’s for

Highlights

Related