vllm-project/semantic-router

What it solves

vLLM Semantic Router is designed to solve the problem of fragmented LLM infrastructure. Instead of hard-coding routing logic into applications, it provides a programmable layer that allows developers to manage a "Mixture-of-Models" system. This helps optimize for quality, cost, latency, and privacy by ensuring the right model path is chosen for each request based on specific signals and policies.

How it works

The system acts as a routing layer that evaluates request signals, user preferences, and application policies to select or compose a model path. It can route requests across heterogeneous compute environments, including GPUs, accelerators, edge devices, and cloud infrastructure, while maintaining data boundaries for privacy.

Who it’s for

It is built for developers and organizations building complex LLM applications that use multiple models across different hardware and locations (edge, private, and cloud) and need a programmable way to manage how requests are routed to those models.

Highlights

• Programmable Routing: Avoids hard-coded logic by making model selection executable via policies.
• Heterogeneous Compute Support: Routes across a mix of GPUs, accelerators, edge, and cloud.
• Mixture-of-Models: Ability to compose personalized model paths for different workloads.
• Privacy-Aware: Capable of keeping data within specific boundaries during the routing process.