vllm-project/semantic-router
Intelligent Mixture-of-Models Router for Efficient Heterogeneous LLMs Inference
What it solves
It addresses the fragmentation in LLM inference by allowing developers to move beyond a single-model approach. Instead of relying on one general-purpose model, it enables the creation of "Mixture-of-Models" architectures where requests are routed to the most appropriate model based on specific signals and user preferences.
How it works
The system acts as a routing layer that turns signals and preferences into executable model paths. This allows it to route workloads across heterogeneous compute environments (such as GPUs, accelerators, edge, and cloud) and different specialized models, ensuring that data remains within its required boundaries while optimizing for the specific needs of the workload.
Who it’s for
It is designed for developers and organizations building production LLM applications that need to optimize inference by distributing tasks across multiple specialized models and diverse hardware environments.
Highlights
- Mixture-of-Models Architecture: Enables the composition of personalized model paths rather than using a single model.
- Heterogeneous Compute Routing: Routes requests across edge, private cloud, and public cloud accelerators.
- Signal-Driven Decisioning: Uses signals to determine the optimal model path for every user and workload.
- Production-Ready: Includes features for stateful production routing and real-time hallucination detection.