ulab-uiuc/LLMRouter

What it solves

LLMRouter optimizes LLM inference by dynamically selecting the most suitable model for a given query. This prevents the waste of expensive, high-capacity models on simple tasks while ensuring complex queries are routed to the most capable models, thereby balancing performance, cost, and latency.

How it works

The system treats routing as a sequential decision process and provides a modular framework with over 16 routing strategies. It uses a data generation pipeline to transform benchmark datasets into routing data, which is then used to train routers. These routers can be based on various techniques including K-Nearest Neighbors (KNN), Support Vector Machines (SVM), Multi-Layer Perceptrons (MLP), Graph-based methods, and personalized user-preference learning.

Who it’s for

It is designed for developers and researchers building LLM applications who need to reduce inference costs and optimize response quality across a diverse set of available models.

Highlights

• Diverse Routing Strategies: Supports 16+ routers across five categories: single-round, multi-round, multimodal, agentic, and personalized.
• Comprehensive Pipeline: Includes a full data generation pipeline supporting 11 benchmark datasets and multimodal tasks.
• Unified Interface: Provides a CLI, a Gradio-based UI, and a ComfyUI visual interface for constructing routing pipelines.
• Production Ready: Integrates with OpenClaw to deploy as an OpenAI-compatible API server supporting streaming and multimodal understanding.