ulab-uiuc/LLMRouter

LLMRouter: An Open-Source Library for LLM Routing

What it solves

LLMRouter addresses the challenge of optimizing LLM inference by dynamically selecting the most suitable model for a given query. It aims to balance task complexity, inference cost, and performance requirements, preventing the use of expensive, high-capacity models for simple tasks while ensuring complex queries are handled by the most capable models.

How it works

The system treats routing as a sequential decision process and provides a modular framework with over 16 routing strategies. It analyzes queries and routes them to an optimal LLM based on various methods:

  • Routing Categories: It supports single-round, multi-round, multimodal, agentic, and personalized routers.
  • Routing Methods: Strategies include K-Nearest Neighbors (KNN), Support Vector Machines (SVM), Multi-Layer Perceptrons (MLP), Matrix Factorization, Elo Rating, graph-based routing, and BERT-based routing.
  • Infrastructure: It includes a unified CLI for training and inference, a Gradio-based UI for interactive chat, and a ComfyUI interface for visual pipeline construction.
  • Data Pipeline: A built-in pipeline generates training data from 11 benchmark datasets, automating API calls and response evaluation to train the routers.

Who it’s for

This tool is designed for developers and researchers who want to reduce LLM operational costs and improve efficiency without sacrificing response quality, particularly those managing multiple LLM providers or deploying multimodal and personalized AI applications.

Highlights

  • Extensive Router Library: Over 16 routing models across five distinct categories.
  • Unified Infrastructure: Provides a CLI, Gradio UI, and a ComfyUI visual interface.
  • Comprehensive Benchmarking: Includes xRouteBench and a data pipeline supporting 11 benchmark datasets.
  • Flexible Deployment: Supports OpenAI-compatible APIs, local LLMs (via Ollama, vLLM, SGLang), and integration with OpenClaw for production API servers.
  • Multimodal Support: Capable of routing time-series, image, and video queries.

Related

  • Project
    NVIDIA-AI-Blueprints/llm-router

    An experimental multimodal router that analyzes user prompts to recommend the optimal LLM or VLM based on trade-offs between accuracy, speed, and cost.

  • Project
    vllm-project/semantic-router

    A programmable decision layer that routes AI requests to the most suitable models and compute backends based on quality, cost, and latency policies.

  • Project
    llm-d/llm-d

    A high-performance distributed inference serving stack for Kubernetes that optimizes LLM production deployments across various hardware accelerators.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    microsoft/LMOps

    A research initiative focused on fundamental technologies to improve LLM prompting, inference speed, and context handling for building AI products.