NVIDIA-AI-Blueprints/llm-router

Route LLM requests to the best model for the task at hand.

What it solves

Modern AI systems often face a trade-off between accuracy, speed, and cost. This project provides a router that automates these trade-offs by analyzing user prompts (text or multimodal) and recommending the most appropriate LLM or Vision-Language Model (VLM) for the task, ensuring the most efficient model is used for each specific request.

How it works

The system consists of a Router Backend built on the NVIDIA NeMo Agent Toolkit that exposes an OpenAI-compatible API. It employs two distinct routing strategies:

  1. Intent-Based Routing: Uses a small LLM (Qwen 1.7B) to classify the user's intent (e.g., "image understanding" or "hard question") and maps that intent to a pre-configured model.
  2. Auto-Routing: Uses CLIP embeddings to encode text and images, then passes them through a trained neural network to predict the optimal model based on quality, latency, and cost metrics.

Unlike earlier versions, this router does not proxy the request; it returns the recommended model name, and the calling application is responsible for executing the API call to that model.

Who it’s for

  • AI Engineers and Developers: Those exploring multimodal routing approaches.
  • MLOps Teams: Teams looking to implement learning-based routing optimization and custom model selection strategies.
  • Research Teams: Those evaluating different routing strategies for production deployment.

Highlights

  • Multimodal Support: Capable of routing based on both text and image inputs.
  • Two Routing Strategies: Offers both zero-shot intent classification and data-driven neural network routing.
  • OpenAI API Compliant: Returns model recommendations via a standard chat completions endpoint.
  • Flexible Configuration: Allows users to update intent mappings or train custom neural network routers on their own data.

Related

  • Project
    ulab-uiuc/LLMRouter

    An open-source library for intelligent LLM routing that dynamically selects the optimal model for each query to balance cost and performance.

  • Project
    aurelio-labs/semantic-router

    A fast decision-making layer for LLMs and agents that uses semantic vector space to route requests based on meaning rather than waiting for slow LLM generations.

  • Dispatch
    vLLM Semantic Router: Enhancing Model Performance via Micro-Agent Collaboration

    vLLM introduces the Semantic Router, a serving-layer primitive that transforms a single model API call into a bounded collaboration of micro-agents to outperform frontier models on complex benchmarks.

  • Project
    vllm-project/semantic-router

    A programmable routing layer for Mixture-of-Models systems that optimizes LLM request routing across heterogeneous compute and infrastructure to improve quality, cost, and latency.

  • Project
    SomeOddCodeGuy/WilmerAI

    A node-based workflow engine and semantic router that orchestrates multiple LLMs and tools using conversation-aware logic to handle complex tasks.