aurelio-labs/semantic-router

Superfast AI decision making and intelligent processing of multi-modal data.

What it solves

Semantic Router provides a high-speed decision-making layer for LLMs and AI agents. It eliminates the need to wait for slow LLM generations to decide which tool to use or which path to take, reducing latency and improving control over conversation flows.

How it works

Instead of using an LLM to classify intent, the project uses semantic vector space. It allows developers to define Route objects containing a name and a set of example utterances. A RouteLayer then uses an embedding model (encoder) to compare the user's query against these predefined routes to find the most semantically similar match, routing the request to the appropriate path.

Who it’s for

Developers building AI agents and LLM-powered chatbots who need fast, deterministic routing of user inputs to specific functions, tools, or conversational prompts without the overhead of a full LLM call.

Highlights

  • Fast Decision Making: Uses embeddings rather than LLM generation for routing decisions.
  • Flexible Encoders: Supports multiple embedding providers including OpenAI, Cohere, Hugging Face, and FastEmbed.
  • Local Execution: Ability to run fully local routing using HuggingFaceEncoder and LlamaCppLLM.
  • Vector Database Integration: Integrates with Pinecone and Qdrant for managing utterance vector spaces.
  • Multi-modal Support: Capable of handling multi-modal routes for identifying different types of visual input.
  • LangChain Integration: Ready-to-use integration with LangChain Agents.

Related

  • Dispatch
    vLLM Semantic Router: Enhancing Model Performance via Micro-Agent Collaboration

    vLLM introduces the Semantic Router, a serving-layer primitive that transforms a single model API call into a bounded collaboration of micro-agents to outperform frontier models on complex benchmarks.

  • Project
    NVIDIA-AI-Blueprints/llm-router

    An experimental multimodal router that analyzes user prompts to recommend the optimal LLM or VLM based on trade-offs between accuracy, speed, and cost.

  • Project
    vllm-project/semantic-router

    A programmable routing layer for Mixture-of-Models systems that optimizes LLM request routing across heterogeneous compute and infrastructure to improve cost, latency, and quality.

  • Project
    SomeOddCodeGuy/WilmerAI

    A node-based workflow engine for advanced semantic prompt routing and multi-LLM orchestration, allowing users to create complex, context-aware AI agents.

  • Project
    microsoft/semantic-kernel

    An enterprise-ready orchestration framework and SDK for building, orchestrating, and deploying AI agents and multi-agent systems across multiple LLM providers.