semantic-router: a superfast decision-making layer for LLMs and agents using semantic vector space for routing

What it solves

Semantic Router provides a high-speed decision-making layer for LLMs and AI agents. It eliminates the need to wait for slow LLM generations to decide which tool to use or how to route a request, reducing latency and improving response times.

How it works

Instead of using an LLM to classify a query, the project uses semantic vector space. It allows developers to define Route objects containing a set of example utterances. When a user query arrives, the project encodes the query into a vector and compares it to the route's utterances to determine the most appropriate path based on semantic meaning.

Who it’s for

Developers building LLM-powered applications and AI agents who need a fast, efficient way to handle intent classification and request routing without relying on expensive or slow LLM calls.

Highlights

• Fast Decision Making: Uses vector embeddings rather than LLM generation for routing decisions.
• Flexible Encoders: Supports multiple embedding providers including Cohere, OpenAI, Hugging Face, and FastEmbed.
• Local Execution: Ability to run fully local versions using HuggingFaceEncoder and LlamaCppLLM.
• Vector DB Integration: Integrates with Pinecone and Qdrant for managing utterance vector spaces.
• Multi-modal Support: Capable of routing based on multi-modal inputs (e.g., identifying images).
• Agent Framework Integration: Works with LangChain Agents.