vLLM Semantic Router v0.3 Themis release notes

vLLM Semantic Router v0.3 Themis release notes

vLLM Semantic Router v0.3, codenamed Themis, transitions semantic routing from a composable tool to a stateful, observable, and production-ready control plane. The release focuses on operational stability, providing a unified configuration contract and a structured pipeline that transforms raw signals into model selection decisions.

Canonical Configuration and Operational Contract

Themis introduces a canonical config.yaml shape to eliminate overlapping layouts across different deployment surfaces (Docker, Helm, Dashboard, and CRDs). This unified contract ensures that policies are consistent whether deployed locally or in Kubernetes.

Key configuration changes include:

  • Unified Shape: The config now follows a strict structure: version, listeners, providers, routing, and global.
  • Simplified Workflow: The vllm-sr init command is removed. Users now use vllm-sr serve for dashboard-first setup or author config.yaml directly.
  • Migration Path: Older configurations can be updated using vllm-sr config migrate --config old-config.yaml.
  • Strict Validation: The system now warns on unknown YAML fields and aligns Python CLI models with Pydantic to prevent silent routing drift.

The Routing Pipeline: Signals, Projections, and Decisions

Themis formalizes the routing mental model into a five-layer pipeline to ensure that routing intelligence remains programmable and explainable.

Layer Responsibility
Signal Extracts evidence from requests, responses, tools, modality, identity, or safety classifiers.
Projection Normalizes raw evidence into policy-ready concepts (e.g., "urgency" or "balance").
Decision Matches named routing policies based on priority and explainable conditions.
Algorithm Selects a specific model from candidates within a matched decision.
Model Serves the request via the selected backend alias or provider.

Signal Families

The router supports a broad catalog of signals, including authz (roles), complexity (reasoning difficulty), context (window demand), conversation (tool-loop shape), domain (business/legal/health), embedding (semantic similarity), event (operational metadata), fact_check (verification needs), jailbreak (injection evidence), kb (knowledge-base matches), keyword (literal/fuzzy matches), language (locale), modality (AR/diffusion/text), pii (sensitive entities), preference (user style), reask (dissatisfaction detection), structure (regex/density), and user_feedback (correction requests).

Projections

Projections transform noisy signal evidence into stable policy bands. This prevents the need to repeat low-level signal thresholds across multiple decisions. Themis supports three projection patterns:

  • Partitions: Selects one winner from an exclusive family.
  • Scores: Combines signals or KB metrics into a continuous value.
  • Mappings: Converts values into policy bands (e.g., support_fast or support_escalated) via calibrated thresholds.

Session-Aware Agentic Routing (SAAR)

SAAR provides stateful routing for multi-turn agentic workloads, ensuring session continuity and preventing unsafe model switches during active tool loops.

Core SAAR capabilities include:

  • Session Memory: Router-owned memory that preserves facts and preferences without requiring a separate session-state DSL.
  • Continuity Rules: Evaluates whether a model switch is justified by weighing quality gaps, switch margins, and prefix locality.
  • Hard Locks: Prevents model switches during active tool loops or provider-state continuations to maintain stability.
  • Diagnostics: Replay records preserve the specific reasoning behind why a session stayed with or switched models.

Protocol Compatibility and Hardware Support

Expanded Protocol Support

Themis expands beyond OpenAI Chat Completions to include native Anthropic /v1/messages ingress, streaming with OpenAI SSE translation, and custom Anthropic upstream routing. The system uses protocol detection from request path headers and provides response headers to indicate when translation is lossy.

Hardware Backend Paths

Themis supports a heterogeneous inference stack across four primary paths:

  • NVIDIA CUDA & AMD ROCm: For served vLLM backends. The AMD path is validated via vllm-sr serve --platform amd.
  • Intel OpenVINO: A new binding for native C++ and Go integration of ModernBERT sequence classification and embedding inference.
  • CPU/Local: For development and smoke testing.

Reliability, Observability, and Performance

Long-Context Optimization

To reduce the cost of long-context routing, Themis introduces:

  • Online Calibration: Learns a calibration ratio from observed response usage to improve token estimation.
  • Chunked Attention: The native mmBERT embedding path now processes attention in query chunks to bound memory and prevent clipping on long inputs.
  • Prompt Compression Profiles: Named profiles (default, coding, medical, security, multi_turn) optimize signal extraction without altering the original prompt sent to the serving model.

Storage and Observability

The release hardens the storage layer with support for Qdrant vector search and Valkey (cache, vector store, and memory). It also replaces O(N) cache-LRU read paths with constant-time list-backed implementations and fixes PostgreSQL insert correctness for router replay.

Benchmarks

According to a snapshot of the RouterArena leaderboard, vLLM-SR has returned to the #1 position with a weighted Arena Score of 75.4, an accuracy of 76.0%, and a cost of $0.11 per 1K queries.

Future Roadmap: v0.4 Hermes

The upcoming Hermes release aims to create a "self-improving router." The roadmap focuses on closing the loop between GPU-scale performance research, DSL recipe tuning, and encoder-model fine-tuning, while further tightening SAAR's model-switch economics and expanding the operator debugging loop via the dashboard.

Sources