vLLM Semantic Router v0.2 Athena release notes / what's new
vLLM Semantic Router v0.2 Athena transforms semantic routing from a simple request bridge into a strategic "system brain" for mixture-of-models and multi-agent deployments. This release introduces a completely rebuilt model foundation, the experimental ClawOS operating layer for orchestrating OpenClaw systems, and a sophisticated model selection framework that allows for dynamic, cost-aware, and quality-aware routing.
Rebuilt Model Foundation and Runtime Acceleration
Athena replaces its fragmented base-model architecture with a unified, long-context multilingual foundation. The system now centers on mmbert-embed-32k-2d-matryoshka (307M parameters, 32K context, supporting 1800+ languages) and the mom-multilingual-class classifier family.
Key Model Components
multi-modal-embed-small: A ~120M parameter embedding model that maps text, images, and audio into a shared 384-dimensional space, enabling cross-modal retrieval with a reported Audio-Text Retrieval R@1 of 36.4%.mmbert-embed-32k-2d-matryoshka: Provides a production-ready backbone with 32K context and 2D Matryoshka controls, allowing truncation from 768d to 256d with ~99% quality retention.mom-multilingual-class: A coherent classifier family covering five core tasks: Intent, Jailbreak, PII, Fact-check, and Feedback, available in both merged and LoRA forms.
Hardware Acceleration and Performance
Athena implements a model-aware ONNX rewrite and a custom ROCm kernel path via onnx-binding/ort-ck-flash-attn. By replacing dense SDPA attention subgraphs with com.ck::CKFlashAttention nodes, the system achieves significant latency reductions on AMD Instinct MI300X hardware.
Latency Comparison (ONNX + GPU vs. CPU):
| Request Size | ONNX + GPU avg | ONNX + CPU avg | Candle + CPU avg |
|---|---|---|---|
| ~500 tokens | 22 ms | 853 ms | 1053 ms |
| ~2000 tokens | 31 ms | 1814 ms | 1805 ms |
| ~8000 tokens | 128 ms | 4796 ms | 1830 ms |
Flash Attention Scaling: CK Flash Attention enables the system to handle sequence lengths up to 32K tokens, where the previous SDPA path would result in Out-of-Memory (OOM) errors. At 4096 tokens, CK Flash Attention is 3.3x faster than SDPA.
Advanced Model Selection Primitives
Model selection is now a first-class routing primitive that occurs after a decision match has been made. Athena allows the system to choose the specific model to serve a request based on a variety of strategic algorithms:
- ML-based Selectors: Includes KNN (historical query similarity), KMeans (cluster-level patterns), SVM (nonlinear decision boundaries), and MLP (neural router via Candle).
- Advanced Strategies: Includes Latency-Aware (TPOT/TTFT data), Elo (user feedback/Bradley-Terry ratings), RouterDC (dual-contrastive similarity), AutoMix (cost-to-quality escalation), and Thompson Sampling (online exploration/exploitation).
- Reasoning-based: Router-R1 uses an external router model to reason about the request before selection.
ClawOS: Orchestration for OpenClaw
ClawOS is an experimental operating layer that allows Semantic Router to orchestrate multiple OpenClaw agent systems. It shifts the router's role from choosing a model to managing a multi-agent environment. Key capabilities include:
- Natural-language MCP control: Users can spin up and manage OpenClaw teams and workers via chat.
- Team Management: Support for explicit leader-and-worker compositions and shared room chats for real-time coordination.
- Operational Visibility: Dashboard views for runtime health, team composition, and worker provisioning.
Core Runtime Enhancements: Memory, RAG, and Signals
State and Memory
Athena integrates state directly into the core runtime, adding Agentic Memory with Milvus storage, hybrid memory search (combining vector similarity, BM25, and n-gram matching), and MINJA defenses to mitigate memory injection attacks.
Expanded Signal Layer
The signal layer now supports a broader range of inputs and matching paths:
- Rich Signals: Addition of language, latency, context, complexity-aware, modality, and authz signals.
- Deterministic Fast Path: Keyword routing is now less brittle through the addition of BM25, n-gram fuzzy matching, and regex, moving beyond exact literal matches.
- Safety Integration: Jailbreak and PII detection are now parallel signals, including contrastive multi-turn detection to catch gradual escalation attacks.
NLP-Based Prompt Compression
To optimize long-context signal extraction, Athena introduces a deterministic NLP pipeline (using TextRank, position weighting, TF-IDF, and novelty scoring). This compresses long prompts for the signal extraction path only, while the original full-fidelity prompt is sent to the serving model. In benchmarks, this reduced jailbreak signal extraction latency from 127 ms to 10 ms when compressing 16K tokens to 512 tokens.
Programmable Configuration and Deployment
Neural-Symbolic Configuration Language
Athena introduces a typed configuration language that combines neural signal extraction with symbolic decision evaluation. This DSL is supported by a full compiler and visual builder, enabling LLM-based coding agents to synthesize routing policies from natural-language specifications.
AMD ROCm First-Class Support
AMD ROCm is now a canonical deployment path. The vllm-sr serve --platform amd command enables ROCm image defaults, GPU-first configurations, and the loading of the AMD CK Flash Attention custom op.
Zero-Config Onboarding
Installation is streamlined via a one-line installer for macOS and Linux, moving the experience from a YAML-first to a dashboard-first onboarding flow where users can bootstrap minimal workspaces automatically.