vLLM Semantic Router Multimodal Routing and Vision Encoder Hardening

vLLM has expanded the Semantic Router (VSR) to support multimodal routing, allowing visual evidence from images, screenshots, and documents to act as typed signals within the same decision fabric as text. This transition moves VSR from prompt-level routing to request-level policy, enabling more precise routing based on the actual content of a multimodal request.

Multimodal Routing as Request-Level Policy

Multimodal routing in VSR is not simple image classification; it is the integration of visual signals into a Signal-Decision architecture. In this model, signals are independent observations that are composed using priority and boolean logic to determine the routing path.

By analyzing the full request rather than just the text prompt, VSR can now handle scenarios where the text is generic but the image contains the decisive evidence. For example:

• PII Risk: A request saying "Summarize this" paired with a passport image is routed as an identifier document with restricted handling.
• Domain Specialization: A vague question like "What does this show?" paired with a chest X-ray triggers a medical-domain policy and routes to a capable Vision-Language Model (VLM).
• Security Review: A "Find the bug" prompt paired with a code screenshot is identified as a code artifact, triggering security review paths and secret leakage checks.
• Out-of-Domain Rejection: A medical prompt paired with an unrelated image of a car is flagged as out-of-domain visual evidence, leading to a clarification or rejection path.

Resolving Vision Signal Inversion

During the implementation of multimodal routing using the multi-modal-embed-small (mmes) encoder, vLLM identified a critical failure where the deployed path was "confidently wrong." An 11-image probe across three verticals and 21 candidate labels showed an 82% inversion rate, where the signal was anti-correlated rather than merely noisy (e.g., medical X-rays were ranked closer to semiconductor candidates than medical ones).

The Misdiagnosis: Encoder Strength

Initial hypotheses suggested the compact multi-modal-embed-small encoder was insufficient for the task. However, testing revealed that other models—including SigLIP2-base, SigLIP-base (via Hugging Face), and the larger multi-modal-embed-large (mmEL)—all scored 10/10 on the same probe. Crucially, the mmes model loaded through the PyTorch reference path also scored 10/10, proving the encoder itself was not the problem.

The Root Cause: Implementation Drift

Investigation revealed a significant gap between the PyTorch reference path and the Rust/Candle binding path. A passport fixture test showed a cosine similarity of 0.7204 in PyTorch versus 0.1576 in the Candle-binding path.

Three specific implementation errors in the Candle path were identified and corrected:

1. Pooling Head Mismatch: The SigLIPVisionEncoder::forward implementation was using BERT-style mean + Linear + tanh pooling instead of the required attentional probe pooling head. This was addressed in PR #1927.
2. Normalization Errors: The Go image loader provided pixels in [0, 1], but SigLIP requires per-channel normalization (x - 0.5) / 0.5. This was fixed in PR #1928.
3. Preprocessing Drift: The Go-side resize path used 4-tap bilinear implementation, differing from the PIL-style preprocessing in the PyTorch reference. PR #1943 moved image decoding, resizing, and conversion into Rust using the image crate with Catmull-Rom filtering to approximate PIL bicubic + antialias behavior.

Validation and Performance

Following these fixes, the production path was validated against the PyTorch reference. A three-vector isolation experiment on a passport fixture showed that the full branch-stack pipeline achieved a cosine similarity of 0.999902 compared to the reference.

Across a 20-image corpus, the system achieved a minimum cosine of 0.999557 and a mean of 0.999919, with 20/20 images scoring $\ge 0.999$ against the PyTorch reference.

Regarding performance, classifier signals in VSR run concurrently via runSignalDispatchers, meaning wall-clock latency is bounded by the slowest enabled classifier. In representative traces, full classification decisions complete in approximately 1.3 seconds on CPU.

Future Roadmap for VSR

With a reliable vision signal path, vLLM intends to evolve VSR into a comprehensive request-level control plane. Future architectural goals include:

• Integrating image-derived signals directly into the same decision layer as text signals.
• Ensuring multimodal decisions are visible in replay, metrics, and debugging tools.
• Making model selection aware of both policy fit and modality capability.
• Maintaining high-fidelity inspection for safety-critical signals like PII and jailbreak.
• Extending the routing fabric to agentic workflows, including tool calls and memory writes.