vLLM Session-Aware Agentic Routing (SAAR) Release

vLLM Session-Aware Agentic Routing (SAAR) Release

vLLM has introduced Session-Aware Agentic Routing (SAAR), a session-aware model selection policy within the vLLM Semantic Router. SAAR addresses the failures of single-turn prompt routers in long-horizon agent scenarios by introducing router-owned session memory, hard locks for tool loops, and prefix-cache-aware switch pricing, reducing model switches by 79.29% and eliminating unsafe switches across deterministic tests.

The Shift from Prompt Routing to Session Routing

Traditional prompt routing selects a model based on the current request's signals. However, LLM agents operate in sessions—planning, calling tools, and processing observations—where turns are only meaningful in the context of the preceding trajectory.

SAAR evolves the router's primary question from "Which model should handle this request?" to "Is it safe to switch models inside this session right now?"

Why Single-Turn Routing Fails for Agents

Single-turn routing can be locally optimal but sessionally incorrect. This leads to several failure modes:

  • Tool-loop breakage: A tool result may be routed to a model that did not initiate the tool call.
  • State loss: Non-portable provider-managed continuation IDs may be sent to the wrong physical backend.
  • Inefficiency: Warm prefix caches for frontier models may be discarded because a specific turn's message is short, increasing costs.
  • Observability gaps: Logical models (e.g., auto) become difficult to debug when the physical model serving each turn is unknown.

The SAAR Design and Architecture

SAAR adds a session-control layer to the existing Semantic Router pipeline. It utilizes five primary components to manage model selection:

Component Function Purpose
Router Memory Tracks last physical model, matched decision, phase, and cache evidence Provides session context without becoming application memory
Hard Locks Prevents switching during active tool loops or non-portable provider state Ensures correctness over cost/quality optimization
Reset Boundaries Permits reselection after idle timeouts or decision drift Prevents routing from becoming permanently "sticky"
Switch Economics Prices handoff costs and prefix-cache checkout Makes switching asymmetric based on session length and model tier
Replay Traces Records the reasoning behind stay/switch decisions Makes logical models like auto inspectable

Hard Continuity Constraints

SAAR enforces two "hard locks" where the router is forbidden from switching models regardless of the base selector's score:

  1. Tool-loop continuity: Tool results must return to the physical model that requested the tool call.
  2. Provider-managed state: Requests with non-portable continuation state must stay with the previous physical backend.

Conversely, Reset Boundaries (idle timeouts and decision drift) allow the router to reopen model selection when the value of continuity decays or the task shape changes.

Prefix Cache and Switch Economics

For long sessions, switching models is a systems decision. SAAR prices a cached-input checkout delta—the difference between normal prompt input pricing and cached-input pricing for the candidate model. As a session grows longer and more expensive, the policy becomes stricter about discarding prefix locality to avoid high input costs.

Operational Workflow and Observability

When a client sends a request with a stable x-session-id to a logical model (e.g., auto), SAAR processes the turn as follows:

  1. Extracts signals and runs the Semantic Router pipeline to get a base model selection.
  2. Loads session state from router memory.
  3. Applies hard locks (tool loops/provider state).
  4. Evaluates reset boundaries (idle/drift).
  5. Adjusts scores based on prefix-cache costs and switch history.
  6. Selects the physical model and emits a replay trace.

Replay traces allow operators to audit why a router stayed with or switched from a model, answering whether a decision was driven by a hard lock, a reset boundary, or cache economics.

Performance and Evaluation Results

Evaluation was conducted across deterministic policy matrices, live AMD ROCm serving, and agent-task traces.

Policy Matrix Results (21,600 turns)

Policy Switches Unsafe Switches Est. Cost Reduction Quality Delta
Single-turn 9,709 3,836 0.00% +0.0000
Sticky session 340 0 98.65% -0.1433
Full SAAR 2,011 0 78.71% -0.0453

Correctness and Reliability

  • Hard Lock Efficacy: Tool-loop switch violations dropped from 3,404 to 0; provider-state violations dropped from 432 to 0.
  • Live Serving: Across 2,896 live requests on AMD ROCm, 0 continuity violations were observed.
  • Fault Recovery: 100% of affected sessions (32/32) recovered after injected HTTP 503 backend faults.
  • Task Completion: 18/18 exact-scored task instances completed successfully with replay headers present on all routed turns.

Sources