Model Routing in Agentic Systems: Moving from Classification to Optimization

Model routing in agentic systems is a systems optimization problem rather than a simple classification task. Effective routing requires balancing the interaction between model pricing, caching behavior, infrastructure state, and compliance constraints to find the optimal operating point for the entire system.

The Hidden Complexity of Model Cost

Actual operational cost is determined by the interaction between the model, the workload, and the serving infrastructure, making sticker price an unreliable metric for routing decisions.

In a test using a CodeAct agent on 417 tasks from the AppWorld Test Challenge, Claude Sonnet 4.6 cost $79 total ($0.19/task), while GPT-4.1 cost $155 ($0.37/task). This occurred despite GPT-4.1 having lower token pricing and Sonnet requiring roughly three times as many reasoning steps. The discrepancy was driven by caching: Sonnet's lower cache-read pricing significantly reduced effective input costs for agent workloads that reuse large chunks of context across steps.

Why Task Difficulty is an Incomplete Signal

Routing based solely on estimated task difficulty fails because difficulty is often invisible at the start of a request and must be balanced against other production constraints.

• Invisible Difficulty: A seemingly simple request, such as "summarize this contract," may trigger complex internal processes including retrieval, compliance checks, and multiple refinement rounds, meaning the actual difficulty is only apparent during execution.
• Systemic Constraints: In enterprise environments, routers must juggle quality and cost alongside data residency rules, privacy constraints, compliance requirements, and approved model lists.

Latency Beyond Model Speed

End-to-end latency is influenced more by infrastructure and routing overhead than by the raw speed of the model itself.

Factors that dominate response times include hardware specifications, cache warmth, and endpoint congestion. Additionally, the granularity of routing introduces a tradeoff: routing once per task adds minimal overhead, but routing at every step of an execution increases flexibility at the cost of higher operational complexity and latency.

Transitioning to an Optimization-Based Routing Approach

Treating routing as a multi-objective optimization problem allows for a flexible cost-accuracy frontier rather than a single fixed decision.

IBM Research shifted from asking "which model is best for this task?" to an algorithm that optimizes cost, quality, and latency simultaneously. Testing on the AppWorld Test Challenge with a CodeAct agent demonstrated that this approach provides a range of operating points:

• Latency-Optimized Configuration: Achieved 84% accuracy for $93 and 83s, representing a 21% cost reduction and 9% latency reduction compared to using Opus alone, with only a 4% drop in accuracy.
• Efficiency: The optimization process is lightweight, requiring approximately 6 ms and 2 kB of memory per task, ensuring the router does not become a system bottleneck.

Compared to standard difficulty-based routers, which may land in similar accuracy ranges but at higher costs, an optimization-based approach explores the full tradeoff space more effectively.

Conclusion: Routing as System Optimization

Model routing is not primarily about selecting the "best" model for a specific task, but about finding the best operating point for the entire system. Models are just one variable among caching behavior, infrastructure state, and workload patterns.