NVIDIA-NeMo/Switchyard

Switchyard – Cost‑aware routing for LLM calls

What it is – Switchyard is a lightweight routing layer that sits in front of one or more large‑language‑model (LLM) providers. For each request it decides, based on a configurable algorithm, whether to send the call to a cheap efficient model or a more capable (and usually more expensive) model, aiming to keep overall cost down while preserving task accuracy.

Why it matters – In many agent‑oriented workloads a single strong model is overkill for the majority of turns. By automatically “escalating” only the hard cases, Switchyard can cut token‑costs by 13‑30 % with only a modest drop (or even a small gain) in success rate, as shown in the Terminal‑Bench 2.1 benchmarks.

---

Core components (pre‑1.0)

Component	Stability	Role
switchyard-libsy	Beta	Pure Rust library exposing the routing algorithms. Embed it in your own gateway or harness; you keep control of HTTP calls, retries, and credentials.
switchyard-llm-client	Alpha	Helper that translates between the library’s neutral request/response types and real HTTP LLM APIs.
switchyard-runner	Alpha	Glue that runs a routing configuration inside another runtime (e.g., NeMo Relay).
switchyard-server	Demo	Stand‑alone proxy that mimics OpenAI/Anthropic endpoints; useful for quick demos or evaluation.

---

How you can use it

1. NeMo Relay plugin – Load a routes.toml file into an existing NeMo Relay deployment. The plugin handles the HTTP dispatch to the selected provider while Relay keeps its usual transport and retry logic.
2. Embed the library – Install the Python package (pip install nemo-switchyard) or add the Rust crate to your project. Build a routing algorithm (e.g., stage_router(picker="efficient_first", confidence_threshold=0.5)) and drive it with a run_stream loop that you feed model‑specific HTTP clients.
3. Standalone proxy – Install the Rust binary (cargo install switchyard-server), write a simple routes.toml that maps target IDs to OpenRouter (or any OpenAI‑compatible) endpoints, and start the server. Any OpenAI/Anthropic client can then point at http://localhost:4000 and will automatically benefit from routing.

---

Routing algorithms (pick one in the TOML config)

• Capability (llm_classifier) – First request is judged by a small LLM; if it deems the task hard, the call is escalated.
• Stage (stage_router) – Uses pattern matching or a judge LLM on tool‑generated responses before deciding to switch models.
• Escalation – Starts with the cheap model, then runs a judge LLM on the result; if problems are detected, the request is re‑sent to the capable model.
• Random / Advisor / Sub‑Agent‑Aware / Custom – Various other strategies for experimentation.

---

Getting started quickly (Python example)

from switchyard.libsy import LlmResponse, Step
from switchyard.libsy.algorithms import stage_router

algorithm = stage_router(picker="efficient_first", confidence_threshold=0.5)

async def route(request, clients):
    async for step in algorithm.run_stream(request, {
        "efficient": ["fast"],
        "capable":   ["quality"],
    }):
        if isinstance(step, Step.CallModel):
            # call your own HTTP client for each candidate model
            resp = await clients[step.models[0]].call(step.request)
            step.respond(LlmResponse.Agg(resp))
        elif isinstance(step, Step.Done):
            return step.outcome.response

The same logic exists in Rust; the switchyard-libsy crate provides the Algorithm::run_stream iterator.

---

Maturity & licensing

• The project is pre‑1.0; APIs and configuration formats may change, so pin the version you integrate.
• The server component is marked Demo and is not recommended for production use yet.
• Licensed under Apache 2.0 (NVIDIA Corporation).

---

Where to learn more

• Core concepts & TOML schema – docs/core_concepts.md and docs/reference/toml_schema.md
• Routing algorithm details – docs/routing_algorithms/…
• Benchmark results – see the “Benchmark Provenance” table in the README.
• Community – open GitHub issues and a Code of Conduct are provided.

Switchyard lets you drop a routing layer into any existing LLM gateway (NeMo Relay, LiteLLM, or a custom server) and start saving on token costs without rewriting your agents.