NVIDIA-NeMo/Switchyard

NVIDIA NeMo Switchyard – Quick Overview

What it is – Switchyard is an open‑source routing library from NVIDIA NeMo that sits between an AI agent (or any LLM‑powered service) and a pool of language models. Instead of sending every request to a single, often expensive, model, Switchyard decides which model should handle each request, letting you blend cheap, fast models with larger, more capable ones.

Why it matters – In production LLM deployments you constantly trade off three things:

• Task success (does the model produce a good answer?)
• Cost (how much you pay per token or API call)
• Latency (how long the user waits). Switchyard gives you a programmable way to optimise those trade‑offs automatically, rather than hard‑coding a single model.

---

How it works (high‑level flow)

1. Define a model pool – List the models you have access to (e.g., a cheap 7B model, a larger 70B model, an Anthropic Claude, etc.).
2. Pick a routing algorithm – Switchyard ships with several strategies (Auto, Task‑based, Execution‑based, Composite, plus more advanced ones like escalation or sub‑agent). The algorithm looks at the incoming request and, optionally, recent tool‑use signals from the agent.
3. Routing decision – The algorithm returns the ID of the model that should handle the request.
4. Your gateway forwards the request – The surrounding service (OpenRouter, LiteLLM, NeMo Relay, or a custom harness) actually calls the chosen model and returns the response to the user.
5. Iterate – As the agent continues its work, the router can re‑evaluate and switch models mid‑conversation if needed.

---

Main features

Feature	What you get
Multiple integration points	Use Switchyard via existing gateways (OpenRouter, LiteLLM, NeMo Relay), run a local proxy server for quick demos, or embed the Rust/Python library directly in your own service.
Pluggable routing algorithms	• Auto – sensible default (efficient‑first, confidence 0.5, no classifier). • Task – a classifier model decides if the cheap model can handle the request. • Execution – looks at recent tool usage and outcomes (stage‑router). • Composite – combines Task and Execution for more robust decisions. • Additional strategies: Plan/Execute, Escalation, Advisor, Sub‑agent, Random.
Config‑driven	All routing behaviour is expressed in a TOML file, making it easy to experiment without code changes.
Cost‑vs‑accuracy evaluation	Built‑in benchmarking docs and visualisations (e.g., cost‑accuracy chart) to compare routing setups against single‑model baselines.
Extensible components	Separate crates for library (switchyard-libsy), HTTP client (switchyard-llm-client), runner (switchyard-runner), and a demo proxy server (switchyard-server).
Open‑source, Apache‑2.0	Free to use, modify, and embed in commercial products (subject to NVIDIA’s license).

---

Getting started options

Path	Typical user	Steps (brief)
OpenRouter gateway	Anyone already using OpenRouter’s API	Set model=nvidia/switchyard in your API calls; OpenRouter hosts the routing service.
LiteLLM plugin	Developers who run their own inference stack	Clone the repo, follow the examples/litellm quick‑start, pin LiteLLM 1.102.0. Note: only Stage and Random routing are supported.
NeMo Relay plugin	Users of NVIDIA’s NeMo Relay serving framework	Build the Rust plugin (crates/switchyard-nemo-relay-plugin) with a Rust toolchain and Python 3, then configure the Relay server.
Local proxy server	Quick demos, evaluation, or personal use	Run switchyard-server (see docs/getting_started.md#server-path) and point your client at the local endpoint.
Embedded library	Teams building a custom gateway or agent harness	Add switchyard-libsy (Rust) or use the Python example examples/libsy.py; supply your TOML config and call the routing API directly.

---

Who should use Switchyard?

• Product teams that need to keep LLM costs low while preserving quality.
• RAG / agent developers who want the model to change mid‑task based on tool usage.
• Researchers evaluating routing strategies or building new routing algorithms.
• Ops engineers looking for a plug‑and‑play proxy that can be benchmarked against a single‑model baseline.

---

Limitations & maturity

• The project is pre‑1.0; APIs and behaviour may change. Pin a specific version for production.
• Some integrations (e.g., LiteLLM) are marked experimental and lack support for intermediate model calls required by certain algorithms.
• When using NeMo Relay, versions 0.8.x/0.9.0 may lose upstream error details for routes outside Switchyard.

---

License & community

• Apache 2.0 – permissive open‑source license.
• Issues and contributions are handled via GitHub (issues, PRs, code of conduct).

---

TL;DR

Switchyard is a routing layer that lets you dynamically pick the best LLM from a pool for each request, balancing cost, latency, and success. It can be used through popular gateways, as a local demo server, or embedded directly in your own service, and it offers several ready‑made routing strategies configurable via TOML.