vLLM Semantic Router Fusion primitive enables programmable multi‑model serving

vLLM Semantic Router Fusion primitive enables programmable multi‑model serving

TL;DR

vLLM released the Fusion primitive for its Semantic Router, enabling production systems to run coordinated panels of heterogeneous models, judge their outputs, and synthesize a single response while keeping policy, configuration, and tracing inside the router. This makes model‑mixing a first‑class, programmable serving pattern rather than an ad‑hoc experiment.

The Need for More Than a Single Model

Traditional serving asked only which single model should handle a request? Modern AI applications now require a portfolio of models—fast cheap ones, private on‑prem models, specialist reasoning models, and external provider APIs. Operators must decide when a request can be satisfied by a single model and when it should trigger a coordinated multi‑model workflow that respects cost, latency, privacy, and safety policies.

Fusion as a Router Primitive

Fusion is introduced as a routing algorithm, not a global endpoint. It is activated by the router’s signal‑decision layer and follows a clear pipeline:

  1. Signal extraction – the request is annotated with domain, complexity, safety, etc.
  2. Decision making – the router chooses either a normal route or a Fusion route based on those signals.
  3. Fusion entry – using model: "vllm-sr/fusion" forces matching only against Fusion‑capable decisions.
  4. Panel execution – a set of analysis models generate independent candidate answers concurrently.
  5. Judging – a judge model evaluates consensus, contradictions, gaps, and unique insights.
  6. Synthesis – the judge (or a separate synthesis model) produces a single user‑facing response, optionally emitting OpenAI‑compatible tool_calls.
  7. Tracing – the router records which models ran, failures, token usage, and the structured analysis for debugging and accounting.

The design keeps each stage explicit and observable, allowing operators to configure per‑decision panel composition, error handling (on_error: skip vs. fail), concurrency limits, and runtime knobs.

Entry Paths and Policy Control

Entry path Behavior
model: "vllm-sr/auto" Runs full signal/decision logic; Fusion runs only if the selected decision’s algorithm.type is fusion.
model: "vllm-sr/fusion" Signals are still extracted, but only Fusion‑capable decisions are considered; a clear error is returned if none match.
Request plugin {"id": "fusion", ...} Overrides panel, judge, and runtime knobs for a single request, building a scoped Fusion execution even without a matching decision.

This separation lets operators keep Fusion as an optional, cost‑controlled feature rather than a default for every request.

Evidence from OpenRouter

OpenRouter’s recent Fusion launch (DRACO benchmark) showed that panels of diverse models can outperform the strongest single model. Their reported scores include:

Configuration DRACO Score
Fusion: Fable 5 + GPT‑5.5, synthesized by Opus 4.8 69.0%
Fusion: Opus 4.8 + GPT‑5.5 + Gemini 3.1 Pro, synthesized by Opus 4.8 68.3%
Solo Claude Fable 5 65.3%
Solo DeepSeek V4 Pro 60.3%

The budget‑panel rows illustrate that a mix of cheaper models can recover quality lost by any single inexpensive model—precisely the trade‑off a router should manage.

Detailed Fusion Workflow

  1. Resolve policy – merge decision‑level Fusion config with any request‑level overrides.
  2. Protect the router – Fusion slugs cannot be used as panel or judge models, preventing recursive Fusion calls.
  3. Run the panel – all analysis models are invoked concurrently, respecting max_concurrent.
  4. Handle failures – on_error: skip continues with remaining models; on_error: fail aborts immediately.
  5. Judge analysis – the judge returns a structured JSON describing consensus, contradictions, partial coverage, unique insights, and blind spots.
  6. Synthesize or tool‑call – the final step produces either a plain answer or an OpenAI‑compatible tool_calls response.
  7. Return trace – the response payload can include the full Fusion trace, intermediate panel outputs, failure records, and aggregated token usage.

Because the trace is explicit, operators can debug routing decisions, monitor cost, and improve policies based on real‑world disagreement patterns.

Fusion Is a Decision, Not a Default

Fusion adds latency and token cost, so the router must decide when it is worthwhile. With vllm-sr/auto, the router evaluates signals (e.g., request complexity, domain, tenant policy) and selects a Fusion decision only for high‑risk or high‑value queries. Simple prompts continue to use fast single‑model routes. The explicit vllm-sr/fusion alias lets clients force Fusion when they know the extra perspective is needed.

Example API Calls

Let the Router Choose

{
  "model": "vllm-sr/auto",
  "messages": [{"role": "user", "content": "What are the strongest arguments for and against carbon taxes?"}]
}

If the matched decision specifies algorithm.type: fusion, the request follows the Fusion pipeline; otherwise it proceeds with the selected single model.

Force Fusion

{
  "model": "vllm-sr/fusion",
  "messages": [{"role": "user", "content": "What are the strongest arguments for and against carbon taxes?"}]
}

Only Fusion‑capable decisions are considered; a missing match yields a clear error.

Override Panel for One Call

{
  "model": "vllm-sr/fusion",
  "messages": [{"role": "user", "content": "..."}],
  "plugins": [{
    "id": "fusion",
    "model": "google/gemini-3-flash-preview",
    "analysis_models": [
      "google/gemini-3-flash-preview",
      "moonshotai/kimi-k2.6",
      "deepseek/deepseek-v4-pro"
    ]
  }]
}

The override is scoped to this request and does not modify global routing config.

Fusion in Agent Loops

Fusion works with OpenAI‑compatible tool calls. Panel models receive the conversation history but do not see tools or tool_choice. Only the final judge can emit tool_calls.

{
  "model": "vllm-sr/fusion",
  "messages": [{"role": "user", "content": "Find the latest benchmark result and explain whether it changes our launch plan."}],
  "tools": [{
    "type": "function",
    "function": {"name": "web_search", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}
  }],
  "tool_choice": "auto"
}

The panel produces analysis; the judge decides whether to answer directly or return a tool_calls payload.

Configuration Layout

Global runtime config only registers entry aliases:

global:
  router:
    auto_model_names:
      - vllm-sr/auto
      - auto
      - MoM

Fusion slugs are registered under the looper integration:

global:
  integrations:
    looper:
      fusion:
        model_names:
          - vllm-sr/fusion

Per‑decision routing config holds the actual Fusion policy:

routing:
  decisions:
    - name: deep-research-fusion
      description: Use model diversity for research prompts with high synthesis risk.
      rules:
        operator: AND
        conditions:
          - type: domain
            name: research
          - type: complexity
            name: needs_reasoning:hard
      algorithm:
        type: fusion
        fusion:
          model: google/gemini-3-flash-preview
          analysis_models:
            - google/gemini-3-flash-preview
            - moonshotai/kimi-k2.6
            - deepseek/deepseek-v4-pro
          max_concurrent: 3
          on_error: skip

This separation keeps global state minimal while allowing fine‑grained, workload‑specific Fusion policies.

Future Directions

The OpenRouter DRACO results motivate a systematic evaluation of Fusion within vLLM‑SR:

  • Large‑scale public benchmarks beyond smoke tests.
  • Comparisons among Fusion, ReMoM, AutoMix, Router‑R1, and single‑model baselines.
  • Analyses of budget‑panel vs. frontier‑model panels.
  • Enhanced trace diagnostics for disagreement, coverage gaps, and judge behavior.
  • Policy research on latency‑cost trade‑offs to decide when Fusion is justified.

The overarching vision is clear: the best answer will increasingly come from a system of models orchestrated by a programmable router, not from the largest single checkpoint. vLLM‑SR’s Fusion primitive makes that system observable, configurable, and production‑ready.

References

Sources