Decision Models in Hugging Face – New Interface for Structured Probabilistic Outputs

TL;DR

Hugging Face introduced a decision‑model API that accepts a context (state) and structured questions, then returns choices, scores, or binary probabilities with confidence values—eliminating the need for text generation when applications only require structured decisions.

What decision models are

A decision model consumes a state (any input data) and a set of questions defined by three primitives—choice, score, and noul—and returns a decision object containing the selected answer (or expectation) and a full probability distribution.

State + Questions + Allowed Answers → Decisions + Probabilities

The model does not generate free‑form text; it directly evaluates logits for each possible answer and applies softmax, yielding calibrated probabilities that applications can consume programmatically.

Why replace generic LLM prompting?

Using a standard LLM to emit JSON suffers from three drawbacks:

  1. Schema violations – models may produce malformed JSON or unexpected keys.
  2. Decoding overhead – constrained‑generation tools (e.g., Outlines, XGrammar) enforce schemas but add computational cost.
  3. Unnecessary generation – autoregressive decoding computes full token sequences even when only a discrete decision is needed.

Decision models avoid these issues by bypassing token‑level generation entirely.

Comparison with existing classification approaches

Property Traditional classifier Zero‑shot classifier Decision model
Probability over choices ✅ ✅ ✅
Labels defined at inference ❌ ✅ ✅
Multiple typed questions per request ❌ ❌ ✅
Native support for ordered scores & binary Bernoulli decisions ❌ (task‑specific) ❌ (requires adaptation) ✅

Traditional classifiers have a fixed head; adding a new label requires architectural changes and retraining. Zero‑shot classifiers allow runtime labels but are limited to a single classification task per request. Decision models extend this paradigm to multiple, typed questions in a single call.

The three primitives in detail

Choice – probability distribution over discrete options

The model returns a logit per option, softmaxed to probabilities. The highest‑probability option is reported as choice, and a confidence metric quantifies how peaked the distribution is.

{
  "department": {
    "type": "choice",
    "choice": "billing",
    "confidence": 0.9974,
    "probabilities": {
      "billing": 0.9983,
      "technical": 0.0010,
      "sales": 0.0007
    }
  }
}

Score – expected value of an ordered scale

For ordinal questions (e.g., urgency), the model returns a probability for each level and computes the expected index as the score.

{
  "urgency": {
    "type": "score",
    "score": 1.70,
    "legend": {"0":"low","1":"medium","2":"high"},
    "confidence": 0.558,
    "probabilities": {"0":0.015,"1":0.265,"2":0.720}
  }
}

Noul – Bernoulli (yes/no) probability

A binary decision returns a single probability noul representing P(yes).

{
  "refund": {"type": "noul", "noul": 0.995}
}

Confidence metric

For choice and score questions, confidence measures how much the top probability exceeds a uniform baseline:

$$C = \frac{p_{\text{max}} - \frac{1}{K}}{1 - \frac{1}{K}}$$

where K is the number of options. A uniform distribution yields C = 0; a deterministic distribution yields C = 1.

Ecosystem integration

1. Serving with 🤗 Transformers

  • New classes: DecisionPipeline, AutoModelForDecision, DecisionConfig.
  • Same API as other pipelines; supports the /v1/systemone HTTP endpoint.
  • Example curl request returns a JSON object with three typed answers and token usage statistics.

2. Serving with llama.cpp

  • llama.cpp implements the identical /v1/systemone API, allowing GGUF decision models (e.g., ggml-org/Kev-4B-GGUF) to be served without code changes.
  • Demonstrates true interface compatibility across back‑ends.

3. Discovering models on the Hub

  • A dedicated decision-model tag aggregates models regardless of architecture (encoder‑only, generative LLMs, multimodal variants).
  • Users can browse trending decision models via the Hub UI.

4. Training decision models

  • Decision models are first‑class objects in the Transformers library; they can be fine‑tuned with the standard Trainer API.
  • A toy example fine‑tunes JEV-9B on a small routing dataset, showing how to prepare inputs with model.decision_config.prepare_inputs and compute cross‑entropy loss on choice logits.
  • Upcoming TRL support (huggingface/trl#8040) will simplify training pipelines.
  • Unsloth provides a LoRA‑based recipe to convert generative LLMs (e.g., Qwen) into decision models.

Implications for developers

  • Reduced latency – No token‑by‑token generation; a single forward pass yields all answers.
  • Higher reliability – Structured outputs guarantee schema compliance, removing post‑processing validation.
  • Flexibility – Runtime‑defined answer sets and multiple question types enable complex decision‑making workflows (routing, scoring, binary verification) in a single request.
  • Ecosystem cohesion – A shared interface lets inference servers, model hubs, and training tools interoperate without model‑specific adapters.

Final thoughts

Decision models are not a reinvention of classification; they are a standardized, non‑generative contract that makes probabilistic decisions a first‑class output for software. By unifying choice, score, and binary primitives under a common API, Hugging Face enables reliable, calibrated decision‑making across diverse model back‑ends and downstream applications.

Sources