Building Custom Decision Models with Constrained Decoding

Decision Models vs. Generative LLMs

Decision models, often referred to as "System 1" models, shift the LLM paradigm from open-ended text generation to single-pass classification. While a standard LLM requires multiple inference passes to generate a structured response (e.g., 11 passes to output a simple JSON object), a decision model constrains the output vocabulary to a fixed set of predefined options. By masking all other tokens in the vocabulary, the model can emit the final answer in a single pass, significantly reducing latency and compute costs.

Implementing a Decision Model with Constrained Decoding

Building a decision model can be achieved by restricting the model's output tokens to a specific set of option identifiers (e.g., "A", "B", "C", "D", "E"). Using a model like Qwen/Qwen3-1.7B, the process involves:

Token Identification: Identify the token IDs for each allowed option.
Prompt Formatting: Structure the prompt to present the question and the options clearly.
Logit Masking: During the final step of the forward pass, extract the logits only for the allowed option tokens.
Probability Distribution: Apply a softmax function to these filtered logits to determine the probability of each option.

In testing against the CommonsenseQA dataset, a base 1.7B model achieved an accuracy of 59.38%, which improved to 62.41% after a targeted fine-tuning process.

The Challenge of Model Calibration

Raw output probabilities from an LLM often reflect the model's confidence in the next token rather than the actual probability of the answer being correct. This leads to overconfidence, where a model may assign a 99% probability to an answer that is only correct 70% of the time.

Temperature Scaling for Calibration

To align a model's confidence scores with its actual accuracy, temperature scaling can be used. By adjusting the temperature parameter ($T$), the output probability distribution is flattened.

Uncalibrated ($T=1.0$): The model may show high confidence (e.g., 94.3%) but have a high expected calibration error.
Calibrated ($T=2.2$ or higher): By increasing the temperature, the distribution is scaled to approximate the model's actual accuracy, reducing the expected calibration error (e.g., from 20.1% down to 2.9%).

Community Perspectives and Technical Critiques

The emergence of "decision models" has sparked debate among machine learning engineers regarding whether this is a fundamental architectural shift or a rebranding of traditional classifiers.

Classifiers vs. Decision Models

Some practitioners argue that decision models are simply zero-shot classifiers, a capability LLMs have possessed for years. One commenter noted:

"So 'decision model' is the new cool term for classifiers? You know, that thing that already existed decades ago"

Others suggest that the value lies in the reduction of the action space—moving from a Turing-complete output (like code) to a multiple-choice selection—which simplifies tool-calling and reliability.

Implementation Alternatives

While the provided implementation uses transformers and PyTorch, community members highlighted more lightweight alternatives for local or browser-based execution, such as Jeffy or Laya, which aim for response times under 200ms.

The "P-Hacking" Debate

There is technical skepticism regarding the use of temperature scaling to fit a model to a specific benchmark after the fact. Some users questioned if this approach is "allowed" or resembles "p-hacking," suggesting that the calibration should be more generalized rather than tuned to a specific dataset's results.