Asking an LLM for a Confidence Score Is a Bad Idea: Why Calibration Needs Data, Not Prompting

Asking an LLM to output a confidence score does not produce a measurement. It produces a plausible-looking number generated the same way any other token is generated, so nothing guarantees that the number reflects the model's actual accuracy. A widely discussed blog post at rumple.github.io, "Asking LLM for confidence," argues that LLM-generated confidence scores are untrustworthy unless they are built from a documented record and checked against real outcomes — and a large Hacker News discussion added several important counterpoints about training-time calibration and structured prompting.

The post reached the Hacker News homepage and was edited on October 22, 2026, to add a link to the discussion thread. This article summarizes the original argument, then synthesizes the strongest objections and additions raised in the comments.

What a confidence score actually means

A confidence score is a prediction about a prediction. When a forecaster says "I am 70% confident that it will rain tomorrow," the 70% is not an opinion about rain. It is a summary of how strongly the available information points toward rain.

The blog post breaks that information into two parts: current observations (air pressure, humidity, wind, radar clouds) and historical base rates (how often it rained on days that looked like this one). Turning that information into a number is, in theory, a calculation rather than a judgment. If two forecasters have identical information and apply the same rule correctly, they must arrive at the same confidence. When forecasters disagree, they either have different information or one of them applied the rule badly.

The author uses a second scenario to show where information content matters. Asked about a day two weeks away, a forecaster has no observations for that day and can only report the seasonal base rate — for example, 30%. As the day approaches and observations accumulate, the confidence can move toward 0% or 100%.

How to verify that a confidence score is accurate

A single prediction cannot validate a confidence score, because the event either happens or it does not. Verification requires grouping many predictions that share the same stated confidence and comparing the stated rate with the observed rate.

If a forecaster said "70% confident" on many days and it rained on about 70% of them, the score is calibrated. If it rained on only 40% of those days, the forecaster was overconfident. If it rained on 90% of them, the forecaster was underconfident. The blog post gives this example record for one year:

They said	Days	Days it rained	Rate
10%	120	13	11%
30%	90	26	29%
70%	80	57	71%
90%	75	67	89%

At every level, the observed rain rate sits close to the stated confidence, so the forecaster is calibrated.

Calibration alone is not sufficient. A forecaster who ignores the weather entirely and always says 30% — the seasonal base rate — is also calibrated, because it rains on roughly 30% of those days. That forecaster never uses instance-level information, so the number carries no signal. The better forecaster in the table is both calibrated and informative, because most forecasts cluster near 10% or near 90%.

The blog post adds a third, easily overlooked requirement: someone must write down every prediction and what actually happened. Without that record, there is no way to know whether a confidence score is accurate.

Two conditions therefore define a trustworthy confidence score: it reflects how much information the prediction is based on, and it has been checked against what actually happened on the same kind of question.

Why an LLM cannot produce a calibrated confidence score on its own

An LLM generates text one token at a time, where each token is conditioned on the tokens that came before it. This is what "autoregressive" means. When a user asks "How confident are you?", the model does not read a value from an internal gauge. It writes the string "0.85" token by token, because that string is a plausible continuation of the conversation.

In the terms above, the number has neither required property. It does not summarize the evidence, because it is a continuation of text rather than a measurement. And it has never been calibrated on the question of whether the stated confidence matches the model's accuracy. Pretraining does score the model against outcomes, but the outcome it is scored against is the next token of text. The model is therefore calibrated to some degree on "what text comes next," not on "how often am I right when I say 0.85." It learned what confidence scores look like in text; it did not learn to make its scores match its accuracy.

Three observable symptoms of unreliable LLM confidence

The blog post lists three behaviors that follow directly from the autoregressive explanation and that anyone can reproduce:

• Asking again changes the number. The same question can receive 0.85 one time and 0.7 the next. The available information did not change, so a rule-based confidence would not change either.
• Changing the wording changes the number. Asking "Are you sure?" typically moves the score even when nothing about the problem has changed. Different phrasing is not new information.
• The text before the number decides the number. If the model has already written "This is clearly the right answer," a high score is the natural continuation. The score follows from the preceding words rather than from the evidence.

The model will always return a number, and the number will look precise. An incorrect score looks exactly like a correct one, and the only way to distinguish them is to grade many scores against an answer key — a step most people who ask an LLM for confidence never perform.

Building confidence scores without asking the LLM

The alternative the post proposes is to do what the forecaster does: construct an explicit rule that turns observable information into a confidence, and keep checking that rule against outcomes. The LLM can still perform the task (for example, writing a reply), while the confidence score comes from the rule.

The worked example is a customer-support agent that searches a help center and drafts replies, where low-confidence drafts should route to a human.

1. Choose the information. List observable signals that do not require asking the model how it feels: did the search find an article on this topic, how many articles were found, has this type of question been answered before, and do two independent generations of the same answer agree?
2. Start with the past. Before any other data exists, the best confidence is the base rate — if reviewers checked 200 past replies and 150 were correct, every new reply starts at 75%.
3. Record every reply and what happened. For each reply, store the signals from step 1, the confidence the rule assigned, and whether the reply turned out to be correct (approved by a reviewer, or accepted without the customer asking again).
4. Update the rule from the record. Group past replies by their signals and measure how often each group was correct. If replies with a matching article and two agreeing generations were correct 95% of the time, the rule should say 95% for the next such reply; if replies with no matching article were correct only 40% of the time, the rule should say 40%.
5. Keep checking. Every week or so, rebuild the calibration table: for each confidence level, how often was the reply actually correct? If the 90% replies were correct only 70% of the time, the rule is overconfident and the new records show how to fix it.

Such a rule has the properties an LLM's number lacks: identical information always yields identical confidence, the score depends on the information rather than the phrasing, and calibration is measured on the organization's own question type because it was built from the organization's own records. The rule also improves with use, since every reply adds to the record. The rule is only as good as the record, though — if nobody checks whether replies were correct, there is nothing to learn from and the system is back to an unmeasured number.

The main counterargument: calibration can be trained into a model

Several Hacker News commenters argued that the post's claim that LLMs "were never scored against correctness" is an assumption about the training process that is no longer accurate. Commenter rahimnathwani pointed to Sebastian Raschka's writing on calibration and to reinforcement learning with a calibration reward:

"RLCR adds a Brier penalty to the reward, so the model's performance isn't just measured on % correctness. The Brier penalty, (q-c)^2, punishes a forecaster who ignores instance-level information."

The commenter illustrates the mechanism with two strategies for a model that is correct 60% of the time. Always answering 0.6 is calibrated but carries no signal, because the expected penalty is 0.6 * (0.6-1)^2 + 0.4 * (0.6-0)^2 = 0.24. A model that can distinguish easy cases (correct 90% of the time) from hard cases (correct 30% of the time) and answers 0.9 and 0.3 respectively has an expected penalty of (0.09 + 0.21) / 2 = 0.15. Both scoring strategies are equally calibrated overall, but the second scores better, which is why a Brier-style penalty pushes training toward instance-specific confidence rather than a constant base rate.

Other commenters made related points:

• Commenter m101 asked whether Jev-style training, which predicts a distribution over next tokens rather than a single next token, is designed to address exactly this.
• Commenter yaoke259 argued that if producing a correct confidence were an RL training target on a real or simulated environment — weather outcomes, game outcomes — models would likely develop a strong intuition for confidence, similar to how Jev and decision models are trained.
• Commenter Ozzie_osman reported that fine-tuning confidence scores for a specific domain into an existing model works: "you basically help the model understand what types of inputs it tends to get right and which ones it tends to get wrong."
• Commenter qznc disputed the framing entirely, noting that the article itself does not show data proving the approach is bad and citing forecasting performance on Metaculus as evidence that LLMs can be good at this.
• Commenter jrflo made the same demand for evidence: "I was hoping the writer would follow the interesting framing of the quality of an LLM's confidence score with some data on how good it is... If someone collected the data on something measurable, I would be willing to bet the LLM's confidence number isn't too far off."

Practical mitigations suggested by practitioners

Commenters converged on several concrete techniques that sit between "ask the LLM for a number" and "build a full statistical rule."

Decompose the rating instead of using a 0-100 scale. Commenter orbital-decay argued that a wide numeric scale is the worst way to elicit a judgment from either a model or a person, and proposed splitting the problem into many dimensions, giving each dimension a small meaningful enum or a coarse scale, then applying weighted scoring. Youtube reportedly dropped its 1-5 rating scale for the same reason.

Use a rubric or measurable criteria. Commenter cityofdelusion described using grading scripts for quantities that can be measured directly and a grading matrix with explicit A-through-F criteria for less well-defined judgments. The commenter's key point is that if a confidence cannot be expressed mathematically or as a rubric, the implementer does not know how to measure the target either, and a human would be "going off gut feeling and vibing a number." Commenter YmFzZTY0ZGFkZHk asked whether coarser descriptive confidence (for example, major versus minor) is a different flavor of the same illusion.

Use log probabilities or repeated sampling instead of self-report. Commenter dummydummy1234 suggested structured output combined with log probabilities over the allowed output classes. Commenter jvanderbot raised the same idea, asking whether restricting the vocabulary to input labels turns next-token likelihood into a usable confidence. Commenter bob1029 proposed issuing many identical requests and measuring the external distribution of tool choices or answers, then using that distribution to drive subsequent inference steps.

Fall back on classical calibrated models. Commenter PaulHoule pointed to scikit-learn's calibration module and described a pipeline of a pretrained encoder, pooling, and a probability-calibrated classifier trained on domain data. The same commenter stressed that calibration depends on the data distribution: a model trained on a different prior than the one it is applied to will be poorly calibrated. Commenter ricardobeat noted that the post's five-step procedure is essentially supervised learning — data, labeling, weight updates, and calibration — and that decision models are trained specifically for this purpose.

Limitations and unresolved questions

Commenter visarga stated the underlying constraint most directly: "you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction." The same commenter added that no model provider can pre-calibrate a model for tasks it has never seen.

Commenter hexapus framed the debate in the broadest terms, describing LLMs as next-token predictors that "can't reason, but they can predict what reasoning looks like," and arguing that when they fail, it is worth being honest about why.

Commenter zug_zug reported that in an experiment asking LLMs to rate the odds of an outcome explaining an observation, the models were systematically biased in ways similar to humans, concluding that Bayesian statistical fundamentals are not reliable in LLMs at this point. Commenter madrox, drawing on a statistics background, observed that applied statistics training spent roughly 10% of its time building models and 90% interpreting their output, and questioned whether machine learning curricula preserve that balance.

The net practical conclusion from the post and the discussion is consistent: a confidence score is only meaningful if it is tied to a written record of outcomes for the specific task being scored. Whether that score is produced by an explicit statistical rule, a calibrated classifier, a fine-tuned model, or a model trained with a proper scoring rule is an engineering choice — but in every one of those cases, the calibration evidence comes from data, not from asking the model how sure it feels.