Anthropic Research: Language Models (Mostly) Know What They Know

Anthropic research indicates that large language models (LLMs) can evaluate the validity of their own claims and predict whether they are likely to answer a question correctly. This capability allows models to express uncertainty and identify gaps in their own knowledge, which is a critical step toward developing more honest and reliable AI systems.

Model Calibration on Multiple Choice and True/False Questions

Large language models are well-calibrated on diverse multiple-choice and true/false questions when provided in the correct format. This calibration means the model's predicted probability of being correct is a closely aligned with the actual accuracy of its answers. As model size increases, this calibration improves, showing a scaling effect where larger models are better at estimating their own accuracy.

Self-Evaluation of Open-Ended Sampling Tasks

For open-ended tasks, Anthropic researchers approached self-evaluation by asking models to first propose an answer and then evaluate the probability, denoted as "P(True)", that the answer is correct.

Key findings include:

• Performance and Scaling: There is encouraging performance, calibration, and scaling for P(True) across a diverse array of tasks.
• Sample Consideration: Self-evaluation accuracy improves when models are allowed to consider multiple samples of their own generated answers before predicting the validity of one specific answer.

Predicting Knowledge Probability (P(IK))

The researchers investigated whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question without referring to a specific proposed answer.

Findings regarding P(IK) predictions include:

• Generalization: Models perform well at predicting P(IK) and show partial generalization across different tasks.
• Calibration Challenges: Models struggle with the calibration of P(IK) when encountering entirely new tasks.
• Contextual Influence: The predicted P(IK) probabilities increase when relevant source materials are provided in the context or when hints are provided for mathematical word problems.

Implications for AI Honesty

These observations suggest that LLMs possess an internal representation of their own knowledge state. By leveraging these probabilities—P(True) and P(IK)—models can be trained to be more honest about what they do not know, reducing hallucinations and improving the reliability of the overall system.