OpenAI Research: Teaching Models to Express Their Uncertainty in Words
OpenAI has demonstrated that a GPT-3 model can learn to express its own uncertainty about an answer using natural language, producing "verbalized probabilities" that are well-calibrated. This capability allows the model to communicate its confidence levels (e.g., "90% confidence" or "high confidence") directly in text, providing a human-readable measure of reliability without needing to access the model's internal logits.
Verbalized Probability and Calibration
GPT-3 can generate both a factual answer and a corresponding confidence level in natural language. These verbalized confidence levels map to actual probabilities that are well-calibrated, meaning the model's expressed confidence accurately reflects the likelihood of the answer being correct. This marks the first time a model has been shown to express calibrated uncertainty about its own answers using natural language.
Performance Under Distribution Shift
The model maintains moderate calibration even when subjected to distribution shift. The research indicates that the model is sensitive to the actual uncertainty of its own answers rather than simply imitating human examples from its training data. This suggests that the model is leveraging internal knowledge to assess its own accuracy.
Comparison with Model Logits
To evaluate these findings, OpenAI introduced the CalibratedMath suite of tasks. The researchers compared "verbalized probability" (uncertainty expressed in words) against uncertainty extracted from model logits. The results showed that both methods are capable of generalizing calibration under distribution shift.
The Role of Pre-trained Latent Representations
Evidence suggests that GPT-3's ability to generalize calibration is linked to its pre-trained latent representations. These representations correlate with epistemic uncertainty—the uncertainty resulting from a lack of knowledge—over the model's answers, allowing the model to translate internal uncertainty into natural language expressions.