TruthfulQA: Measuring how models mimic human falsehoods

OpenAI has introduced TruthfulQA, a benchmark designed to evaluate whether large language models (LLMs) generate truthful answers or simply mimic common human misconceptions found in their training data. This research highlights a critical gap in model truthfulness, revealing that larger models are often less truthful because they more effectively imitate the false beliefs prevalent in human-generated text.

TruthfulQA Benchmark Design

TruthfulQA consists of 817 questions spanning 38 distinct categories, including high-stakes domains such as health, law, finance, and politics. The questions are specifically crafted to target areas where humans frequently hold false beliefs or misconceptions. To achieve a high score, a model must avoid generating answers that are common but incorrect, requiring it to diverge from the simply imitating the patterns of human text it was trained on.

Model Performance and Human Baseline

OpenAI tested several models, including GPT-3, GPT-Neo/J, GPT-2, and a T5-based model. The results demonstrate a significant performance gap between AI models and humans:

• Human Performance: Humans achieved a truthfulness rate of 94%.
• Model Performance: The best-performing model was truthful on only 58% of questions.

Models frequently generated false answers that mirrored popular misconceptions, which the researchers note have the potential to deceive human users.

The Relationship Between Model Size and Truthfulness

Unlike many natural language processing (NLP) tasks where performance typically improves as model size increases, TruthfulQA revealed an inverse relationship. The largest models were generally the least truthful.

This trend occurs because larger models are more capable of imitating the training distribution—which includes human falsehoods—more accurately. Consequently, scaling up model size alone is not an effective strategy for improving truthfulness.

Implications for Model Training

Because truthfulness is not an emergent property of scale, OpenAI suggests that improving model accuracy requires a shift in training objectives. The researchers argue that fine-tuning models using objectives other than the simple imitation of web-based text is necessary to reduce the likelihood of models mimicking human misconceptions.