OpenAI SimpleQA benchmark release

TL;DR

OpenAI released SimpleQA, an open‑source factuality benchmark of 4,326 short, fact‑seeking questions designed to evaluate and compare the factual accuracy and calibration of frontier language models.

What is SimpleQA?

SimpleQA is a factuality benchmark that focuses on short queries with a single, verifiable answer. By limiting scope to concise fact‑seeking questions, the benchmark makes it tractable to measure whether a model’s response is correct, incorrect, or not attempted. The dataset contains 4,326 questions covering a wide range of topics—including science, technology, TV shows, and video games—ensuring both diversity and challenge for state‑of‑the‑art models.

Dataset Construction and Quality Assurance

  • High correctness – Two independent AI trainers generated reference answers supported by sources; a third trainer validated a random 1,000‑question sample, achieving 94.4% agreement. Manual inspection suggests an inherent error rate of roughly 3%.
  • Diversity – Topics span multiple domains, illustrated by a pie‑chart distribution in the original post.
  • Frontier difficulty – Benchmarks such as TriviaQA (2017) and Natural Questions (2019) are saturated; SimpleQA remains challenging, with GPT‑4o scoring below 40%.
  • Researcher UX – Questions and answers are concise, enabling fast evaluation via the OpenAI API or other model APIs. The modest size yields low variance across runs.

Grading Methodology

SimpleQA uses a prompted ChatGPT classifier that receives both the model’s predicted answer and the ground‑truth answer. The classifier assigns one of three grades:

Grade Definition Example (question: Which Dutch player scored an open‑play goal in the 2022 Netherlands vs Argentina World Cup match?; answer: Wout Weghorst)
Correct The predicted answer fully contains the ground‑truth answer without contradiction. "Wout Weghorst"; "Wout Weghorst scored at 83’ and 90+11’"
Incorrect The predicted answer contradicts the ground‑truth in any way, even if hedged. "Virgil van Dijk"; "Virgil van Dijk and Wout Weghorst"; "Wout Weghorst and I think van Dijk scored, but I am not totally sure"
Not attempted The answer does not provide the target fact and contains no contradiction. "I don’t know the answer"; "Please browse the internet yourself"

A model should maximize correct answers while minimizing incorrect ones; “not attempted” responses are preferable to hallucinations.

Model Comparisons on SimpleQA

Using the above grading scheme, OpenAI evaluated several models without browsing:

  • GPT‑4o‑mini and o1‑mini answered fewer questions correctly than GPT‑4o and o1‑preview, reflecting lower world knowledge in smaller models.
  • o1‑mini and o1‑preview opted to “not attempt” more often, suggesting they can recognize uncertainty rather than hallucinate.

These patterns align with expectations that larger or more reasoning‑intensive models produce higher factual accuracy.

Calibration Measurements

SimpleQA also serves to assess calibration—whether a model’s confidence matches its actual accuracy.

Stated‑confidence calibration

Models were prompted to give a confidence percentage with each answer. Plotting confidence versus observed accuracy showed a positive correlation, confirming models have some notion of confidence. However, all models fell below the ideal y = x line, indicating systematic over‑confidence. Notably, o1‑preview calibrated better than o1‑mini, and GPT‑4o calibrated better than GPT‑4o‑mini, echoing prior findings that larger models tend to be better calibrated.

Frequency‑based calibration

Repeating each question 100 times allowed measurement of answer consistency. Grouping identical strings revealed that higher response frequency corresponded to higher accuracy across all models. o1‑preview exhibited the strongest alignment between frequency and accuracy, again outperforming its smaller counterpart and GPT‑4o‑mini.

Limitations and Future Work

SimpleQA’s scope is intentionally narrow: it evaluates factuality only for short, single‑answer queries. Whether performance on SimpleQA predicts factual reliability in longer, multi‑fact generations remains an open question. Expanding benchmarks to cover richer answer formats will be necessary to fully understand model truthfulness.

Impact and Call to Action

By open‑sourcing SimpleQA (see the GitHub repo at github.com/openai/simple-evals), OpenAI invites the research community to benchmark, improve, and provide feedback on factuality evaluation. Wider adoption could accelerate development of more trustworthy language models and guide calibration‑aware training methods.

Authors: Jason Wei, Karina Nguyen, Hyung Won Chung, Joy Jiao, Spencer Papay, Mia Glaese, John Schulman, Liam Fedus

Sources