Google DeepMind FACTS Benchmark Suite Release

Google DeepMind has released the FACTS Benchmark Suite in partnership with Kaggle to systematically measure and improve the factuality of large language models (LLMs). This suite provides a standardized framework to identify specific failure modes in model accuracy across four distinct information-retrieval scenarios: parametric knowledge, web search synthesis, multimodal interpretation, and grounded context.

The FACTS Benchmark Suite Composition

The FACTS Benchmark Suite consists of 3,513 curated examples across four specialized benchmarks. To maintain evaluation integrity, Google DeepMind uses a split of public sets for transparency and private held-out sets managed by Kaggle to prevent data contamination and calculate the final FACTS Score (the average accuracy across all four benchmarks).

FACTS Parametric Benchmark

The Parametric benchmark measures a model's ability to access its internal knowledge without external tools. It focuses on "trivia style" factoid questions that can be answered using Wikipedia, testing the model's internal world knowledge.

  • Dataset Size: 1,052 public items and 1,052 private items.
  • Example Task: Answering niche factual questions, such as identifying the harmonica player on "The Rockford Files" theme song.

FACTS Search Benchmark

The Search benchmark evaluates how effectively a model uses a web search tool to retrieve and synthesize information. It is specifically designed to be challenging by requiring the sequential retrieval of multiple facts to answer a single complex query.

  • Dataset Size: 890 public items and 994 private items.
  • Tooling: All models use the same web search tool to ensure that results reflect model capability rather than differences in retrieval infrastructure.
  • Example Task: Calculating the sum of birth years for three specific Olympic boxers identified through multiple search steps.

FACTS Multimodal Benchmark

The Multimodal benchmark tests the integration of visual grounding and parametric knowledge, requiring models to generate factually accurate text based on image inputs.

  • Dataset Size: 711 public items and 811 private items.
  • Core Requirement: The model must accurately interpret visual information and connect it to its internal knowledge to provide a complete and correct response.
  • Example Task: Identifying the genus of an animal provided in an image.

FACTS Grounding Benchmark v2

An update to the original FACTS Grounding benchmark, version 2 tests a model's ability to provide answers that are strictly grounded in the context provided within a specific prompt.

Performance Results and Model Benchmarking

Google DeepMind evaluated 15 leading LLMs using the suite. Gemini 3 Pro achieved the highest overall performance with a FACTS Score of 68.8%.

Gemini 3 Pro Improvements

Gemini 3 Pro demonstrated significant factuality gains over Gemini 2.5 Pro, particularly in tool-use and internal knowledge:

  • FACTS Search: Error rate reduced by 55%.
  • FACTS Parametric: Error rate reduced by 35%.
  • SimpleQA Verified: Accuracy increased from 54.5% (Gemini 2.5 Pro) to 72.1% (Gemini 3 Pro).

Industry-Wide Challenges

Despite the lead of Gemini 3 Pro, the Multimodal benchmark yielded the lowest scores across all evaluated models. No model achieved an overall accuracy above 70% in this category, indicating that multimodal factuality remains a significant area for future research and development.

Sources