Google DeepMind Measuring Progress Toward AGI: A Cognitive Framework

Google DeepMind has introduced a cognitive framework and taxonomy to provide a scientific foundation for measuring progress toward Artificial General Intelligence (AGI). This framework aims to address the lack of empirical tools for evaluating general intelligence in AI systems by leveraging research from psychology, neuroscience, and cognitive science.

A Cognitive Taxonomy for General Intelligence

The framework identifies 10 key cognitive abilities essential for general intelligence. These abilities are categorized as follows:

  1. Perception: Processing and extracting sensory information from the environment.
  2. Generation: Producing outputs including actions, speech, and text.
  3. Attention: The ability to focus cognitive resources on relevant information.
  4. Learning: Acquiring new knowledge via instruction and experience.
  5. Memory: The retrieval and storage of information over time.
  6. Reasoning: Using logical inference to draw valid conclusions.
  7. Metacognition: The monitoring and knowledge of one's own cognitive processes.
  8. Executive functions: Cognitive flexibility, inhibition, and planning.
  9. Problem solving: Finding effective solutions to specific domain problems.
  10. Social cognition: Interpreting social information and responding appropriately in social situations.

Three-Stage Evaluation Protocol

To benchmark AI performance against human capabilities, Google DeepMind proposes a three-stage evaluation protocol:

  • Stage 1: Evaluate AI systems using a broad suite of cognitive tasks for each ability, utilizing held-out test sets to prevent data contamination.
  • Stage 2: Collect human baselines for the same tasks using a demographically representative sample of adults.
  • Stage 3: Map the AI system's performance relative to the distribution of human performance for each specific ability.

Implementation via Kaggle Hackathon

Google DeepMind is partnering with Kaggle to translate this theoretical framework into practice through the "Measuring progress toward AGI: Cognitive abilities" hackathon. The competition invites the research community to design evaluations for the five cognitive abilities where the evaluation gap is currently the largest: learning, metacognition, attention, executive functions, and social cognition.

Participants utilize Kaggle's Community Benchmarks platform to test evaluations against frontier models. The hackathon features a $200,000 total prize pool, with $10,000 awards for the top two submissions in each of the five tracks and $25,000 grand prizes for the four best overall submissions. Submissions are open from March 17 through April 16, with results announced on June 1.

Sources