centerforaisafety/hle

Humanity's Last Exam

What it solves

It addresses the need for a high-difficulty, closed-ended academic benchmark to test the limits of AI models' knowledge across a wide range of subjects. It is designed to be the final benchmark of its kind, pushing the frontier of human knowledge to prevent models from simply memorizing existing test sets.

How it works

The project provides a dataset of 2,500 multi-modal questions across dozens of subjects, including mathematics, humanities, and natural sciences. These questions are multiple-choice or short-answer, allowing for automated grading. It includes a simple evaluation pipeline using the openai-python interface to generate predictions and judge results.

Who it’s for

AI researchers and model builders who need to evaluate the same level of advanced academic knowledge that human subject-matter experts possess.

Highlights

  • Multi-modal benchmark covering dozens of subjects.
  • 2,500 expert-developed questions.
  • Includes a canary string to help model builders filter the dataset from training data to prevent data contamination.
  • Automated grading for multiple-choice and short-answer responses.

Related

  • Project
    MMMU-Benchmark/MMMU

    A massive multi-discipline benchmark for evaluating multimodal AI models on college-level subject knowledge and complex reasoning across 30 different subjects.

  • Project
    rdi-berkeley/agents-last-exam

    An open evaluation framework for AI agents that measures their ability to complete long-horizon, professional real-world tasks across 55 subdomains using verifiable outcomes in OS sandboxes.

  • Dispatch
    OpenAI MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

    OpenAI introduces MLE-bench, a benchmark using 75 Kaggle competitions to measure the machine learning engineering capabilities of AI agents, with o1-preview achieving bronze medal levels in 16.9% of tasks.

  • Project
    braintrustdata/autoevals

    A library for evaluating AI model outputs using LLM-as-a-judge, RAG-specific metrics, and heuristic methods through a unified interface.

  • Project
    EleutherAI/lm-evaluation-harness

    A unified framework to evaluate generative language models on a large number of standard academic benchmarks, serving as the backend for the Open LLM Leaderboard.