centerforaisafety/hle

Humanity's Last Exam

What it solves

It addresses the need for a high-difficulty, closed-ended academic benchmark to test the limits of AI models' knowledge across a wide range of subjects. It is designed to be the final benchmark of its kind, pushing the frontier of human knowledge to prevent models from simply memorizing existing test sets.

How it works

The project provides a dataset of 2,500 multi-modal questions across dozens of subjects, including mathematics, humanities, and natural sciences. These questions are multiple-choice or short-answer, allowing for automated grading. It includes a simple evaluation pipeline using the openai-python interface to generate predictions and judge results.

Who it’s for

AI researchers and model builders who need to evaluate the same level of advanced academic knowledge that human subject-matter experts possess.

Highlights

  • Multi-modal benchmark covering dozens of subjects.
  • 2,500 expert-developed questions.
  • Includes a canary string to help model builders filter the dataset from training data to prevent data contamination.
  • Automated grading for multiple-choice and short-answer responses.

Related

  • Project
    MMMU-Benchmark/MMMU

    A massive multi-discipline benchmark for evaluating multimodal AI models on college-level subject knowledge and complex reasoning across 30 different subjects.

  • Project
    rdi-berkeley/agents-last-exam

    Agents' Last Exam (ALE) is an open‑source benchmark framework that evaluates AI agents on 55 real‑world professional tasks across 13 industry clusters. It provides sandboxed Linux/Windows environments (cloud VMs, Docker, QEMU/KVM, etc.), a run orchestrator (`ale_run`) that hands a task description to the agent, lets it work autonomously, then grades the output against a hidden reference. Scores are reproducible, and logs capture every tool call and screenshot. The project is led by UC Berkeley RDI, released under Apache‑2.0 (code) and CC‑BY‑4.0 (data), with a website, leaderboard, and community resources.

  • Dispatch
    OpenAI MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

    OpenAI introduces MLE-bench, a benchmark using 75 Kaggle competitions to measure the machine learning engineering capabilities of AI agents, with o1-preview achieving bronze medal levels in 16.9% of tasks.

  • Project
    braintrustdata/autoevals

    Autoevals is a library that provides a unified interface for evaluating AI model outputs using a variety of LLM-based, statistical, and heuristic methods. It simplifies the process of testing, debugging, and normalizing performance metrics for AI applications.

  • Project
    EleutherAI/lm-evaluation-harness

    A unified framework to evaluate generative language models on a large number of standard academic benchmarks, serving as the backend for the Open LLM Leaderboard.