Agents' Last Exam (ALE)

What it is

Agents' Last Exam (ALE) is an open‑source benchmark framework for evaluating AI agents that can operate like a real‑world worker. It provides:

55 concrete professional tasks (e.g., data‑entry, video editing, software build) grouped into 13 industry clusters, all derived from the U.S. O*NET occupational taxonomy.
Sandboxed execution environments (Linux, Windows, GPU, Docker, QEMU/KVM, or cloud VMs) that faithfully reproduce the software a human would use.
A run‑time orchestrator (ale_run) that provisions the sandbox, hands the task description to the agent, lets the agent work autonomously, then grades the result against a hidden reference.
Uniform logging of every tool call, screenshot, file change and final score, making runs reproducible and auditable.

The goal is to measure whether an agent can actually get the job done on economically valuable, long‑horizon workflows, rather than just answering a static question.

Why it matters

Real‑world relevance – tasks are taken from actual occupations and validated by industry experts, so scores reflect economic productivity.
Long‑horizon, multimodal – agents must handle both command‑line and GUI actions over many steps, something most existing benchmarks ignore.
Standardised comparison – by providing a common sandbox, hidden reference, and scoring schema, ALE lets researchers compare very different agent architectures (Claude Code, Codex, Open‑claw, etc.) on the same footing.
Open data & code – the framework, task definitions, and reference implementations are all under permissive licenses (Apache‑2.0 for code, CC‑BY‑4.0 for data).

How to get started

Clone the repo and install host dependencies (Ubuntu/Debian supported).

git clone https://github.com/rdi-berkeley/agents-last-exam.git
cd agents-last-exam
bash env/install_host_deps.sh
uv sync --all-packages --locked
bash env/verify_host_env.sh

Configure an experiment – copy the example YAML, fill in your cloud credentials (or choose a local provider), and pick a task list.
```
cp example_exp.yaml my_experiment.yaml
# edit my_experiment.yaml → set provider, tasks, output_path, etc.
```

Run a dry‑run to check everything, then launch the full benchmark.

uv run python -m ale_run run my_experiment.yaml --dry-run
uv run python -m ale_run run my_experiment.yaml

Inspect results – ALE stores a unified trajectory (step‑by‑step actions, tool calls, screenshots) together with the final numeric score and raw logs. You can replay or audit any run.

Provider options (choose one)

Provider	Typical use	Docs
Google Cloud VMs	Scalable batch runs on Ubuntu/Windows + GPU images	`docs/quickstart.md`
AWS EC2 + S3	Existing CPU‑profile workloads	AWS guide
Alibaba Cloud ECS + OSS	Same as AWS for Ali‑Yun users	Alibaba guide
QEMU/KVM on a Linux host	CPU‑only VMs without cloud costs	Local guide
Docker (Ubuntu subset)	Light‑weight local testing	Docker guide
Existing sandbox	Debug against a machine you already control	Static guide

Who is behind it

UC Berkeley RDI (Research, Development & Innovation) in partnership with the RDI Foundation.
The project is co‑led by a large team of researchers (see the extensive author list in the citation) and built with input from hundreds of industry experts.
A public website, mailing list, Discord community, and regular releases keep the ecosystem active.

Where to find more

Website & docs: https://agents-last-exam.org/ (full design docs, provider guides, task gallery)
Paper (arXiv): https://arxiv.org/abs/2606.05405
Leaderboard & demo runs: https://agenthle.org/leaderboard and https://agenthle.org/demo
Data & model hub: https://huggingface.co/agents-last-exam (task data, VM images)
License: Apache‑2.0 for code, CC‑BY‑4.0 for benchmark data.

TL;DR

Agents' Last Exam is a comprehensive, open‑source benchmark that lets you test any AI agent that can control a computer (CLI + GUI) on real, economically‑valuable tasks. It handles sandbox provisioning, hidden‑reference grading, and produces reproducible logs, making it a go‑to platform for research on generalist, tool‑using agents.

rdi-berkeley/agents-last-exam