rdi-berkeley/agents-last-exam

What it solves

Agents' Last Exam (ALE) addresses the lack of broad-coverage benchmarks for AI agents that can measure performance on long-horizon, economically valuable, real-world professional tasks. It moves beyond simple prompts to evaluate whether an agent can actually complete complex work across diverse industry domains with verifiable outcomes.

How it works

ALE provides an open evaluation framework that pairs an agent harness (the system being tested) with a realistic environment (a Windows or Linux sandbox) and a specific professional task.

Key components include:

• CUA-agents: The framework evaluates agents that combine both Command Line Interface (CLI) and Graphical User Interface (GUI) capabilities. It uses a cross-OS CUA MCP bridge to expose desktop actions like clicking and typing as tools for the agent.
• Sandboxes: Tasks are run on real machines (VMs or containers) that reproduce production contexts, including professional software and data.
• Deterministic Grading: Each task has a hidden reference used by a grader to score the agent's output on a scale of 0 to 1 after the agent has finished its work.
• Execution Loop: The system provisions a sandbox, stages inputs, runs the agent to completion, stages the hidden reference, and then grades the result.

Who it’s for

ALE is designed for developers of frontier agent systems and researchers who need to benchmark their agents' ability to perform multi-step, professional workflows in real-world operating system environments.

Highlights

• Broad Coverage: Covers 55 subdomains across 13 industry clusters based on the U.S. federal occupational taxonomy.
• Verifiable Outcomes: Uses hidden references and deterministic graders to prevent leakage and ensure objective scoring.
• Multi-Platform Support: Supports sandboxes via Google Cloud, AWS, Alibaba Cloud, QEMU/KVM, and Docker.
• Full Auditing: Records uniform trajectories, raw logs, and artifacts for every run to allow end-to-end replay.