ukanwat/aaabench

What it solves

AAABench is a benchmark that tests whether an AI coding agent can build a complete, believable open-world video game entirely on its own. Most AI benchmarks ask a model to solve a narrow task or answer a question. This one gives an agent a real game engine, a demanding brief, and hours of unsupervised time, then measures whether the agent can design a city, build it, and notice and fix its own mistakes without any human help.

How it works

The harness runs a coding agent (Claude by default, but any headless CLI works) against a live Unreal Engine 5 editor. The agent controls the editor through the Model Context Protocol (MCP), using Epic's built-in MCP server plus the VibeUE plugin, which adds 31 service toolsets and 85 skills for spawning actors, editing Blueprints, capturing viewport images, authoring materials, and more.

The agent is handed a single demanding prompt (PROMPT.md) that asks it to build an open-world game. It has access to a handbook of production knowledge, 21 skill packs, and tools that let it see its own work through viewport capture and screenshots. The harness scripts boot the editor, hand over the prompt, resume the session if it stops early, relaunch the editor if it crashes, and supervise long unattended runs. The key rule is that the human provides conditions, resources, and the demand — never diagnosis, never a fix, never an answer. Whether the model notices its own mistakes is the capability being measured.

Who it's for

Researchers and engineers who want to evaluate frontier AI models on a long-horizon, real-world task that cannot be faked by pattern-matching. It is also useful for anyone studying agent capabilities like causal reasoning, self-verification, systems thinking, and sustained autonomous execution. Running it requires a macOS machine with Apple Silicon, Xcode, the Metal toolchain, Unreal Engine, and an authenticated agent CLI.

Highlights

• Tests capabilities that standard benchmarks miss: real-world understanding, causal reasoning, long-horizon execution, writing quality, self-verification, systems thinking, and working under a hostile surface (a crashing editor, silent tool failures).
• The agent gets full control of a real Unreal Engine 5 editor over MCP, with 31 service toolsets and 85 skills via VibeUE.
• The agent has eyes: viewport capture and screenshots let it inspect its own work and fix what looks fake.
• The harness is agent-agnostic — you can test Claude, Codex, Gemini, or any custom headless CLI.
• Includes unattended supervision scripts with exponential backoff, health checks, and safe restart logic.
• The bar for passing is high: a place that survives a stranger looking at it, and a game that opens like a game.