Designing AI-Resistant Technical Evaluations at Anthropic

Evaluating technical candidates is becoming increasingly difficult as AI capabilities advance, as traditional take-home tests that distinguished human skill levels today can be trivially solved by models tomorrow.

Anthropic has iterated through multiple versions of its performance engineering take-home test to ensure it remains robust against AI assistance, ultimately shifting from realistic, domain-specific problems to highly constrained, novel puzzles to maintain a measurable signal of human expertise.

The Challenge of AI-Resistant Evaluation

The initial take-home test, used to hire engineers for Anthropic's new scale, proved highly predictive, with one candidate scoring substantially higher than all others. However, the rapid improvement of Claude models quickly rendered the test obsolete.

Claude Opus 4: This model outperformed most human applicants within the 4-hour time limit, though it still allowed for the distinction of the strongest candidates.
Claude Opus 4.5: This model matched the performance of the top human candidates under the constraints of the take-home test, eliminating the ability to distinguish between the best human and the most capable model.

This forced Anthropic to abandon the original design and pursue increasingly unusual methods to ensure the evaluation retained a meaningful signal.

Original Take-Home Design and Goals

In November 2023, the original take-home was designed to efficiently capture the demands of performance engineering, addressing the limitations of standard interview pipelines.

Design Goals

The test was structured to be genuinely engaging and high-signal, avoiding generic problems. Key design principles included:

Longer Time Horizon: A 4-hour window (later reduced to 2 hours) better reflects the reality that performance engineering tasks rarely have short deadlines.
Realistic Environment: Candidates work without distraction, mirroring real-world coding conditions.
Time for Comprehension and Tooling: The format allows time for candidates to understand existing systems and build necessary debugging tools.
Compatibility with AI Assistance: The problem was designed to be complex enough that candidates could use AI tools (as they would on the job) while still needing to demonstrate independent skills.

The Simulated Machine

The core of the original problem was a Python simulator for a fake accelerator resembling TPUs. Candidates optimized code running on this machine, using a Perfetto trace that showed every instruction.

The simulated machine incorporated features critical to accelerator optimization, including:

Manually managed scratchpad memory.
VLIW (Very Long Instruction Word) architecture, requiring efficient instruction packing.
SIMD (Single Instruction, Multiple Data) vector operations.
Multicore processing.

The task itself was a parallel tree traversal, deliberately chosen to be non-deep learning flavored, allowing candidates to demonstrate fundamental performance engineering skills that could be learned on the job.

Iterations to Counter AI

As Claude models defeated the original test, Anthropic pursued two major redesign attempts to maintain evaluation rigor.

Attempt 1: Harder Optimization Problem (Data Transposition)

The first attempt involved a problem based on efficient data transposition on 2D TPU registers while avoiding bank conflicts. Claude Opus 4.5 quickly found a highly efficient optimization that the human designer had not anticipated by realizing it could transpose the entire computation rather than just the data.

This attempt failed because the problem relied on concepts (data transposition and bank conflicts) that were widely documented and present in Claude's training data, allowing the model to draw on a large toolbox of prior experience.

Attempt 2: Constrained Puzzles (Zachtronics Style)

To force human reasoning to win over Claude's vast experience base, the second attempt focused on highly out-of-distribution problems inspired by Zachtronics puzzle games. This new take-home used a tiny, heavily constrained instruction set, requiring candidates to optimize solutions for minimal instruction count.

Design: The puzzles were intentionally provided without visualization or debugging tools, requiring candidates to build their own tooling or use print statements.
Result: Early results were promising, with scores correlating well with the caliber of past work, and one capable colleague scoring higher than any candidate tested so far.

This new design shifts the focus from simulating realistic work to simulating novel, highly constrained work.

Performance Benchmarks and Open Challenge

Anthropic is releasing the original take-home as an open challenge, noting that human experts retain an advantage over current models at sufficiently long time horizons.

The following benchmarks, measured in clock cycles from the simulated machine, demonstrate the gap between human and AI performance:

2164 cycles: Claude Opus 4 after many hours in the test-time compute harness.
1790 cycles: Claude Opus 4.5 in a casual Claude Code session, approximately matching the best human performance in 2 hours.
1579 cycles: Claude Opus 4.5 after 2 hours in the test-time compute harness.
1548 cycles: Claude Sonnet 4.5 after many more than 2 hours of test-time compute.
1487 cycles: Claude Opus 4.5 after 11.5 hours in the harness.
1363 cycles: Claude Opus 4.5 in an improved test time compute harness after many hours.

Individuals who optimize below 1487 cycles—beating Claude's best performance at launch—are invited to submit their code and resume to Anthropic.