Designing AI-Resistant Technical Evaluations

Anthropic has redesigned its technical hiring evaluations multiple times to counter the increasing capabilities of AI models. As Claude models evolved, they began to match or exceed the performance of top human candidates on traditional optimization tasks, forcing the company to move from realistic simulations of real-world work to highly unconventional, "out-of-distribution" problems to maintain a valid signal for hiring.

The Evolution of the Performance Engineering Take-Home

Anthropic's performance engineering team utilizes a take-home test where candidates optimize code for a simulated accelerator. The goal is to identify engineers capable of high-level performance optimization without requiring specific domain knowledge of deep learning.

Version 1: The Simulated Machine

Launched in November 2023, the original test used a Python simulator for a fake accelerator with characteristics similar to TPUs, featuring:

  • Manually managed scratchpad memory: Requiring explicit memory management.
  • VLIW (Very Long Instruction Word): Requiring efficient instruction packing across parallel execution units.
  • SIMD (Single Instruction, Multiple Data): Vector operations on many elements.
  • Multicore distribution: Distributing work across multiple cores.

The task involved a parallel tree traversal. This format was chosen over live interviews to allow for a longer time horizon (initially 4 hours), a realistic environment, and time for candidates to build their own debugging tools.

Version 2: Increasing Depth and Constraints

By May 2025, Claude 3.7 Sonnet could solve over 50% of the test, and a pre-release version of Claude Opus 4 outperformed most human applicants within the 4-hour limit.

To counter this, Anthropic implemented Version 2:

  • Increased Depth: The starting point was shifted to where Claude Opus 4 began to struggle.
  • Simplified Scope: Multicore parallelism was removed as Claude had already solved it.
  • Tighter Constraints: The time limit was reduced from 4 hours to 2 hours to increase pipeline efficiency.
  • Focus: The test shifted toward clever optimization insights rather than code volume or debugging.

Version 3: Out-of-Distribution Puzzles

Claude Opus 4.5 eventually matched the best human performance within the 2-hour limit, even identifying clever tricks to bypass memory bandwidth bottlenecks. Attempts to create a harder but realistic problem (such as 2D TPU register data transposition) failed because the model could draw on vast training data regarding bank conflicts and transposition.

To find a signal that human reasoning could still win over AI experience, Anthropic moved to a "weirder" approach inspired by Zachtronics games. The current evaluation consists of:

  • Highly Constrained Instruction Sets: Puzzles using tiny, unconventional instruction sets that force non-standard programming methods.
  • Minimal Tooling: No visualization or debugging tools are provided; candidates must decide whether to build their own or use AI to generate them.
  • Novelty: By using problems that are sufficiently out-of-distribution, the test simulates "novel work" rather than "real work," as the latter often resembles patterns already present in AI training data.

Benchmarking AI vs. Human Performance

Anthropic has released the original take-home as an open challenge. While AI can match humans in short windows, human experts still retain an advantage over long time horizons.

The following clock cycle benchmarks (lower is better) illustrate the progression of Claude's capabilities on the original simulator:

Model/Condition Clock Cycles
Claude Opus 4 (many hours in harness) 2164
Claude Opus 4.5 (casual session) 1790
Claude Opus 4.5 (2 hours in harness) 1579
Claude Sonnet 4.5 (many hours in harness) 1548
Claude Opus 4.5 (11.5 hours in harness) 1487
Claude Opus 4.5 (improved harness, many hours) 1363

Implications for Technical Hiring

The shift in evaluation design highlights a growing challenge for technical recruiters: as AI models gain the ability to solve common and niche technical problems, the "dominant strategy" for candidates may shift toward steering AI rather than demonstrating first-principles reasoning.

Anthropic's experience suggests that for a test to remain AI-resistant, it must either be extremely long-horizon or based on problems so unconventional that the model cannot rely on its training data, effectively trading realism for the ability to distinguish top-tier human talent.

Sources

Related

  • Dispatch
    Anthropic Recursive Self-Improvement Progress Report

    Anthropic reports that AI agents are now authoring over 80% of their production code and demonstrating superhuman performance in experimental optimization, signaling a trajectory toward recursive self-improvement.

  • Dispatch
    Optimizing AI Engineering: A Post-Mortem Approach to Claude Code

    Following Anthropic's recent postmortem, this article explores a refined approach to using Claude Code, shifting from a focus on minimizing token cost to optimizing for output quality and efficiency, treating AI as a valuable engineering resource. It details strategies across model selection, configuration, prompting, and agent use, while also considering the overhead of managing AI tools.

  • Dispatch
    Claude 4 release notes / what's new

    Anthropic has released Claude Opus 4 and Claude Sonnet 4, introducing state-of-the-art coding capabilities, extended thinking with tool use, and the general availability of Claude Code.

  • Dispatch
    Claude 3.5 Sonnet release notes / what's new

    Anthropic has released Claude 3.5 Sonnet, a model that outperforms Claude 3 Opus and competitors on reasoning and coding benchmarks while operating at twice the speed of Opus.

  • Dispatch
    Claude Opus 4.8 Release Notes

    Anthropic has released Claude Opus 4.8, an upgraded model featuring improved reasoning, higher honesty, and new agentic capabilities alongside the introduction of dynamic workflows and effort controls.