Claude 4 Cyber Evaluations

Anthropic and Pattern Labs have conducted rigorous cyber offense evaluations of Claude Opus 4 and Claude Sonnet 4. The models demonstrate a marked improvement in vulnerability identification and the execution of complex, multi-step attack chains, signaling a progress toward human-level cyber offense capabilities in specific scenarios.

Advanced Offensive Security Capabilities

Claude 4 models exhibit a significant increase in flexibility and adaptability. Unlike previous models that often persisted with failed, unchanging approaches, Claude Opus 4 demonstrates an improved ability to think flexibly and adapt its strategy when facing challenges.

Key technical improvements include:

• Vulnerability Identification: The models show significant improvement in identifying security vulnerabilities.
• Attack Chain Execution: Claude 4 consistently succeeds in executing complex multi-step attack chains in scenarios where previous models failed.
• Evaluation Scope: The testing ranged from standalone capture the flag (CTF) challenges to complex network environment simulations.

Critical Limitations in Long-Horizon Planning

Despite the advances in offensive capabilities, Claude 4 models still face challenges with long-term goal maintenance. Specifically, the models struggle to maintain coherent, long-horizon plans and goals when presented with unexpected obstacles.

Safety Implications and Ongoing Work

These evaluations, conducted in partnership with Pattern Labs, inform Anthropic's ongoing safety work. The results highlight both the technical progress in the models' ability to perform offensive security tasks and the remaining gaps in their own strategic planning capabilities, which is essential for understanding the risk profile of frontier AI models as they approach human-level cyber capabilities.