OpenAI Introducing EVMbench

OpenAI and Paradigm have introduced EVMbench, a benchmark that measures the ability of AI agents to detect, patch, and exploit high-severity smart contract vulnerabilities. This tool allows researchers to track emerging cyber risks and encourages the defensive use of AI to audit and strengthen smart contracts securing billions in crypto assets.

EVMbench Dataset and Environment

EVMbench is composed of 117 curated vulnerabilities sourced from 40 audits, primarily from open code audit competitions. The benchmark also incorporates vulnerability scenarios from the security auditing process of the Tempo blockchain, a Layer 1 designed for high-throughput, low-cost stablecoin payments, to ground the benchmark in payment-oriented smart contract code.

To ensure a reproducible and objective evaluation, OpenAI developed a Rust-based harness with the following characteristics:

Isolated Execution: Exploit tasks run in a local Anvil environment rather than on live networks.
Deterministic Replay: The harness deploys contracts and replays agent transactions deterministically.
Security Constraints: The system restricts unsafe RPC methods to maintain environment integrity.
Task Validation: Environments were created by adapting existing proof-of-concept exploits and deployment scripts, with additional quality control provided by Paradigm and automated task auditing agents.

Evaluation Modes and Model Performance

EVMbench evaluates AI agents across three distinct capability modes:

Detect

Agents audit a smart contract repository and are scored based on the recall of ground-truth vulnerabilities and associated audit rewards. Performance in this mode is currently limited, as agents sometimes stop after finding a single issue rather than conducting an exhaustive audit.

Patch

Agents must modify vulnerable contracts to eliminate exploitability while preserving intended functionality. Success is verified through automated tests and exploit checks. Maintaining full functionality while removing subtle vulnerabilities remains a significant challenge for current models.

Exploit

Agents perform end-to-end fund-draining attacks against deployed contracts in a sandboxed environment. Grading is performed programmatically via transaction replay and on-chain verification.

In the exploit mode, GPT-5.3-Codex (via Codex CLI) achieved a score of 71.0%, marking a significant improvement over GPT-5, which scored 33.3% approximately six months prior.

Limitations of the Benchmark

EVMbench does not capture the full complexity of real-world smart contract security. Key limitations include:

Scope of Vulnerabilities: The vulnerabilities are drawn from Code4rena competitions; heavily deployed contracts often undergo more scrutiny and may be harder to exploit.
Grading Constraints: In 'detect' mode, the system cannot currently determine if an agent's discovery of a vulnerability missed by human auditors is a true positive or a false positive.
Structural Constraints: The exploit setting uses a clean local Anvil instance rather than a mainnet fork and replays transactions sequentially, meaning behaviors dependent on precise timing mechanics are out of scope.
Environment Limits: The benchmark currently supports only single-chain environments, necessitating the use of mock contracts in some instances.

Defensive Initiatives and Cyber Safeguards

Because cybersecurity capabilities are dual-use, OpenAI is implementing an evidence-based approach to accelerate defense while slowing misuse. These efforts include:

Technical Mitigations: The use of safety training, automated monitoring, threat intelligence enforcement pipelines, and trusted access for advanced capabilities.
Ecosystem Support: Expanding the private beta of Aardvark (a security research agent) and providing free codebase scanning for widely used open-source projects.
Financial Commitment: OpenAI is committing $10M in API credits via its Cybersecurity Grant Program to support open-source software and critical infrastructure systems engaged in good-faith security research.