CWE-bench v2 Release Notes

Overview of CWE-bench v2

CWE-bench v2 is a benchmark designed to test the limits of frontier AI cyber capabilities by requiring agents to identify and fix security vulnerabilities in real-world open-source repositories. The benchmark consists of 100 tasks that cover 91 different CWEs, ensuring that models must demonstrate a broad understanding of various vulnerability surfaces rather than simply recognizing familiar patterns. The evaluation process is deterministic and programmatic, requiring agents to fix security weaknesses without compromising existing software functionality.

Methodology and Quality Assurance

To ensure the benchmark remains rigorous and fair, every task in CWE-bench v2 underwent a meticulous review process. Each task was evaluated against a set of quality criteria, including:

  • Realism: Instructions are designed to reflect practical development scenarios rather than artificial evaluation settings.
  • Inference: All necessary information for a fix must be derivable from the provided instructions.
  • Verification: The grading system checks for correct behavior rather than enforcing a specific code structure.
  • Robustness: The starter code must fail, the reference fix must pass, and the original functionality must remain intact.
  • Anti-Memorization: Verifiers are designed to fail known upstream patches to prevent models from relying on memorized solutions.

Collinear AI utilized LLM judge panels to refine these programmatic checks, focusing on increasing task difficulty and fairness. The benchmark also maintains a held-out set of tasks for ongoing evaluation, which researchers can access by contacting the team.

Performance and Leaderboard

Results from 11 evaluated models indicate that while frontier models are making progress, significant room for improvement remains. Gemini 4 Argon currently leads the leaderboard with a 67.8% pass@1 rate, followed closely by Claude Opus 5.5 at 67.3%. Despite these scores, even the top-performing models fail 19% of the tasks on every attempt, highlighting the complexity of the long-tail security challenges included in the set.

Regarding cost-efficiency, the Pareto frontier is defined by three models:

  • DeepSeek V4.1 Flash: 57.0% pass@1 at $0.04 per rollout.
  • Claude Opus 5.5: Higher performance at $1.83 per rollout.
  • Gemini 4 Argon: Leading performance at $3.92 per rollout.

Running the Benchmark

CWE-bench v2 is integrated with the Harbor framework, allowing users to execute the entire benchmark directly from the Hugging Face repository without manual cloning. The tasks are self-contained, including a Dockerized environment and a deterministic verifier. Users can run the benchmark using the following command structure:

harbor run --repo "https://huggingface.co/datasets/collinear-ai/cwe-bench" -d cwe-bench@2.0 -k 4 \
  --agent <agent> --model <model> --env docker \
  --allow-agent-host <model API host>

Because the tasks are tagged as rl-environment, the same infrastructure used for evaluation can also be utilized to provide rewards for training defensive AI agents.

Sources