Tencent/VulnGym

What it solves

VulnGym addresses the limitations of existing security benchmarks that rely on isolated code snippets or simple binary labels. It provides a realistic, project-level environment to evaluate how well AI agents can identify and trace vulnerabilities within complex, multi-file software projects.

How it works

The benchmark consists of a dataset of real-world vulnerabilities mapped to specific commits in open-source projects. Each entry includes a verifiable "chain" of evidence: a reachable entry point, a critical operation where the defect occurs, and a cross-module reasoning trace. This allows users to test their vulnerability-hunting agents by checking if they can correctly locate these specific points in the code.

Who it’s for

This tool is designed for security researchers and developers building AI agents for automated vulnerability detection. It is particularly useful for those who need to evaluate an agent's ability to reason about business-logic flaws—such as authorization bypasses—which require understanding how different parts of a large application interact.

Highlights

• Real-world scope: Evaluates agents on complete, multi-module engineering projects rather than isolated functions.
• Comprehensive coverage: Includes both traditional security flaws (like injection) and complex business-logic vulnerabilities.
• Fully verified: Every entry in the current dataset has been human-audited to ensure accuracy and reproducibility.
• Explainable results: Provides a clear path of evidence for each vulnerability, making it easier to understand why an agent succeeded or failed.