sunblaze-ucb/cybergym

What it solves

CyberGym provides a standardized, large-scale framework for evaluating how well AI agents can handle real-world cybersecurity tasks, specifically vulnerability analysis. It addresses the need for a rigorous way to test if AI agents can actually find and exploit vulnerabilities in software at scale.

How it works

The framework operates as a submission server and task generation system. It provides a large dataset of real-world vulnerability tasks (including binary-only and full compilation environments). Users generate a specific task for an AI agent, and the agent attempts to create a Proof-of-Concept (PoC) exploit. The agent submits this PoC to a local server that verifies if the exploit successfully triggers the vulnerability. To ensure security and fairness, CyberGym includes a built-in firewall proxy that restricts agent containers' internet access to a predefined allowlist of domains.

Who it’s for

Researchers and developers building AI agents for cybersecurity, specifically those focusing on automated vulnerability research and exploit generation.

Highlights

• Real-World Scale: Includes a massive dataset of tasks sourced from real vulnerabilities.
• Secure Execution: Features a dedicated firewall and isolated Docker networks to prevent agents from accessing the public internet.
• Flexible Deployment: Supports both full compilation environments and a binary-only mode for static analysis tasks.
• Verification System: Includes a dedicated server to track and verify PoC submissions from agents.