sunblaze-ucb/exploitgym

ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities designed to evaluate AI agents' ability to develop exploits.

What it solves

ExploitGym provides a standardized, large-scale benchmark to evaluate how well AI agents can develop functional exploits from real-world security vulnerabilities. It addresses the need for realistic testing environments to measure an agent's ability to turn a vulnerability into a successful attack.

How it works

The system uses a collection of 869 instances based on real vulnerabilities found in userspace programs, the Linux kernel, and Google's V8 engine. It employs a controller, a firewall for outbound network isolation, and an LLM proxy to manage the execution of AI agents. Agents interact with target Docker images containing the specific vulnerabilities to attempt to derive the expected flags.

Who it’s for

This tool is designed for security researchers and AI developers working on the capabilities and safety of AI agents in the context of cybersecurity and exploit development.

Highlights

  • Real-world data: Built from actual vulnerabilities in the Linux kernel, V8 engine, and userspace programs.
  • Large scale: Includes 869 distinct benchmark instances in version 1.0.
  • Isolated environment: Uses Docker and a dedicated firewall to ensure agent containers are network-isolated.
  • Comprehensive tooling: Includes a controller and LLM proxy to facilitate the evaluation of AI agents.

Related

  • Project
    sunblaze-ucb/cybergym

    A large-scale evaluation framework for assessing AI agents' ability to perform real-world vulnerability analysis and Proof-of-Concept exploit generation.

  • Project
    ServiceNow/BrowserGym

    An open-source framework that provides a unified interface for researching and evaluating web agents across multiple benchmarks and open-ended web tasks.

  • Project
    InternLM/WildClawBench

    An end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using tools like browsers and bash terminals.

  • Project
    vxcontrol/pentagi

    An autonomous AI-powered penetration testing platform that uses a multi-agent system to automate security vulnerability research and exploitation in isolated environments.

  • Project
    Kritt-ai/open-kritt

    An open-source security research platform that orchestrates AI agents to find, validate, and rank vulnerabilities in code through focused, parallelized workflows.