sunblaze-ucb/exploitgym

ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities designed to evaluate AI agents' ability to develop exploits.

What it solves

ExploitGym provides a standardized, large-scale benchmark to evaluate how well AI agents can develop functional exploits from real-world security vulnerabilities. It addresses the need for realistic testing environments to measure an agent's ability to turn a vulnerability into a successful attack.

How it works

The system uses a collection of 869 instances based on real vulnerabilities found in userspace programs, the Linux kernel, and Google's V8 engine. It employs a controller, a firewall for outbound network isolation, and an LLM proxy to manage the execution of AI agents. Agents interact with target Docker images containing the specific vulnerabilities to attempt to derive the expected flags.

Who it’s for

This tool is designed for security researchers and AI developers working on the capabilities and safety of AI agents in the context of cybersecurity and exploit development.

Highlights

  • Real-world data: Built from actual vulnerabilities in the Linux kernel, V8 engine, and userspace programs.
  • Large scale: Includes 869 distinct benchmark instances in version 1.0.
  • Isolated environment: Uses Docker and a dedicated firewall to ensure agent containers are network-isolated.
  • Comprehensive tooling: Includes a controller and LLM proxy to facilitate the evaluation of AI agents.

Related

  • Project
    sunblaze-ucb/cybergym

    A large-scale evaluation framework for assessing AI agents' ability to perform real-world vulnerability analysis and Proof-of-Concept exploit generation.

  • Project
    ServiceNow/BrowserGym

    BrowserGym is an open‑source Python library that turns web browsing into Gymnasium environments, letting researchers plug LLM‑based agents into many web‑automation benchmarks (MiniWoB, WebArena, WorkArena, etc.). It provides a unified API, modular install options, Playwright‑driven browsers, and easy extensibility via an `AbstractBrowserTask` base class. The ecosystem includes AgentLab for large‑scale evaluation and a demo agent that works with OpenAI models.

  • Project
    InternLM/WildClawBench

    WildClawBench is an end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using real tools like browsers and bash terminals.

  • Project
    vxcontrol/pentagi

    PentAGI is a self-hosted, multi-agent system that automates penetration testing by using AI agents to plan and execute security tools within isolated Docker sandboxes.

  • Project
    Kritt-ai/open-kritt

    open·kritt is an open‑source, self‑hosted platform that orchestrates AI agents (via Codex, Claude Code, Grok Build, etc.) to run custom, prompt‑driven security workflows on code repositories. It builds focused tasks, runs them in parallel Docker containers, validates findings, de‑duplicates and ranks them, and exports a complete report bundle. Designed for bug‑bounty researchers and DevSecOps, it offers a web UI and a headless CLI, supports any model provider, and is released under AGPL‑3.0.