0ca/BoxPwnr

A modular framework for benchmarking LLMs and agentic strategies on security challenges across HackTheBox, TryHackMe, PortSwigger Labs, Cybench, picoCTF and more.

What it solves

BoxPwnr is an experimental framework designed to test how effectively Large Language Models (LLMs) and AI agents can autonomously solve Capture The Flag (CTF) challenges and security labs. It provides a standardized way to benchmark different agentic architectures across various cybersecurity platforms.

How it works

The system operates through an iterative execution loop:

  1. Environment: Commands are typically executed within an isolated Kali Linux Docker container, with automatic VPN setup where required.
  2. Agent Loop: An LLM (or a CLI agent like Claude Code or Grok) receives a system prompt and suggests commands. These commands are executed in the environment, and the output is fed back to the LLM for analysis.
  3. Automation: Agents are instructed to provide fully automated commands to avoid manual interaction.
  4. Tracking: The system records full conversation logs (traces), tracks token usage and costs, and generates summaries upon success.

Who it’s for

It is intended for researchers and developers interested in AI security, agentic workflows, and benchmarking the reasoning and problem-solving capabilities of LLMs in complex, real-world security scenarios.

Highlights

  • Broad Platform Support: Integrates with 16 different platforms including HackTheBox, PortSwigger, picoCTF, TryHackMe, and XBOW.
  • Flexible Solver Selection: Supports various solvers including single_loop, hacksynth (a planner-executor-summarizer architecture), and external CLI tools like Cursor and Claude Code.
  • Extensive Model Compatibility: Works with a wide range of models via OpenRouter, NVIDIA NIM, and local providers like Ollama.
  • Trace Analysis: Provides a web viewer to replay solving traces step-by-step to analyze LLM reasoning.

Related

  • Project
    InternLM/WildClawBench

    An end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using tools like browsers and bash terminals.

  • Project
    benchflow-ai/benchflow

    A universal environment framework that runs AI agents against task environments and scores them through a standardized contract, treating benchmarks as frozen environments.

  • Project
    FishCodeTech/muteki

    An autonomous multi-model AI agent swarm for offensive and defensive security automation, designed to solve CTFs and perform penetration testing by coordinating diverse LLMs via a shared blackboard.

  • Project
    microsoft/AIOpsLab

    A holistic framework for developing and benchmarking autonomous AIOps agents by automating microservice deployment, fault injection, and telemetry in Kubernetes environments.

  • Project
    harbor-framework/harbor

    A framework for evaluating and optimizing AI agents and language models by running benchmarks across scalable, containerized environments.