0xSteph/pentest-ai

pentest‑ai (ptai) – AI‑coordinated, oracle‑verified penetration testing

What it is – ptai is a Python‑based command‑line tool that uses a large language model (LLM) only to coordinate a suite of existing security scanners, exploit tools and custom probes. After a finding is discovered, the tool re‑runs a deterministic, non‑destructive proof‑of‑concept (the “oracle”) against the target. A finding is marked VERIFIED only when the oracle reproduces the issue N/​N times, so the report contains only proven vulnerabilities, not the noisy output typical of scanners like Nuclei or ZAP.

Key capabilities

• LLM‑orchestrated workflow – an LLM (Claude, OpenAI, Ollama, etc.) drives a pipeline of 200+ wrapped tools and ~60 web probes, deciding which to run and how to chain results.
• Oracle verification – each candidate exploit is re‑executed by a dedicated “machine oracle” (e.g., logging in with a leaked secret, re‑sending a crafted request). Only when the oracle succeeds does the finding earn a VERIFIED badge and a portable proof capsule that can be replayed later (ptai replay).
• Multi‑phase agents – recon, authentication handling, web/API/AD/cloud testing, privilege‑escalation, exploit‑chain building, PoC validation, detection rule generation, reporting, plus optional red‑team, mobile and wireless agents.
• Zero‑trust execution – all network traffic originates from the user’s machine; no cloud service is required unless the user supplies an LLM endpoint. The tool can run completely offline with Ollama.
• CI/CD integration – GitHub Action example shows how to fail a PR when a verified high‑severity finding appears, with SARIF, JUnit or PDF reports.
• Scope safety – active tools are host‑locked; the scanner never follows third‑party URLs scraped from a page.
• Cost guard – for the standalone (non‑MCP) mode the LLM spend is capped at $10 USD by default; the limit is configurable via PTAI_PRICE_LIMIT.

How it works (high‑level flow)

1. ptai start <target> launches an engagement.
2. Recon gathers ports, DNS, service fingerprints.
3. Auth logs in (credentials can be referenced from env vars, 1Password, Vault, etc.) and maintains a session.
4. Web / API / AD / Cloud agents run deterministic probes (60+ SPA‑aware checks covering OWASP Top 10, API Top 10, etc.).
5. Findings are stored in a local SQLite DB.
6. Exploit‑chain agent correlates findings into multi‑step attack paths.
7. Poc‑validator (oracle) re‑executes each finding; only those that succeed become VERIFIED and are packaged into a replayable capsule.
8. Report agent emits markdown, HTML, PDF, SARIF, JUnit, and optional detection‑rule exports.

Installation & usage

pip install ptai               # core package
ptai start https://target.com   # interactive run (prompts for auth, scope, etc.)
ptai demo                       # quick demo against bundled vulnerable app
ptai replay <capsule>           # replay a proof capsule yourself

Three execution paths are documented:

• Path 1 – drive ptai from Claude Code (or other MCP‑compatible client) with no API key; the client’s existing Anthropic subscription supplies the LLM.
• Path 2 – auto‑configure other MCP clients (Cursor, VS Code Copilot, etc.) via ptai setup --mcp.
• Path 3 – standalone CLI for CI, air‑gapped terminals, or users without an MCP client; requires an LLM API key or a local Ollama server.

Benchmarks (as reported in the README)

• On a custom honeypot containing 14 vulnerability classes, ptai verified 23 findings with 100 % precision and zero false positives.
• On the OWASP Juice Shop, a single scan produced 12 oracle‑verified findings (e.g., JWT alg:none, BOLA, sequential IDOR, type confusion).
• The tool ships the honeypot harness and clean‑app test suite so these numbers are reproducible.

When to use it

• Internal red‑team or penetration‑testing engagements where you need high‑confidence, reproducible proof of each vulnerability.
• CI/CD pipelines that must gate merges on verified security findings.
• Environments with strict data‑privacy requirements (runs locally, no telemetry, optional offline LLM).
• Teams that already use an MCP‑compatible LLM client (Claude Code, Cursor, etc.) and want to extend it with security tooling.

Limitations / cautions

• The LLM is only a coordinator; detection coverage depends on the curated probe library (≈60 web probes, 200+ wrapped tools). Novel bugs outside that library will not be found.
• Oracle verification adds runtime overhead; a full scan can take several minutes on a laptop.
• In the standalone mode the LLM usage incurs API costs; the default $10 spend cap may abort long engagements unless raised.
• The tool is offensive and requires an Authorized Use Policy agreement; it must only be run against systems you have permission to test.

---

TL;DR – pentest‑ai (pip package ptai) is an open‑source, LLM‑orchestrated penetration‑testing framework that distinguishes itself by automatically re‑executing every discovered exploit to produce a machine‑verified proof. It runs locally, integrates with existing security tools, supports CI/CD, and offers both cloud‑based (MCP) and fully offline usage modes.