toby-bridges/api-relay-audit

API Relay Audit – What It Is

API Relay Audit is an open‑source, locally‑run security‑audit tool for AI API relays and LLM proxy services. It lets you probe a third‑party relay (e.g., an OpenAI‑compatible, Claude‑compatible, or Web3‑enabled gateway) before you trust it with production or wallet‑related traffic.

How It Works

• Zero‑dependency script – The core audit.py is a single Python file that only needs the Python standard library and curl. Your API key is sent only to the relay URL you specify.
• Four query families – The audit is split into separate “families” so the evidence for each risk stays isolated:
  1. API relay audit – General integrity checks (prompt injection, context truncation, tool‑call rewriting, SSE stream anomalies, etc.).
  2. Prompt‑injection audit – Detect hidden instructions or prompt leakage.
  3. Model‑substitution signals – Look for clues that the upstream model is different from what the relay claims (identity strings, latency patterns, stream fingerprints).
  4. Web3 relay audit – Wallet‑safety probes (transfer guidance, signed‑transaction refusal, private‑key leakage refusal).
• Profiles – Choose general (default), web3, or full (both) to run the appropriate set of probes.
• Output – After the run you get a structured Markdown report that lists per‑step findings and a final risk verdict (LOW / MEDIUM / HIGH). The report is meant for human review, not as a formal safety certificate.

Quick Start (30 seconds)

# Grab the latest audited script
AUDIT_SCRIPT_REF=v2.4.0
curl -fsSL "https://raw.githubusercontent.com/toby-bridges/api-relay-audit/${AUDIT_SCRIPT_REF}/audit.py" -o audit.py

# Run a basic audit
python audit.py --key <YOUR_KEY> --url <RELAY_BASE_URL> --output report.md

# Include Web3 checks
python audit.py --key <YOUR_KEY> --url <RELAY_BASE_URL> --profile web3 --output report.md

A sanitized example report is provided in the repo (docs/examples/sanitized-audit-report.md).

When You’d Use It

• You rely on a third‑party AI API relay, mirror, gateway, or LLM proxy.
• You need to verify that the relay does not inject hidden prompts, truncate context, rewrite tool calls, or swap the underlying model.
• Your workflow involves coding‑assistant automation, package‑install suggestions, or Web3 wallet actions and you want a repeatable, local audit before sending real traffic.
• You prefer a CLI‑only, offline tool over a web service that asks for your API key.

What It Doesn’t Do

• It does not certify a relay as safe; it only surfaces evidence.
• It does not replace a full manual security review or ongoing operational monitoring.
• An inconclusive result is not treated as clean – the ambiguous step stays visible in the report.

Distribution Modes

Mode	Description
audit.py	Stand‑alone, zero‑dependency script for quick local runs.
api_relay_audit/ + scripts/	Modular source tree with tests (≈ 808 pytest cases) for developers who want to extend or contribute.

DeepSeek Harness (DSH) Plugin

The repository also ships an installable dsh-api-relay-audit bundle for the DeepSeek Harness TUI ecosystem. Installing the plugin re‑uses the current DSH provider’s baseURL, model, and credential reference, keeping secrets out of command‑line arguments and logs.

DSH_PLUGIN_REF=v2.4.0
# Add to a DSH profile (e.g., web or cc‑tui)
dsh plugin --profile web add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"

The plugin provides commands such as /relay-audit and /relay-audit --profile web3.

Agent Skill Compatibility

• OpenClaw and Hermes skill files are retained for users who run those agents locally. They simply invoke the same 14‑step audit and produce a Markdown report; they are not published as separate registry packages.

Project Status (v2.4)

• Audit steps: 14
• CLI flags: 22
• Runtime profiles: general, web3, full
• Tests: 808 pytest cases
• License: AGPL‑3.0‑only

Getting Involved

• No code is required to contribute – you can file detector‑gap reports, give run‑feedback, improve documentation, or translate guides.
• Follow the contribution links in the README (e.g., Local Run Feedback, Detector Gap templates).

---

TL;DR: API Relay Audit is a lightweight, locally‑run Python tool that checks whether an AI API relay or LLM proxy is tampering with your requests or responses. It produces a reproducible Markdown audit report and can be used directly from the command line or as a DeepSeek Harness plugin. Ideal for developers and security‑conscious users who want to vet third‑party AI gateways before sending real traffic.