API Relay Audit – Local security audit for AI API relays and LLM proxies

What it is

API Relay Audit is a Python‑based, zero‑dependency command‑line tool that lets you test a third‑party AI API relay (also called an LLM proxy) before you send real traffic to it. It runs entirely on your machine, sending your API key only to the relay URL you specify, and produces a human‑readable Markdown (or JSON) report that flags suspicious behaviour.

Why it matters

AI providers such as Anthropic or OpenAI are increasingly accessed through unofficial “relay” services that forward requests. Those relays can:

• inject hidden prompts or instructions,
• truncate or rewrite the context you send,
• swap the underlying model, or
• leak credentials in error messages. The tool is motivated by recent research (e.g., the Your Agent Is Mine paper) and threat reports showing malicious relays harvesting keys.

Core features (as described in the README)

Feature	What it does
14‑step audit	Runs a fixed sequence of probes that check prompt injection, context truncation, tool‑call rewriting, SSE stream integrity, model‑identity signals, and (optionally) Web3‑wallet safety.
Profiles	general (default), web3 (wallet‑focused checks), and full (both).
Output formats	Markdown report (default) with per‑step evidence and a LOW/MEDIUM/HIGH risk summary; --format json for automation.
Zero‑dependency script	A single audit.py file that only needs Python and curl.
DeepSeek Harness plugin	Installable as dsh-api-relay-audit for the DeepSeek Harness TUI, re‑using the same audit logic.
Agent skill compatibility	Skill files for OpenClaw and Hermes agents let those agents invoke the audit locally.
Evidence boundaries	The tool records signals (e.g., model self‑identification, latency) but clearly marks them as signals, not proof of a malicious relay.
Safety guarantees	Does not certify a relay as safe; inconclusive results are shown, and manual review is still required.

How to use it (quick start)

# Grab the exact released script
AUDIT_SCRIPT_REF=v2.5.0
curl -fsSL "https://raw.githubusercontent.com/toby-bridges/api-relay-audit/${AUDIT_SCRIPT_REF}/audit.py" -o audit.py

# Run a basic audit
python audit.py --key <YOUR_KEY> --url <RELAY_URL> --output report.md

# Include Web3 checks
python audit.py --key <YOUR_KEY> --url <RELAY_URL> --profile web3 --output report.md

For CI pipelines you can request JSON output:

python audit.py --key-env API_RELAY_AUDIT_KEY --url <RELAY_URL> \
    --format json --output report.json

When you’d want it

• You rely on a third‑party Claude‑compatible or OpenAI‑compatible proxy and want a repeatable local audit before production use.
• You are building an autonomous agent that will call a relay for tool use or Web3 transactions.
• You need a scriptable way to generate evidence for a security review or bug‑bounty report.

What it does not do

• It is not a formal safety certification.
• It cannot guarantee a relay is clean; inconclusive probes are left visible.
• It does not replace continuous monitoring or manual code review.

Project health (as of the README)

• Latest release: v2.5.0
• 14 audit steps, 23 CLI flags, 885 pytest‑collected tests.
• AGPL‑3.0‑only license (source must stay open).
• Active documentation site, example reports, and a contribution guide.

---

All details above are taken directly from the repository’s README; no additional features have been inferred.