canarybyte/veridrop

Veridrop – Open‑source checker for AI‑API relay (proxy) services

What it is

• A Python‑based tool that verifies whether a third‑party “relay” or proxy for Claude, OpenAI or Gemini really forwards requests to the claimed model and preserves the model’s full capabilities.
• It can be used via the public web UI at veridrop.app (no sign‑up, API keys are never stored) or self‑hosted as a CLI + FastAPI service.

Why it matters

• Many cheap relay services claim to expose the latest LLMs but often downgrade the model, strip features (PDF reading, tool use, function calling, long‑context windows) or return malformed responses that break client code.
• Veridrop runs a battery of probe requests, compares the replies to official‑API baselines, and produces a scored report indicating:
  1. Authenticity – is the relay truly forwarding to the advertised model?
  2. Capability – are advanced features being hidden?
  3. Compliance – do fields, streaming events and usage accounting follow the official spec?

How it works

• Baseline collection – bench.sh can fetch reference responses from the official Anthropic, OpenAI or Gemini endpoints (stored under data/baselines/).
• Detectors – protocol‑specific detector modules (Claude 11, OpenAI 8, Gemini 7) issue targeted requests (e.g., a PDF‑in‑base64 payload, function‑call schema, long‑context “needle‑in‑haystack” token bursts) and examine:
  • cryptographic signature in Claude’s thinking block (cannot be forged)
  • presence of model‑specific fields, usage fingerprints, ID prefixes, streaming SSE sequences, etc.
• Scoring – each detector returns a score and weight; the overall score is a weighted average. Critical failures (e.g., missing signature, stripped PDF handling) cap the final verdict at marginal even if the total score is high.
• Report – JSON output plus a human‑readable HTML page and a shareable JPG card. The web UI also aggregates public reports into a leaderboard.

Installation / self‑hosting

git clone https://github.com/canarybyte/veridrop.git
cd veridrop
python3 -m venv venv
./venv/bin/pip install -e ".[dev,web]"
# run the API locally
./venv/bin/uvicorn web.server:app --host 0.0.0.0 --port 8000

• The CLI (relay-detector) supports three modes:
  • quick (5 core checks, ~15 s, ~$0.005)
  • standard (8 checks, ~40 s, ~$0.012)
  • full (all 11 checks, ~70 s, ~$0.020) – required for baseline comparison.
• Example CLI usage:

# ping a relay
./venv/bin/relay-detector ping --model claude-haiku-4-5
# full detection
./venv/bin/relay-detector detect \
  --model claude-haiku-4-5 --mode full -o out.json
# compare with official baseline
./venv/bin/relay-detector compare out.json

Key features

• Cryptographic verification for Claude via the thinking_signature field (unique to Anthropic).
• Long‑context probing with multi‑stage token bursts to catch “advertised 1 M tokens but actually 200 k”.
• Capability checks for PDF extraction, tool‑use, function calling, structured output, and streaming compliance.
• Public leaderboard showing aggregated scores for thousands of relays.
• Privacy‑first – API keys live only in memory; reports redact keys automatically.

License

• AGPL‑3.0‑or‑later. Free for personal, research or internal use. Running a public SaaS that offers the service to others requires you to open‑source any modifications (network‑use clause of AGPL).

Who might use it

• Developers who rely on third‑party relay endpoints and need to ensure they are getting the model they pay for.
• Platform operators who want to audit the relays they expose to users.
• Security/research teams investigating fraudulent AI‑API services.
• Anyone curious about the real capabilities and compliance of a relay before handing over an API key.