canarybyte/veridrop

What it solves

Veridrop addresses the problem of "fake" AI API relay or proxy services. Many third-party providers claim to offer access to high-end models (like Claude, GPT, or Gemini) but may actually be routing requests to cheaper, smaller models or stripping away advanced capabilities to save costs. Veridrop provides a way to verify if a relay is authentic, compliant with official API specifications, and capable of delivering the features it claims.

How it works

The tool takes a base_url, api_key, and model name and runs a suite of probe requests. It then compares the responses against "official baselines" (verified responses from the actual providers) at three levels:

• Field Level: Checks if response fields and IDs match official formats.
• Protocol Level: Verifies streaming event sequences and SSE (Server-Sent Events) compliance.
• Cryptographic Level: For Claude, it specifically validates the signature field in thinking blocks, which is nearly impossible for relays to forge.

It also includes a "needle-in-a-haystack" test to verify if the claimed long context window is actually available or if the input is being silently truncated.

Who it’s for

• Developers and Users who purchase API access from third-party relay services and want to ensure they aren't being scammed.
• Service Providers who want to prove the authenticity and quality of their relay services through transparent, open-source evidence.

Highlights

• Cryptographic Verification: Uses Anthropic's thinking signatures to provide mathematical proof of authenticity for Claude models.
• Context Window Testing: Employs multi-layer probes to detect "high-end fraud" where providers claim large context windows but deliver much smaller ones.
• Multi-Protocol Support: Works with Anthropic Messages API, OpenAI Chat Completions, and Gemini's OpenAI-compatible API.
• Privacy-First: The hosted version does not persist API keys to disk and requires no registration.
• Detailed Scoring: Provides a weighted score and a verdict (Passed, Marginal, Failed) based on critical, major, and minor discrepancies.