chen-006/meow-llm-detector

What it solves

This tool helps users identify which LLM is actually powering a specific API endpoint. It addresses the problem of "API spoofing" or routing, where a service provider might claim to be providing one model (e.g., GPT-4) but is actually routing requests to a cheaper or different model.

How it works

The detector sends a series of short, specific questions to the API. It then compares the distribution of the answers received against a set of built-in benchmarks (baselines) for known models like GPT and Claude. Because different models tend to choose different answers for the same open-ended prompt, the tool can determine which model's response pattern most closely matches the target API.

Who it’s for

Users who want to verify the authenticity of LLM API providers and developers who want to ensure they are receiving the model they paid for.

Highlights

• Built-in Benchmarks: Includes pre-collected baselines for various GPT and Claude versions.
• Customizable Testing: Users can choose between low, medium, and high intensity tests (20, 50, or 100 requests).
• Local Execution: The program runs locally on the user's machine, ensuring API keys are handled securely.
• Benchmark Generation: Allows users to manually enter, import, or use AI to generate new test questions and sample them from trusted APIs.
• Cross-Platform Support: Compatible with Windows, macOS, and Linux via Python 3.11+.