Ikaleio/lm-detector

What it solves

This project addresses the issue of "model swapping" in third-party API providers. It helps users verify whether an API claiming to provide a specific large language model (LLM) is actually delivering that model or a different one.

How it works

It uses a non-semantic "fingerprinting" technique. The tool prompts an LLM to generate approximately 300 integers between 1 and 355. Because LLMs have inherent biases in how they produce "random" numbers, these outputs create a stable numerical distribution unique to specific model families. The detector compares this distribution against a reference database of known models to identify the most likely candidate. Optionally, it can also perform tokenizer probing to analyze how the model breaks down text, providing additional, though secondary, evidence.

Who it’s for

It is for developers, researchers, and users who rely on third-party API aggregators or "model transit stations" and want to verify the authenticity of the models they are paying for or using.

Highlights

• Multi-protocol support: Works with OpenAI, Chat Completions, and Anthropic Messages formats.
• Dual interface: Offers both a web-based dashboard and a command-line interface (CLI) for batch testing and local data management.
• Extensible database: Includes a reference library of major models like GPT, Claude, Gemini, Grok, Qwen, and DeepSeek, with tools to sample and retrain the detector locally.
• Privacy-conscious: Supports direct API connections, browser-based requests, or self-hosted Cloudflare Worker proxies to avoid relying on the project's own servers.