hanlinwenyuan/hlwy-ai-checker

检查第三方AI API是否掺假以及渠道一致|基于llm指纹的AI模型识别

What it solves

It detects whether third-party AI API providers are providing the genuine models they claim to be or are using "fake" substitutes. It also verifies if the API channels are consistent.

How it works

The tool leverages the fact that LLMs are not truly random number generators. When asked to "pick a random number," different models exhibit distinct statistical biases based on their training data, architecture, RLHF alignment, and tokenization strategies. By sampling these responses over a large volume, the tool creates a statistical "fingerprint" of the model, which is difficult to override with system prompts.

Who it’s for

Users of third-party AI API services who want to verify the authenticity of the models they are paying for.

Highlights

  • High precision and clear differentiation between models.
  • Low token consumption during testing.
  • Strong adaptability through a calibration process using an official API key.
  • One-click testing functionality for ease of use.

Related

  • Project
    canarybyte/veridrop

    An open-source authenticity and compliance checker for AI API relay services that verifies if providers are actually using the models they claim.

  • Project
    AISBench/benchmark

    AISBench Benchmark is an open‑source evaluation framework for LLMs, multimodal models, and AI agents. It supports both accuracy (using many public datasets) and performance (latency, throughput, cache) testing of locally‑run or service‑based models, with extensive recent additions such as Harbor‑agent benchmarking, prefix‑cache stress tests, and response‑anomaly detection.

  • Project
    Ahmet-Dedeler/ai-llm-comparison

    An open-source AI model comparison tool that provides real-time pricing and feature comparisons across multiple LLM providers.

  • Project
    SeraphimSerapis/tool-eval-bench

    A tool-calling quality benchmark for LLMs that evaluates tool selection, parameter precision, and agentic workflow reliability across various self-hosted serving stacks.