hanlinwenyuan/hlwy-ai-checker

hlwy‑ai‑checker – What it does

Purpose – A locally‑run tool that lets you compare the behaviour of an official LLM API (e.g., OpenAI) with a third‑party “reseller” API. By sending many identical prompts to both endpoints and analysing the statistical distribution of the responses, the program produces a fingerprint for each service. If the fingerprints differ enough, the tool flags the third‑party channel as potentially altered or “fake”.

Why it matters – Many companies rebadge large‑language‑model APIs and sell them under their own keys. Users may want to verify that the service they are paying for actually runs the same model as the official provider, without exposing their API keys or data to the internet.

---

How to use it

1. Download the latest release ZIP from the Releases page and unzip it.
2. Run python start.py.
3. Automatic mode – one‑click testing with an optional “anti‑targeting” switch (the screenshot in the README shows the UI).
4. Manual mode –
   • Enter the official API key and base URL (must contain /v1).
   • Enter the third‑party API key and base URL, select the same model you calibrated with, and start the test.
5. After the run finishes, the UI shows similarity scores, statistical charts and token usage.

---

Main features

Feature	What you get
Statistical fingerprinting	Repeated sampling of the same prompt reveals subtle biases (training data, tokeniser, sampling parameters) that act like a model’s “signature”.
High discrimination	The screenshots claim the tool can clearly separate genuine from altered endpoints.
Low token consumption	Designed to use as few tokens as possible while still gathering enough data for a reliable fingerprint.
Adaptive after calibration	You first calibrate on the official endpoint; the same settings are then applied to the third‑party endpoint, improving comparability.
Local‑only execution	No network calls other than the API requests themselves, so your keys and test data never leave your machine.

---

Underlying idea (as described in the README)

• Large language models are not pure random generators. Their “random” outputs are influenced by training data, architecture, RLHF alignment, tokenisation and sampling settings.
• By sending many prompts and collecting the distribution of answers, you can build a statistical “fingerprint”.
• Comparing the fingerprint from a third‑party service with the one obtained from the official service indicates whether the two behave the same way.
• The method is statistical, not forensic – results depend on model version, number of samples, network stability, etc., and cannot be used as legal proof.

---

Limitations & disclaimer

• Results are indicative only; they cannot definitively prove that a third‑party API is using a different model.
• Randomness in the model and network issues can affect the similarity score.
• The author disclaims any liability for commercial disputes arising from the tool’s output.

---

Who might find this useful?

• Developers or companies that purchase LLM access from resellers and want an inexpensive sanity‑check.
• Security‑oriented users who are cautious about exposing their API keys.
• Researchers interested in model‑level fingerprinting techniques.

---

Quick start checklist

1. Python installed (the repo does not list dependencies, but requests/httpx and a GUI library such as tkinter or PyQt are likely required).
2. Official API key and base URL (e.g., https://api.openai.com/v1).
3. Third‑party API key and base URL.
4. Run the script and follow the on‑screen prompts.

---

In short: hlwy‑ai‑checker is a niche, locally‑run utility for statistically comparing the output patterns of official versus third‑party LLM APIs, helping users detect possible tampering or model substitution while keeping their credentials private.