ModelTrace – Local Model Attribution Tool

What it is

• A Python‑based utility that tries to identify which LLM (e.g., OpenAI GPT‑series or Anthropic Claude) generated a given response.
• It works by sending three specially crafted “challenge” prompts to the target model, extracting numeric fingerprints from the outputs, and comparing them against a local library of known model fingerprints.

Key components

File / Folder	Role
app.py	FastAPI/Gradio web server that serves the UI at http://127.0.0.1:7860/.
fingerprint.py	Implements the fingerprint extraction and the attribution algorithm.
bank_builder.py	Builds and calibrates the fingerprint banks (the reference libraries).
challenge_suite.py	Generates the three independent integer‑generation challenges used for testing.
enrollment.py	Handles API calls to OpenAI or Anthropic, collects the model’s answers, and stores them for later analysis.
rebuild_unified_bank.py	Merges individual banks into a single global bank used at inference time.
static/ & templates/	Front‑end assets for the browser‑only demo that runs entirely client‑side.
data/	Stores the pre‑computed fingerprint banks (*_bank.json, unified_bank.json) and raw reference logs (*_reference.jsonl).

How it works

1. Challenge generation – Three deterministic prompts ask the model to emit long integers. The exact numbers differ slightly between model families because of subtle biases in their random number generators.
2. Fingerprint extraction – The responses are turned into two features:
   • A Hellinger‑distance based similarity to the global model‑center (after removing environment‑specific drift).
   • An ordered‑block numeric‑sequence feature. The final score is 0.75 * Hellinger + 0.25 * sequence.
3. Probability calculation – Scores for all candidate models are fed through a soft‑max (temperature β) to obtain a probability distribution. Model‑family probabilities are summed from their members.
4. Result – The tool reports the most likely model (and its family) together with the computed probabilities. If the target model is not in the library, the closest known model is returned.

Running the tool

python -m pip install -r requirements.txt
python start.py   # launches the web UI on http://127.0.0.1:7860/

• Manual mode – Paste the three challenge prompts, run them on the model you want to test, and paste the three full outputs into the UI.
• API mode – Provide a base URL, API key, and model name; the tool will call the model automatically (supports OpenAI Chat Completions and Anthropic Messages) and perform the three challenges for you.

Fingerprint library

• Currently ships with two families (GPT‑like and Claude) covering 18 concrete model versions (e.g., gpt-5.6-luna, claude-opus-5).
• The library is built from official Codex subscriptions (GPT) and the OAIPro service (Claude).
• New libraries can be created or extended via the provided API or by running bank_builder.py on fresh challenge data.

Integration with Codex

• A separate plugin, ModelTrace Guard, can be installed in the Codex CLI. It automatically runs the three challenges in the background for each Codex task, records results, and alerts the user if the model’s fingerprint changes unexpectedly.
• The plugin works without consuming extra model quota for the UI refresh; only the three challenge calls use inference credits.

Limitations & cautions

• The attribution is probabilistic and only as good as the fingerprint bank; results are suggestive, not definitive.
• Models not present in the bank will still be assigned to the nearest known model, which may be misleading.
• Claude’s “Code” endpoint shows larger bias due to system‑prompt effects, so the tool advises against using it there.
• API keys are never written to disk; they exist only for the duration of a request.

Acknowledgements

• Inspired by hanlinwenyuan/hlwy-ai-checker, which first demonstrated that subtle numeric generation biases can be used for cross‑API consistency checks.

---

ModelTrace offers a lightweight, locally‑run way to fingerprint LLMs, useful for developers who need to verify which model version they are actually querying, especially in environments where the provider’s model name may be ambiguous or hidden.