Detecting LLM-Generated Text with Classical Machine Learning – Results and Insights

TL;DR – What the demo shows

As of early 2026, a simple TF‑IDF + LinearSVC model trained on a few thousand Chinese sentences can flag LLM‑generated text with about 85 % accuracy per sentence. The author provides a browser‑based demo, source code, and model files on GitHub.

Why classical ML works for AI‑text detection

Modern large language models (LLMs) still exhibit statistical regularities in word choice, token distribution, and phrasing that differ from human authors. Even a naïve bag‑of‑words representation captures enough of these patterns for a linear classifier to separate the two classes.

"LLMs have detectable word‑choice patterns; even a Naive Bayes classifier should pick them up. I just didn’t expect the signal to be this strong." – author

The result is a lightweight detector that runs entirely in the browser without needing a server‑side LLM inference step.

Data generation strategy

Human corpus – ~10 000 Chinese articles (2010‑2022) scraped from a popular platform, filtered for length and engagement.
Synthetic LLM corpus – For each human article, the author:
- Prompted an LLM to generate a chapter summary.
- Fed the summary back to the same LLM to regenerate a full article.
Model diversity – Seven different LLM APIs were used (Gemini‑3‑flash, Gemini‑3‑pro, Qwen‑coder‑plus, GLM‑5, GLM‑4.7, Kimi‑k2.5, Deepseek‑v3.2) to avoid over‑fitting to a single model’s style.
Cost‑saving tricks – Low‑cost or free API access via CLI proxies, reverse‑engineered endpoints, and promotional credits.

The final dataset comprised roughly 8 500 human and 8 500 AI‑generated sentences, split 80 %/20 % for training and testing.

Model training and performance

Model	Accuracy	F1
Gemini	0.8809	0.8082
Qwen	0.8911	0.8974
GLM‑5 (Pony)	0.8493	0.8286
Kimi‑k2.5	0.8721	0.8473
GLM‑4.7	0.8436	0.8222
Doubao	0.8940	0.8700
Deepseek‑v3.2	0.8529	0.8403

All seven binary classifiers exceed 85 % accuracy and 80 % F1, confirming that the signal is robust across architectures.

Multi‑class attempt

A single 8‑class model (human + 7 AIs) achieved only ~50 % accuracy, indicating that the LLMs are stylistically similar enough that a binary distinction is far easier.

Ensemble voting

The final detector uses majority voting across the seven binary models: a sentence is labeled AI‑generated if ≥2 models flag it. This reduces false positives while preserving high recall.

Browser implementation

The TF‑IDF matrix (≈500 k features) and LinearSVC weights are exported as JSON.
Inference runs via ONNX Web Runtime (initial plan) but was later re‑implemented directly in JavaScript for simplicity.
Classification of a 1 M‑character document takes ~10 seconds on a typical laptop; typical few‑thousand‑character inputs are near‑instant.
Model size on disk: 107 MB (gzipped ≈38 MB). A reduced 80 k‑feature version loses only 3–4 % accuracy.

Real‑world testing

Known‑training models

Doubao Seed Code – 93 % AI score
Deepseek‑v3.2 – 78 % AI score

Unseen models (generalization)

Claude Sonnet 4.6 – 71.9 %
GPT 5.2 – 73.3 %
Other recent models (MiMo‑V2, GPT‑4o) – 70 %–90 % range

Human‑written control set

Ten pre‑2022 web novels: AI scores 4.9 %–29.1 %, all well below the 50 % “maybe AI” threshold.
Random high‑engagement fanfics (pre‑2022, 10 000 samples): false‑positive rate 0.04 % at a 60 % AI‑score cutoff.

Attempts to evade detection

Technique	Effect on AI score
Google Translate round‑trip (CN→EN→CN)	85 % → 89.9 % → 85.0 %
Youdao Translate round‑trip	89.9 % → 79.2 %
Prompted rewrite (“minimize AI flavor”)	89.9 % → 83.0 %
Complex “flavor‑remover” prompt	89.9 % → 79.3 %

Even aggressive post‑processing only reduces the detector’s confidence by a few percentage points, indicating the classifier is robust to simple obfuscation.

Community reactions (Hacker News highlights)

"Text is simply not information dense enough to decode an arbitrary provenance signal; today you might catch model‑specific quirks, but tomorrow models will be undetectable." – @akersten

"A browser‑extension that flags every paragraph would be as useful as an ad‑blocker." – @Krssst

"Small encoder‑only transformers can achieve AUROC 99.81 on‑device; the arms race will continue." – @woadwarrior01

"If detection becomes cheap, model developers will likely retrain to evade it, turning this into an arms race." – @gleenn

These comments underscore both optimism (practical tools) and skepticism (future cat‑and‑mouse dynamics).

Limitations and future work

Language scope – The current pipeline is trained on Chinese text; applying it to English or other languages would require a new labeled corpus.
Model drift – As LLMs evolve, the statistical fingerprints may shift, necessitating periodic retraining.
Fine‑grained attribution – The binary detector cannot reliably identify which LLM produced a text; multi‑class attempts showed limited separability.
Human‑in‑the‑loop – Even with 85 % sentence accuracy, a human reviewer is still needed for high‑stakes decisions (e.g., academic plagiarism).

Takeaway

A straightforward TF‑IDF + LinearSVC pipeline, trained on a modestly sized Chinese corpus, can reliably flag modern LLM‑generated text with ~85 % sentence‑level accuracy. The approach is lightweight enough to run entirely in the browser, scales to millions of characters, and resists simple translation‑or‑prompt‑based evasion. While not a silver bullet—especially as models improve—it demonstrates that “classical” machine learning remains a viable tool in the emerging AI‑text detection arms race.