Detecting LLM-Generated Text with Classical Machine Learning

Classical ML can effectively distinguish AI from human text

Mainstream Large Language Model (LLM) generated text exhibits strong statistical patterns that can be distinguished from human-written content using traditional machine learning. A project utilizing a combination of TF-IDF (Term Frequency-Inverse Document Frequency) and Linear Support Vector Classification (SVC) achieved approximately 85% accuracy in detecting AI-generated sentences, demonstrating that "classical" ML is often more effective and less computationally expensive than using another LLM for detection.

Methodology: TF-IDF and Linear SVC

Rather than relying on text perplexity—which often results in high inference costs and poor cross-model generalization—this approach uses a standard classification pipeline from scikit-learn.

Data Generation

To train the classifier, a balanced dataset was created using:

• Human-written samples: Approximately 10,000 multi-thousand-character texts published between 2010 and 2022 (pre-ChatGPT).
• AI-generated samples: The same human texts were summarized by an LLM, and those summaries were then used to regenerate full articles. This ensured the AI content closely matched the original human content in genre and subject.

Seven different models were used to generate the AI training data to ensure diversity: gemini-3-pro, qwen-coder-plus, glm-5, glm-4.7, kimi-k2.5, doubao-seed-code, and deepseek-v3.2.

Training and Architecture

The system employs a majority voting mechanism across seven separate binary classifiers. A sentence is flagged as AI-generated if at least two of the seven models detect it.

Performance Metrics:

• Sentence-level accuracy: ~85% across most models.
• F1 Score: Over 80% for all binary classifiers.
• Feature set: The final web-demo version uses 500,000 features to maintain accuracy while optimizing for browser-based inference.

Performance and Generalization

Detection Rates

Testing on both training and unseen models shows a high degree of robustness:

• Known Models: Deepseek V3.2 and Doubao Seed Code were detected at rates of 78.4% and 93.0%, respectively.
• Unseen Models: Claude Sonnet 4.6 and GPT 5.2 were detected at rates of 71.9% and 73.3%, respectively.
• Human Baseline: Pre-2022 web novels and fanfics consistently scored under 30% AI detection, with a false positive rate of less than 0.01% when using a 70% threshold.

Bypassing Detection

Attempts to evade the classifier using common "anti-AI" techniques showed minimal success:

• Translation Roundtripping: Translating text from Chinese to English and back to Chinese only slightly reduced detection rates (e.g., 89.9% $\rightarrow$ 85.0% via Google Translate).
• Prompt Engineering: Using prompts like "Rewrite the above article to minimize AI flavor" reduced detection from 89.9% to 83.0%.

Technical Implementation: Browser-Based Inference

To avoid server maintenance and adhere to a serverless philosophy, the model was implemented in JavaScript for client-side execution. The TF-IDF and SVM logic was ported to JS, allowing for near-instant inference on typical inputs. While the full model is approximately 107MB in JSON format (roughly 38MB gzipped), reducing the feature set to 50k-80k caused significant fluctuations in false positives, leading the author to retain the 500k feature set.

Community Perspectives and Counterpoints

Discussion among technical peers highlights a fundamental debate regarding the longevity and reliability of AI detection:

• The "Arms Race" Argument: Some argue that any detectable pattern will eventually be used by model trainers to refine LLMs, making these patterns disappear. As one user noted, "Any cheap pattern can and presumably will be used to retrain if it becomes an effective way to catch AI."
• Information Density: Critics suggest that text is too sparse to contain a permanent signal of provenance. One commenter argued that while images have tell-tale artifacts, words do not, and that current detection is merely identifying "today's tells" rather than a fundamental difference in nature.
• The Risk of False Positives: There is significant concern regarding the ethical implications of false positives in high-stakes environments, such as academic dissertations, where a "good enough" detector could unfairly penalize human writers who happen to write in a structured, predictable style.
• Alternative Approaches: Some suggest shifting focus from "who wrote it" to "how much effort went into it," arguing that the value of text lies in its ability to engage the reader and remove extraneous information—tasks that still require human effort regardless of the tool used.