Measuring AI Writing on arXiv: Method, Results, and Limitations
Measuring AI Writing on arXiv: Method, Results, and Limitations
Quick takeaway
A calibrated AI‑text detector flags about one‑third of new arXiv papers as machine‑written, rising from a 0.4% baseline in 2021‑2022 to a peak of 39% in early 2026; computer science papers show the highest flagged share at roughly 65%.
How the measurement was anchored
The study avoids the common pitfall of headline‑level AI‑writing claims by fixing a false‑positive floor. Papers from 2021‑2022 (pre‑ChatGPT) were treated as genuine human text, and the detector threshold was set so that exactly 0.4% of these papers are flagged. This baseline anchors every reported share: any increase above 0.4% reflects a signal that cannot be explained by the detector’s inherent error rate.
"If a tool marks 40% of new papers as machine‑written but also marks 20% of papers written before ChatGPT, the real story is the 20% nobody mentioned." – author’s blog
Because the same 0.4% rate holds for the control period, the detector’s rise in later years cannot be an artifact of the model itself.
Dataset and scoring procedure
- Sample size: 12,750 papers (≈25 papers per field per month) from Jan 2023 – Jul 2026, plus eight control months in 2021‑2022.
- Fields: Ten broad groups (e.g., computer science, quantitative biology, economics & finance, etc.).
- Version handling: Only the first‑version PDF of each paper was used, preventing later revisions from contaminating earlier slots.
- Full‑text scoring: The detector evaluated the entire body text, not just abstracts, because abstracts under‑represent the AI‑like signal (some papers scored <20% on abstracts but >70% on bodies).
- Statistical confidence: Every share is reported with a bootstrap 95 % confidence interval.
Core results
| Field | Pre‑LLM control (0.4% baseline) | Recent flagged share (Jul 2026) | 95 % CI |
|---|---|---|---|
| Computer science | 0.2 % | 65.0 % | [59.3, 70.3] |
| Quantitative biology | 3.5 % | 56.3 % | [51.0, 61.7] |
| Electrical eng. & systems | 1.7 % | 51.3 % | [46.0, 57.0] |
| Economics & finance | 2.5 % | 47.0 % | [41.3, 52.7] |
| Applied physics | 1.3 % | 34.0 % | [29.0, 39.7] |
| Statistics | 1.8 % | 31.3 % | [26.0, 36.7] |
| Condensed matter | 0.0 % | 24.0 % | [19.3, 29.0] |
| High‑energy physics | 0.5 % | 14.0 % | [10.0, 18.0] |
| Astrophysics | 0.0 % | 10.7 % | [7.3, 14.3] |
| Mathematics | 0.0 % | 0.7 % | [0.0, 1.7] |
- The flagged share stays at the 0.4 % floor through 2021‑2022, then lifts off within months of ChatGPT’s release and climbs in two waves to about 32 % overall, peaking near 39 % in early 2026.
- Computer science leads with a flagged share of ~65 %, while mathematics remains near 0.7 %.
- The rise is not driven by fields with higher pre‑LLM control rates; fields with the strongest post‑2022 increase are those that were lowest initially.
Key limitations
1. Small control sample per field
Only 200 pre‑ChatGPT papers per field were used to estimate the 0.4 % floor. With such a low flag rate, the control counts are coarse (≈8 flagged papers across all fields). The pooled floor is reliable, but per‑field baselines are noisy.
2. Detector blind spots, especially in mathematics
Mathematics papers contain dense notation and sparse prose, which differs from the scientific English the detector was trained on. Consequently, a heavily AI‑assisted math paper may score low, making the 0.7 % figure a lower bound rather than definitive evidence of low adoption.
3. Incomplete coverage of generation models
The detector is more sensitive to some LLMs than others and cannot be evaluated against the exact, private mix of models and prompts authors use. This means the reported prevalence is a minimum estimate; the true share could be higher.
4. Flag ≠ authorship
A flag indicates that the text reads like machine‑generated language, not that a specific author used an LLM. The detector cannot distinguish light editing from wholesale generation, so the numbers reflect machine‑like writing, not proven AI authorship.
Community reactions and concerns
- False positives on older work: Users reported unexpectedly high scores on pre‑LLM papers (e.g., a 2011 workshop paper flagged at 27 %). This suggests the detector may sometimes misclassify older prose styles.
- Field‑specific skepticism: Several commenters doubt that 65 % of recent CS papers are substantially AI‑written, noting that boilerplate language could inflate scores.
- Impact on non‑native speakers: One comment highlighted that many non‑native authors rely on LLMs to polish English, which could cause legitimate human work to be flagged.
- Potential bias toward modern jargon: Concerns were raised that the detector might learn to associate newer terminology (e.g., “large language model”) with AI‑generated text, inflating post‑2022 rates.
- Tool reliability: Multiple users questioned the detector’s robustness across domains (e.g., Stack Overflow answers, LaTeX‑heavy texts) and noted that removing LaTeX markup dramatically changes scores.
How to try the detector yourself
The author provides a free, low‑cost web interface:
- Check any arXiv paper: https://unslop.run/check
- Test arbitrary text: https://unslop.run/app
Bottom line
By calibrating an AI‑text detector to a 0.4 % false‑positive rate on pre‑ChatGPT arXiv papers, the study reveals a rapid rise in machine‑like writing across scientific fields, especially computer science. While the numbers are lower bounds due to detector blind spots and limited coverage, they signal a substantial shift in how academic prose is produced post‑2022. The findings should be interpreted with the outlined limitations in mind, and the community is encouraged to use the publicly available tool to explore individual papers.