Measuring AI‑Written Text on arXiv: Method, Findings, and Limitations

Quick takeaway

A detector calibrated to flag only 0.4% of pre‑ChatGPT arXiv papers as machine‑written flags about 32% of new papers overall – up to 39% in early 2026 – with computer science showing the highest share (≈65%) and mathematics the lowest (≈0.7%).

How the measurement was anchored

The study set a false‑positive floor of 0.4% by treating all 2021‑2022 arXiv submissions as genuine human text and adjusting the detector threshold until exactly 0.4% of those papers were flagged. This floor provides a built‑in control: if the detector were simply drifting over time, the pre‑ChatGPT years would show the same flag rate as later years, which they do not.

"We took papers submitted in 2021 and 2022, before ChatGPT, treated them as ground‑truth human, and set the flag threshold so that exactly 0.4% of them trip it. That line is the floor."

Data collection and scoring

  • Sample size: 12,750 papers (≈25 papers per field per month) from Jan 2023 – Jul 2026, plus eight control months in 2021‑2022.
  • Fields: Ten broad groups (e.g., computer science, quantitative biology, economics & finance, etc.).
  • Versioning: Only the version‑1 PDF of each paper was used to avoid contamination from later revisions.
  • Scoring: The detector evaluated the full body text rather than abstracts, because abstracts under‑represent the AI‑like signal.
  • Uncertainty: All reported shares include bootstrap 95 % confidence intervals.

Main results

Field Pre‑LLM control (0.4% floor) Recent flagged share (Jul 2026) 95 % CI
Computer science 0.2 % 65.0 % [59.3, 70.3]
Quantitative biology 3.5 % 56.3 % [51.0, 61.7]
Electrical eng. & systems 1.7 % 51.3 % [46.0, 57.0]
Economics & finance 2.5 % 47.0 % [41.3, 52.7]
Applied physics 1.3 % 34.0 % [29.0, 39.7]
Statistics 1.8 % 31.3 % [26.0, 36.7]
Condensed matter 0.0 % 24.0 % [19.3, 29.0]
High‑energy physics 0.5 % 14.0 % [10.0, 18.0]
Astrophysics 0.0 % 10.7 % [7.3, 14.3]
Mathematics 0.0 % 0.7 % [0.0, 1.7]
  • The flagged share stays at 0.4 % through 2021‑2022, rises sharply after ChatGPT’s release, and climbs in two waves to a peak of ≈39 % in early 2026.
  • Computer science leads the surge, while mathematics remains near the floor, likely due to detector blind spots (see limitations).

Key limitations

1. Small control sample per field

Only ~200 pre‑ChatGPT papers per field were used, yielding about eight flagged papers total across all fields. This makes per‑field baseline rates noisy, though the pooled 0.4 % floor is well‑estimated.

2. Detector blind spots in low‑prose domains

Fields dominated by formulas and terse theorem‑proof structures (e.g., mathematics) produce sparse prose that differs from the detector’s training data. Consequently, heavily AI‑assisted math papers may score low, so a low flag rate does not prove low AI usage.

3. Incomplete coverage of generation models

The detector is more sensitive to some LLMs than others and cannot be evaluated against the exact, proprietary mix of models authors employ. Hence, the reported prevalence is a lower bound on AI‑like writing.

4. Flag ≠ authorship

A flag indicates that the text reads like machine‑generated language; it does not differentiate between light editing and wholesale generation, nor can it attribute authorship.

Community reactions

  • Skepticism about false positives: Users reported high scores on pre‑2020 papers they authored, suggesting the detector may over‑flag certain writing styles or LaTeX artifacts.

    "I uploaded a 2011 workshop paper and got 27 % machine. My 2012 dissertation returned 40 %…"

  • Field‑specific concerns: Several commenters noted that mathematics’ low score could stem from the detector’s inability to handle notation‑heavy prose.

    "Mathematics papers are dominated by notation… a mathematics paper drafted with heavy model assistance may score low because its prose is out of distribution."

  • Broader implications: Some participants argued that AI‑assisted writing may inflate publication volume without necessarily improving scientific quality, while others see it as a productivity boost for non‑native speakers.

    "LLMs may unlock researchers who hate writing, leading to more papers, but also risk more low‑quality work."

  • Methodological doubts: A few users questioned whether the detector is simply learning post‑2022 jargon (e.g., “large language model”) rather than true AI‑generated patterns.

    "Could the detector be flagging newer terminology as AI‑like?"

How to try the detector yourself

The author provides a free web interface:

Bottom line: By anchoring a detector to a 0.4 % false‑positive rate on pre‑ChatGPT literature, the study reveals a rapid rise in AI‑like writing on arXiv, reaching roughly one‑third of recent papers. The results are robust at the aggregate level but must be interpreted cautiously for individual fields—especially those with sparse prose—because of control‑sample size, detector blind spots, and incomplete model coverage.

Sources

Related