OpenAI and the Ethics of Unpublished Mathematics

The Conflict: Intellectual Property in the Age of LLMs

Researchers are raising serious concerns about whether frontier AI labs, specifically OpenAI, can be trusted with unpublished mathematical research. The core of the controversy centers on the risk that proprietary ideas, shared with an AI for brainstorming or verification, are ingested into training sets or used by internal teams to scoop academic breakthroughs.

This debate was reignited by mathematician Andreas Thom, who detailed an exchange with OpenAI regarding the discovery of a non-sofic group. Thom noted that after discussing the expander matching problem and related extensions with ChatGPT, OpenAI announced a finding that mirrored his work. When questioned, OpenAI representatives provided what Thom describes as a categorical denial that the conversations influenced the result, but later statements suggested that "de-identified data derived from [user] usage" might have helped improve the models.

The "Opt-Out" Paradox and Data Ingestion

A critical point of contention is the effectiveness of privacy toggles and opt-out settings. While users can disable the "Improve the model for everyone" setting, several technical and procedural loopholes are highlighted by the community:

Temporal Gaps: Opting out of training typically applies to new conversations. Data shared prior to the opt-out remains in the training corpus.
Implicit Feedback: Actions such as clicking "thumbs up/down" on a response or selecting a preferred output may trigger data ingestion regardless of the general training opt-out.
De-identification: OpenAI has stated that it cannot rule out the use of de-identified data. Critics argue that in narrow, high-signal-to-noise domains like advanced mathematics, "de-identifying" a unique proof or a novel approach does not remove the intellectual value of the idea, effectively allowing the model to "scoop" the researcher while claiming no specific user data was accessed.
Backend Activations: Some argue that model activations—the internal mathematical representations triggered by a prompt—can be mapped and used to improve the model's latent representations without technically using the raw text as "training data."

The "Laundering" of Academic Attribution

Beyond the technicalities of training, there is a broader ethical concern regarding the "laundering" of information. If a model is trained on the unpublished work of Researcher A, and then Researcher B (or an internal OpenAI team) uses that model to solve a problem, the original contribution of Researcher A is stripped of attribution.

"OpenAI, as a middle man, has laundered information from B to A, stripping out attribution. A scooped B without even knowing it!"

This creates a scenario where the AI company acts as an unintentional (or intentional) conduit for intellectual theft, where the model's output is presented as a breakthrough of the AI or its developers, rather than a synthesis of unpublished human inputs.

Counter-Arguments: Brute Force vs. Intuition

Some observers argue that the suspicions of mathematicians "seeding" AI intuition are overestimated. They suggest that the current trajectory of AI—specifically the use of massive compute and Reinforcement Learning (RL) on verifiable domains like mathematics—allows models to discover superhuman techniques independently.

In this case, the argument is that mathematics is a fully verifiable domain amenable to self-play. A search agent with sufficient compute could potentially solve Millennium Prize problems (such as the Navier-Stokes existence and smoothness problem) through heuristic brute force and RL, making any specific user prompts irrelevant to the final breakthrough.

Summary of Risks for Researchers

For academics and researchers working on the frontier of science, the current consensus among critics is to assume a lack of total privacy. Recommended safeguards include:

Local LLMs: Running inference on personal hardware to ensure data never leaves the local environment.
Business Accounts: Utilizing enterprise tiers where terms of service explicitly forbid training on user data.
Strict Data Hygiene: Treating any prompt sent to a cloud-based LLM as public information, regardless of the most recent privacy settings.