OpenAI Accused of Training on User Conversations to Claim Technical Breakthroughs

OpenAI Faces Allegations of Training on User Conversations for Breakthroughs

Researchers have accused OpenAI of utilizing private user conversations as training data to achieve technical breakthroughs, then presenting those results as original discoveries. The controversy centers on claims that OpenAI may have "stolen" proofs or solutions from academic researchers who interacted with the model, effectively using the user's intellectual property to claim a breakthrough.

The Conflict Between User Opt-Outs and Data Leakage

A central point of contention is whether users are responsible for opting out of training or whether AI labs have a moral and technical obligation to prevent data leakage in their training sets.

The "Opt-Out" Argument

Some observers argue that users are responsible for their own data security. In ChatGPT, the "Improve the model for everyone" setting is enabled by default, allowing OpenAI to use content for training. Critics of the researchers argue that if a user does not explicitly opt out of this setting, they have effectively consented to their data being used.

The "Leakage" Argument

Other experts argue that the burden of proof should lie with the AI lab. They contend that OpenAI should actively check for data leakage—where specific, unique solutions or proofs are memorized and regurgitated—before claiming a breakthrough.

"OpenAI can and should check their training data for leakage in the face of big breakthroughs like these. It’s the burden of the author to appropriately cite their sources."

Technical Nuances of Training on User Data

Not all uses of user data are equal in terms of risk. Former OpenAI researcher John Schulman has highlighted different degrees of "training on conversations," each with varying levels of regurgitation risk:

Pretraining on user data: Using user tokens as prediction targets. This carries a high risk of regurgitation (memorization) of the original text.
Distillation: Using user prompts to distill large models into smaller ones. This has a lower risk of regurgitation.
RL (Reinforcement Learning) Tasks: Using user traces to construct RL tasks. This generally has low memorization abilities but can still extract customer IP depending on the implementation.

Operational Security (OpSec) for AI Users

Given the lack of transparency regarding how data is handled, technical users and researchers have suggested several "common sense opsec" measures to protect intellectual property:

Explicit Opt-Out: Manually disable training settings in the account data controls.
Avoid Feedback Buttons: Using the "thumbs up/down" feedback buttons often triggers the collection and retention of the entire conversation for training purposes.
Use Temporary Chats: Utilize "incognito" or temporary chat modes available in ChatGPT and Claude, which are designed to be excluded from training.
Be Wary of Safety Classifiers: Some providers, such as Anthropic, may retain and train on chats that trigger safety classifiers, even if the user has opted out of general training.

Proposed Solutions for Transparency

To resolve these disputes, some community members suggest adopting standardized decontamination methods. One proposal is the use of "canary GUIDs" (similar to those used in BIG-bench) and the public disclosure of n-gram thresholds (e.g., the 13-gram threshold used for GPT-3) to prove that a model has not memorized specific training samples.