DALL·E 2 Pre-training Mitigations

OpenAI has detailed the technical mitigations applied during the pre-training of DALL·E 2 to reduce harmful content, prevent the amplification of demographic biases, and eliminate the verbatim reproduction of training images.

Reducing Harmful Content via Active Learning

OpenAI used an iterative active learning process to improve classifiers designed to filter violent and sexual content from the training dataset. This process focused on two primary goals: reducing false positives and reducing false negatives.

Reducing False Positives

To minimize the frequency with which benign images were misclassified as harmful, OpenAI assigned human labels to images that the current model classified as positive. The classification threshold was tuned for nearly 100% recall but a high false-positive rate, ensuring that human labelers primarily encountered and corrected truly negative cases.

Reducing False Negatives

To find harmful images the model was missing, OpenAI employed a nearest neighbor search. The process involved:

  1. Running many-fold cross-validation to identify positive samples in the labeled dataset that the model frequently misclassified as negative.
  2. Scanning the large collection of unlabeled images for nearest neighbors of these misclassified samples within a perceptual feature space.
  3. Assigning human labels to the discovered images.

Mitigating Bias Amplification from Data Filtering

While filtering training data reduces explicit content, it can inadvertently create or amplify biases. OpenAI observed that the filtered model was more biased than the unfiltered model; for example, the prompt "a ceo" produced almost exclusively images of men in the filtered model, whereas the unfiltered model produced more women.

Causes of Bias Amplification

OpenAI hypothesizes that this amplification occurs because:

  • Women may be presented in more sexualized contexts in the original dataset, leading filters to remove more images of women than men.
  • The classifiers themselves may be biased due to implementation or class definition.

Measuring Filter-Induced Bias

To quantify this effect, OpenAI used the multimodal nature of the dataset to measure keyword frequencies in captions. Using Apache Spark, they compared the frequency of keywords (e.g., "woman", "man") across filtered and unfiltered datasets. They found, for instance, that the word "woman" was reduced by 14%, while "man" was reduced by only 6%.

Reweighting the Dataset

To prevent the filtered model from becoming more biased than the unfiltered model, OpenAI implemented a reweighting scheme to match the distribution of the filtered dataset to the unfiltered one.

  1. Classifier Training: A linear probe was trained on top of a small CLIP model to predict the probability that an image belonged to the unfiltered dataset ($P(\text{unfiltered}|\text{image})$).
  2. Weight Calculation: Each image was assigned a weight based on the ratio $P(\text{unfiltered}|\text{image}) / P(\text{filtered}|\text{image})$.
  3. Application: This weight was applied to the training loss of the image, effectively emulating the repetition of underrepresented samples.

After fine-tuning, the relative frequency reductions for "man" and "woman" shifted from 14% and 6% to -1% and 1%, respectively.

Eliminating Image Regurgitation

OpenAI identified a problem where predecessors to DALL·E 2 reproduced training images verbatim, particularly simple vector graphics with many near-duplicates in the training set. This "regurgitation" can lead to copyright and privacy concerns.

Deduplication via Clustering

To eliminate this, OpenAI deduplicated the dataset. To avoid the computational cost of comparing every image pair in a dataset of hundreds of millions, they used a clustering approach:

  • Clustering: The dataset was clustered, and deduplication was performed only within each cluster.
  • Multiple Clusterings: To account for duplicate pairs that might fall across cluster boundaries, OpenAI used five different random clusterings. This method discovered 97% of all duplicate pairs on a tested subset.

Impact on Performance and Regurgitation

Despite removing nearly 25% of the dataset, human evaluators slightly preferred the model trained on deduplicated data, suggesting that redundant images were hindering performance.

Following deduplication, a search over 50,000 prompts from the training dataset revealed that the model never regurgitated a training image, even when given the exact prompt associated with the image.

Sources