Hugging Face Ethics and Society Newsletter #4: Bias in Text-to-Image Models

Hugging Face has identified a critical need for improved evaluation methods to detect and address bias in text-to-image (TTI) models. Because these models are increasingly deployed in diverse sociotechnical contexts, they risk amplifying existing societal inequities and cultural stereotypes.

Sources of Bias in Text-to-Image Systems

Bias in TTI models is not the result of a single failure but emerges from multiple stages of the machine learning pipeline. Hugging Face identifies five primary sources:

  • Training Data: Popular multimodal datasets (such as LAION-5B, MS-COCO, and VQA v2.0) contain harmful associations and biases. For example, the Stable Bias project noted a lack of diversity in image generations and the perpetuation of identity group stereotypes, such as Dall-E 2 generations of CEOs and managers lacking diversity.
  • Pre-training Data Filtering: Filtering processes intended to clean datasets can inadvertently amplify biases. OpenAI reported that filtering training data for Dall-E 2 may have increased bias, potentially due to the existing dataset's tendency to represent women in sexualized contexts.
  • Inference (CLIP Model): Models like Stable Diffusion and Dall-E 2 rely on the CLIP model for prompt encoding. CLIP has documented biases regarding race, gender, and age, often treating "white, middle-aged, male" as the default when prompts are underspecified.
  • Latent Space: The internal structure of a model's latent space can reflect biases. While some work (e.g., Fair Diffusion) explores guiding generation along axes like gender to improve representation, more research is needed to understand how the latent space influences output.
  • Post-hoc Filtering: Built-in safety filters often lack robustness. Red-teaming of the Stable Diffusion safety filter revealed that while it effectively identifies sexual content, it often fails to flag violent, gory, or disturbing content.

Technical Approaches to Bias Detection

Addressing bias is a socio-technical challenge that cannot be solved by technology alone. Hugging Face suggests a combination of the following methods to gain insights into model limitations:

Exploration Tools

As part of the Stable Bias project, Hugging Face developed tools to visualize and compare biases across models:

  • Average Diffusion Faces: This tool computes average representations for specific professions (e.g., "janitor") to compare how different models (such as Stable Diffusion v1.4, v2, and Dall-E 2) visualize that role.
  • Face Clustering and Colorfulness Profession Explorer: These tools allow users to identify patterns and stereotypes in generated data without relying on pre-defined labels.

Red-Teaming

Red-teaming involves stress-testing models via prompting to uncover vulnerabilities and biases. Hugging Face notes that this process is currently ad hoc and lacks systematic benchmarks or leaderboards. Currently, the only major open-source resource for red-teaming prompts is Anthropic's dataset, which is limited to English natural language text.

Documentation and Evaluation

To standardize the reporting of bias, Hugging Face advocates for the use of model cards, datasheets, and READMEs. By sharing the results of red-teaming and exploration tools alongside model weights, creators can provide transparency regarding a model's known biases.

The Challenge of Value Alignment

Beyond detection, Hugging Face raises the ethical dilemma of whether models should simply mimic data or actively promote a specific version of an "ideal" society. This approach introduces the problem of "whose values" are being programmed, as values vary across cultures and individuals.

The impact of these biases also depends heavily on the downstream application:

  • Low-Risk: In human-in-the-loop settings like graphic design (e.g., RunwayML), users can manually correct bias by adjusting prompts.
  • High-Risk: In high-stakes environments, such as using Dall-E 2 to create police sketches for forensic artists, biases can reinforce dangerous racial and societal stereotypes.

Additional Ethics and Society Updates

  • Content Policy: Hugging Face updated its Content Policy to emphasize consent as a core value.
  • AI Accountability Policy: Hugging Face submitted a response to the NTIA request for comments on AI accountability, emphasizing the need for transparency, documentation, and open collaboration.

Sources