Evaluating Language Model Bias with 🤗 Evaluate

TL;DR

Hugging Face released new bias evaluation metrics—toxicity, language polarity (Regard), and HONEST—in the 🤗 Evaluate library, allowing researchers to quantify harmful or stereotypical outputs from causal language models such as GPT‑2 and BLOOM.

Overview of the New Bias Evaluation Workflow

The bias evaluation workflow consists of two steps:

  1. Prompting a causal language model (CLM) with a predefined set of prompts, which are hosted on 🤗 Datasets.
  2. Scoring the generated completions with a metric from 🤗 Evaluate. This approach works with any CLM that can generate free‑form text and does not depend on a specific prompt dataset.

Toxicity Measurement

Takeaway: A simple pronoun change can double the measured toxicity ratio, showing gender‑related bias in model completions.

  • Dataset used: Prompts extracted from the WinoBias dataset.
  • Model: GPT‑2 generates completions for male‑pronoun and female‑pronoun prompts.
  • Metric: The toxicity measurement, which wraps Facebook’s R4 hate‑speech classifier.
  • Result example:
    {"toxicity_ratio": 0.0}   // male prompts
{"toxicity_ratio": 0.333} // female prompts
    The female‑pronoun completions contain a higher proportion of toxic outputs.
  • Usage: Load the metric with evaluate.load("toxicity") and call compute(predictions=..., aggregation="ratio"). Omitting aggregation returns raw scores per completion (e.g., 0.0002 vs. 0.85).
  • Caveat: High toxicity scores may contain triggering language; users should handle such content responsibly.

Language Polarity (Regard) Measurement

Takeaway: Model completions for CEO prompts receive a more positive polarity than those for truck‑driver prompts, revealing profession‑based bias.

  • Dataset used: Subsets of the BOLD dataset, which contains prompts for various demographic groups.
  • Model: GPT‑2 generates completions for two profession groups (truck drivers vs. CEOs).
  • Metric: The regard measurement (loaded with evaluate.load("regard", "compare")). It returns a polarity distribution across negative, neutral, other, and positive.
  • Result example:
    {"negative": 0.14, "neutral": 0.29, "other": -0.11, "positive": -0.32}
    Positive scores are higher for CEO completions, indicating a more favorable view of that profession.
  • Interpretation: By computing the difference in regard scores between groups, practitioners can surface systematic favorability or disparagement toward specific identities.

Hurtful Sentence Completions (HONEST)

Takeaway: The HONEST metric flags more hurtful completions for lesbian prompts than for gay prompts, demonstrating gender‑and‑sexual‑orientation bias.

  • Dataset used: The HONEST prompt set (English templates for LGBTQAI+ groups).
  • Model: GPT‑2 generates completions for prompts targeting "lesbian" and "gay" groups.
  • Metric: The honest measurement, loaded with evaluate.load("honest", "en").
  • Result example:
    {"honest_score_per_group": {"lesbian": 0.333, "gay": 0.0}}
    Higher scores indicate more hurtful completions; the lesbian group receives a higher score.
  • Flexibility: Users can vary the top‑k sampling parameter to explore how many alternative completions affect the HONEST score, as shown in the original HONEST paper.

Discussion and Limitations

Takeaway: Existing bias datasets are limited in scope and often reduce complex identities to binary or categorical labels; multiple complementary metrics should be used together.

  • Hugging Face encourages the community to contribute additional datasets that capture under‑represented dimensions such as ability status and age.
  • The blog stresses that bias evaluations based on current datasets should not be treated as exhaustive truth; they provide partial perspectives on model behavior.
  • Combining toxicity, regard, and HONEST scores offers a more holistic view of a model’s appropriateness across different social axes.

Acknowledgements

The authors thank Federico Bianchi, Jwala Dhamala, Sam Gehman, Rahul Gupta, Suchin Gururangan, Varun Kumar, Kyle Lo, Debora Nozza, and Emily Sheng for their contributions to adding the datasets and evaluation scripts to the 🤗 Evaluate and Datasets libraries.

Sources