OpenAI Defining and Evaluating Political Bias in LLMs

OpenAI has introduced a comprehensive framework for defining and measuring political bias in large language models (LLMs) to ensure ChatGPT remains objective by default. The company reports that its latest models, GPT-5 instant and GPT-5 thinking, have reduced political bias by approximately 30% compared to previous models, with an estimated less than 0.01% of all production responses showing signs of bias.

A New Framework for Measuring Political Bias

OpenAI's approach shifts away from traditional multiple-choice benchmarks, such as the Political Compass test, which the company argues cover only a narrow slice of everyday use. Instead, OpenAI developed an evaluation that mirrors real-world usage through open-ended scenarios and nuanced interactions.

Evaluation Scope and Methodology

The evaluation focuses on text-based responses, excluding web search behavior as it involves separate retrieval systems. The process involves three primary steps:

  1. Representative Prompt Set: A dataset of approximately 500 prompts covering 100 topics (derived from U.S. party platforms and cultural issues). Each topic includes five variations: neutral, slightly liberal, slightly conservative, liberal charged, and conservative charged.
  2. Measurable Axes of Bias: OpenAI identified five specific ways bias manifests in model outputs:
    • User invalidation: Dismissing or delegitimizing the user's viewpoint (e.g., using scare quotes).
    • User escalation: Mirroring and amplifying the political stance of the prompt.
    • Personal political expression: Presenting political opinions as the model's own rather than as external viewpoints.
    • Asymmetric coverage: Selectively emphasizing one perspective or omitting others when multiple legitimate viewpoints exist.
    • Political refusals: Declining to engage with a political query without a valid justification under the Model Spec.
  3. Automated LLM Grader: An LLM grader (GPT-5 thinking) was used to score responses on a scale of 0 to 1 (where 0 is perfectly objective) based on detailed rubrics and reference responses.

Key Findings on Model Objectivity

OpenAI's evaluation reveals that while bias exists, it appears infrequently and at low severity across its model families.

Performance Across Model Generations

GPT-5 instant and GPT-5 thinking demonstrate the highest alignment with objectivity targets. These models reduced bias scores by roughly 30% compared to prior models, such as GPT-4o and o3.

Conditions for Bias Emergence

Models generally remain objective on neutral or slightly slanted prompts. However, moderate bias emerges when models are faced with challenging, emotionally charged prompts. OpenAI noted an asymmetry in this effect, stating that "strongly charged liberal prompts exert the largest pull on objectivity across model families, more so than charged conservative prompts."

Common Forms of Bias

When bias does occur, it most frequently manifests in three ways:

  • Personal opinion: Framing political views as the model's own.
  • Asymmetric coverage: Emphasizing one side of an issue where multiple perspectives are warranted.
  • Emotional escalation: Using language that amplifies the user's political slant.

Real-World Prevalence and Future Work

Analysis of a representative sample of production traffic indicates that less than 0.01% of all ChatGPT responses exhibit signs of political bias. This low rate is attributed to both the rarity of politically slanted queries and the general robustness of the models.

OpenAI is continuing to invest in improvements to further align models with its Model Spec, specifically targeting the emotionally charged prompts that are most likely to elicit biased responses.

Sources