Anthropic Introduces GlobalOpinionQA Framework to Measure Subjective Global Opinion Representation in LLMs

TL;DR

Anthropic unveiled GlobalOpinionQA, a benchmark and metric for quantifying how closely large language model (LLM) responses align with country‑specific public opinions, exposing a default bias toward US and some European viewpoints and highlighting the limits of prompting and translation as bias‑mitigation tools.

Dataset Construction: Capturing Cross‑National Survey Opinions

The authors compiled GlobalOpinionQA, a dataset of question‑answer pairs derived from established cross‑national surveys that probe public attitudes on a range of societal issues. Each entry includes the original survey question, the country‑specific human response distribution, and a translated version of the question for non‑English languages. The dataset is publicly released on Hugging Face (https://huggingface.co/datasets/Anthropic/llm_global_opinions) and accompanied by an interactive visualization (https://llmglobalvalues.anthropic.com/).

Metric Definition: Country‑Conditioned Opinion Similarity

To evaluate how well an LLM mirrors a given population, the paper defines a similarity metric that compares the model’s generated answer to the empirical distribution of human responses for each country. The metric operates as follows:

  1. Generate a response to a GlobalOpinionQA question using the target LLM.
  2. Encode the response into a probability distribution over the survey’s answer choices.
  3. Compute similarity (e.g., cosine similarity or KL divergence) between this distribution and the country‑specific human distribution.
  4. Aggregate scores across questions to obtain a per‑country similarity profile.

Experiment 1 – Baseline Bias in a Helpful‑Honest‑Harmless LLM

When the model (trained with Anthropic’s Constitutional AI approach) answered questions without any country cue, its responses were most similar to the opinions of respondents from the United States, as well as several European and South American nations. This baseline pattern indicates an inherent bias toward the perspectives of these regions, suggesting that the model’s training data or alignment process over‑represents them.

Experiment 2 – Prompt‑Based Perspective Shifting

The researchers introduced explicit prompts asking the model to adopt the viewpoint of a specified country (e.g., “Answer as if you were a citizen of Japan”). The prompts successfully shifted similarity scores toward the targeted population, demonstrating that the model can be steered to reflect different cultural outlooks. However, the shifted responses sometimes reproduced harmful cultural stereotypes, raising concerns about the safety of naïve prompting as a bias‑mitigation strategy.

Experiment 3 – Language Translation Effects

The team translated GlobalOpinionQA questions into the native language of each target country before querying the model. Contrary to expectations, translation did not consistently increase similarity to the opinions of speakers of those languages. This result suggests that merely providing input in a local language is insufficient to align model outputs with local public sentiment.

Implications for Model Alignment and Fairness

  • Bias Visibility – The framework makes it possible to surface systematic over‑representation of certain national viewpoints, a prerequisite for targeted remediation.
  • Prompt Engineering Limits – While prompting can re‑weight opinion similarity, it may also amplify stereotypical content, indicating a need for more robust alignment techniques beyond surface cues.
  • Language‑Centric Approaches Insufficient – Aligning models to local cultures likely requires deeper data diversification and representation learning, not just multilingual input handling.

Resources for the Community

Conclusion

Anthropic’s GlobalOpinionQA provides the first systematic, quantitative tool for measuring how LLMs represent subjective global opinions. The initial findings reveal a default US‑centric bias, limited efficacy of prompting and translation for bias correction, and the risk of reinforcing stereotypes. By releasing the dataset and metrics, Anthropic invites the research community to build more culturally balanced and safe language models.

Sources

Related