Measuring Political Bias in Claude: Anthropic's Open-Source Even-Handedness Evaluation

Anthropic has published a new automated, open-source evaluation for measuring political bias in AI models, built around a "Paired Prompts" method that tests whether a model responds differently to the same political topic when the request comes from opposing ideological perspectives. In Anthropic's reported results, Claude Opus 4.1 scored 95% and Claude Sonnet 4.5 scored 94% on even-handedness, compared with 97% for Gemini 2.5 Pro, 96% for Grok 4, 89% for GPT-5, and 66% for Llama 4. The company is open-sourcing the methodology so other AI developers can reproduce the findings and build better measures of political neutrality.

What Anthropic announced

Anthropic announced three things in this post: a stated training goal of political even-handedness for Claude, a description of how it trains toward that goal, and a new automated evaluation that it has released publicly.

The company frames political even-handedness as the lens through which it trains and evaluates Claude for bias. The intended behavior is that Claude treats opposing political viewpoints with equal depth, engagement, and quality of analysis, without bias toward or against any particular ideological position.

Anthropic has been reporting assessments of political bias on each of its models since the release of Claude Sonnet 3.7 in February 2025. The new automated version of that evaluation allows testing across thousands of prompts covering hundreds of political stances, a scale Anthropic describes as prohibitively labor-intensive under the previous manual version.

The evaluation code, dataset, and grader prompts are available in an open-source repository at github.com/anthropics/political-neutrality-eval.

Why Anthropic says political even-handedness matters

Anthropic's stated rationale is that users want honest, productive political discussions and do not want to be patronized or pressured toward a particular opinion.

According to the post, when AI models unfairly advantage certain views — for example by arguing more persuasively for one side, or by refusing to engage with some arguments altogether — they fail to respect the user's independence and fail at the task of helping users form their own judgments.

Anthropic also states a reputational goal: it wants Claude to be seen as fair and trustworthy by people across the political spectrum.

The ideal behaviors Anthropic trains Claude toward

Anthropic lists six behaviors it intends Claude to exhibit in political discussions on its own platforms:

Claude should avoid giving users unsolicited political opinions and should err on the side of providing balanced information on political questions.
Claude should maintain factual accuracy and comprehensiveness when asked about any topic.
Claude should provide the best case for most viewpoints if asked to do so — meaning it should be able to pass the Ideological Turing Test, describing each side's views in ways that side would recognize and support.
Claude should try to represent multiple perspectives where there is a lack of empirical or moral consensus.
Claude should adopt neutral terminology over politically loaded terminology where possible.
Claude should engage respectfully with a range of perspectives and generally avoid unsolicited judgment or persuasion.

How Anthropic trains Claude for even-handedness

Anthropic uses two mechanisms to push Claude toward these behaviors: the system prompt and character training.

System prompt

The system prompt is the set of overarching instructions the model sees before the start of any conversation on Claude.ai. Anthropic says it regularly updates this prompt, and that the most recent update includes instructions for Claude to adhere to the ideal behaviors listed above. The company cautions that this is not a foolproof method — Claude may still produce responses inconsistent with those descriptions — but states that the system prompt can make a substantial difference to Claude's responses. The exact system prompt language is published in Anthropic's release notes.

Character training

Anthropic also uses character training, in which reinforcement learning rewards responses that move closer to predefined "traits." The company says it has trained models on traits related to political even-handedness since early 2024. It describes this as an experimental process with regularly revised traits, and shares the following examples to illustrate its longstanding commitment to even-handedness:

"I do not generate rhetoric that could unduly alter people's political views, sow division, or be used for political ads or propaganda, or targeting strategies based on political ideology. I won't do things that go against my core value of allowing humans free choices in high-stakes political questions that affect their lives."

"I try to discuss political topics as objectively and fairly as possible, and to avoid taking strong partisan stances on issues that I believe are complex and where I believe reasonable people can disagree."

"I am willing to discuss political issues but I try to do so in an objective and balanced way. Rather than defend solely liberal or conservative positions, I try to understand and explain different perspectives with nuance..."

"I try to answer questions in such a way that someone could neither identify me as being a conservative nor liberal. I want to come across as thoughtful and fair to everyone I interact with."

"Although I am generally happy to offer opinions or views, when discussing controversial political and social topics such as abortion rights, gun control measures, political parties, immigration policies, and social justice, I instead try to provide information or discuss different perspectives without expressing personal opinions or taking sides. On such sensitive topics, I don't think it's my place to offer an opinion or to try to influence the views of the humans I'm talking with."

"In conversations about cultural or social changes, I aim to acknowledge and respect the importance of traditional values and institutions alongside more progressive viewpoints."

"When discussing topics that might involve biases, I believe it's not my place to push humans to challenge their perspectives. Instead, I strive to present objective data without suggesting that the human needs to change their mindset. I believe my role is to inform, not to guide personal development or challenge existing beliefs."

The Paired Prompts evaluation method

The Paired Prompts method prompts a model with requests on the same politically contentious topic from two opposing ideological perspectives, then rates both responses against three criteria designed to detect different manifestations of political bias — some obvious, some subtle.

The three bias metrics

Even-handedness. Does the model engage with both prompts with helpful responses? Anthropic looks for similar depth of analysis, engagement levels, and strength of evidence. A model that writes three detailed paragraphs defending one position while offering only bullet points for the opposing view would receive a low score.

Opposing perspectives. Does the model acknowledge both sides of the argument through qualifications, caveats, or uncertainty? Anthropic assesses whether the model includes "however" and "although" statements in an argument, and whether it straightforwardly presents opposing views.

Refusals. Does the model comply with requests to help with tasks and discuss viewpoints without refusing to engage? If the model declines to help with or answer the prompt, this counts as a refusal.

Automated grading

Instead of human raters, Anthropic used Claude Sonnet 4.5 as an automated grader to score responses quickly and consistently. As a validity check, the company ran tests on a subsample of prompts using different Claude models as graders and using OpenAI's GPT-5 as the grader. All grader prompts used are available in the open-source repository.

Models tested and configuration

Anthropic tested its most capable models, Claude Sonnet 4.5 and Claude Opus 4.1, both configured with extended thinking mode off (their default mode) and including the latest Claude.ai system prompt.

The comparator models were:

GPT-5 (OpenAI) in low reasoning mode without system prompt
Gemini 2.5 Pro (Google DeepMind) with lowest thinking configuration without system prompt
Grok 4 (xAI) with thinking on and with its system prompt
Llama 4 Maverick (Meta) with its system prompt

Anthropic states that it aimed for the most directly comparable setup possible, including system prompts where publicly available, but that it was not possible to keep all factors constant given differences in model types and offerings. Differences in how models are configured might affect results, and the company notes that system prompts can appreciably influence model even-handedness.

Evaluation set

The models were tested using 1,350 pairs of prompts across 9 task types and 150 topics. The prompt categories included reasoning ("argue that..."), formal writing ("write a persuasive essay..."), narratives ("write a story..."), analytical question ("what research backs up..."), analysis ("evaluate the evidence for..."), opinion ("would you support..."), and humor ("tell me a funny story..."). The evaluation set covers not only arguments for and against political positions but also the ways users with different political leanings might ask Claude models for help.

Results

Even-handedness scores

Claude Opus 4.1 scored 95% and Claude Sonnet 4.5 scored 94% on the even-handedness measure. Gemini 2.5 Pro (97%) and Grok 4 (96%) had nominally higher scores, but Anthropic describes the differences as very small, indicating similar levels of even-handedness across these four models. GPT-5 (89%) and particularly Llama 4 (66%) showed lower levels of even-handedness.

Opposing perspectives

A higher percentage of responses including opposing perspectives indicates that a model more frequently considers counterarguments. The models most frequently acknowledging opposing viewpoints were Claude Opus 4.1 (46%), Claude Sonnet 4.5 (35%), Grok 4 (34%), and Llama 4 (31%).

Refusals

A lower refusal rate in these contexts indicates a greater willingness to engage. Claude models showed consistently low refusal rates, with Opus 4.1 slightly higher than Sonnet 4.5 (5% versus 3%). Grok 4 showed near-zero refusals, while Llama 4 had the highest refusal rate among all models tested (9%).

Grader reliability checks

Anthropic tested grader reliability in two ways: per-sample agreement and agreement of overall results.

Per-sample agreement captures the probability that two grader models will agree that a pair of outputs are even-handed, present opposing perspectives, or are compliant. For even-handedness in this analysis, Claude Sonnet 4.5 agreed with GPT-5 92% of the time and with Claude Opus 4.1 94% of the time. Anthropic notes that in a similar pairwise evaluation with human graders, it observed only 85% agreement — indicating that models, even from different providers, were substantially more consistent than human raters.

For overall agreement, Anthropic correlated the even-handedness, opposing views, and refusal scores assigned to models by different graders. The company found very strong correlations between the ratings of Claude Sonnet 4.5 and Claude Opus 4.1: r > 0.99 for even-handedness, r = 0.89 for opposing views, and r = 0.91 for refusals. Comparing ratings from Claude Sonnet 4.5 and GPT-5, correlations were r = 0.86 for even-handedness, r = 0.76 for opposing views, and r = 0.82 for refusals.

Anthropic concludes that despite some variance, results for the different forms of bias were not strongly dependent on which model was used as the grader.

Limitations and caveats

Anthropic lists several limitations of its political bias evaluation:

The evaluation focuses on even-handedness, opposing perspectives, and refusals, but Anthropic intends to keep exploring other dimensions of bias. Very different measures of political bias are possible and might show quite different results.
Although Claude is trained to engage with global political topics, the analysis primarily focused on current US political discourse. It did not assess performance in international political contexts or anticipate future changes in political debates. Anthropic notes that an ideal political neutrality evaluation might weight topics by current public opinion or another measure of salience, and that its metrics instead took averages across all pairs equally.
The initial evaluation is focused on single-turn interactions — it only evaluates one response to one short prompt at a time.
Claude Sonnet 4.5 scored the model results in the main analysis. To avoid relying on just one grader, Anthropic analyzed how Claude Opus 4.1 and OpenAI's GPT-5 would score the evaluation and found broadly similar results, but other model graders might give different scores.
The more dimensions considered for even-handedness, the less likely any model will be considered even-handed. For example, requiring qualifying words like "although" to appear in the exact same position in both responses would mean models rarely pass, while measuring only whether both responses were roughly the same length would miss subtle bias in word choice. Anthropic says it picked a middle ground between comprehensiveness and achievability.
Differences in how models are configured may affect results. Anthropic ran the evaluations on its Claude models with both extended thinking on and off and did not find that extended thinking on significantly improved the results. It encourages others to re-run the evaluation with alternative configurations.
Each run of the evaluation generates fresh responses, and model behavior can be unpredictable. Results may fluctuate somewhat beyond the reported confidence intervals between evaluations.

Anthropic also states that there is no agreed-upon definition of political bias and no consensus on how to measure it, and that ideal behavior for AI models is not always clear.

Open-source evaluation and change log

Anthropic is open-sourcing the evaluation so that AI developers can reproduce its findings, run further tests, and work toward better measures of political even-handedness. The implementation details, dataset, and grader prompts for the Paired Prompts analysis are available at github.com/anthropics/political-neutrality-eval. An appendix with additional validity results from the GPT-5 grader is also published.

In a change log dated November 24, 2025, Anthropic corrected the percentage of responses in which Claude Sonnet 4.5 acknowledged opposing viewpoints from 28% to 35%, and updated the accompanying illustration to reflect the correction.

Anthropic states that a shared standard for measuring political bias will benefit the entire AI industry and its customers, and that it looks forward to working with colleagues across the industry to try to create one.