OpenAI shares a playbook for trustworthy third‑party evaluations of frontier AI models

OpenAI released a shared playbook that outlines how third‑party evaluators should design evaluations of frontier models so that results are valid and interpretable.

The role of the harness in evaluation

The harness— the surrounding setup that provides tools, state tracking, and error recovery— can change key aspects of a system’s performance, including how it uses tools, keeps track of information, or recovers from mistakes. When models can act over many steps, the harness can determine whether a capability appears in the evaluation at all. A harness that preserves state and retries failed actions may let a model finish a multi‑step task that the same model never completes in a simpler harness.

Three types of claims evaluated

Evaluations typically test one of three kinds of claims:

• Capability elicitation: Can a model plausibly produce the capability being evaluated?
• Safeguard performance: How robust are the tested safeguards against the behavior or attack being evaluated?
• Comparison: How do different models perform under equivalent conditions?

Validity hazards to check

Trustworthy reports must examine factors that can distort results:

• Reward hacking: Exploiting shortcuts in the task or scorer to gain credit without demonstrating the intended behavior.
• Refusals: Refusing in ways that obscure the behavior being tested.
• Contamination: Overperforming because evaluation tasks, answers, or close variants appeared in training data or were discoverable during the evaluation.
• Broken problems: Underperforming because tasks are invalid (e.g., unfair scoring, missing files, unsolvable environments).
• Sandbagging: Deliberately underperforming when the model shows awareness of being evaluated.

Recommendations for trustworthy reports

Evaluation reports should explicitly state the claim being tested and share evidence that the result is valid. They must describe the harness choices that shaped the result, detail when those choices change between evaluations, and include supporting evidence such as tool setup, elicitation guidance, budget, and why the setup is a credible proxy for the claimed capability.

How OpenAI supports stronger evaluations

OpenAI is taking several concrete steps:

• Sharing specific maximum‑elicitation guidance with evaluators.
• Asking capability evaluators to use Codex as a common floor for OpenAI models, ensuring tests run through the same agentic interface users rely on.
• Making reasoning traces and other intermediate artifacts available where needed to assess deception, sandbagging, or evaluation awareness.
• Prioritizing research to understand when and how harness choices materially change results, from context management and tool access to retry behavior, scoring, and resource budgets.

Implications for standards and future research

The playbook is intended to inform emerging national and international standards for frontier AI evaluation. Standards should require enough detail for decision makers to understand:

• The claim being made (capability ceiling, system comparison, or safeguard robustness).
• Evaluation content: tasks or task distribution that reveal the skills, behaviors, or failure modes under test.
• The tested system: model, reasoning setting, tool access, harness, and safeguards.
• The budget: turns, tokens, attempts/retries, wall‑clock time, inference cost, and where applicable expected cost per successful solve.
• Elicitation methods: harness choices used to draw out the result and how closely the tested setup reflects the broader claim.
• Validity checks: how assessors looked for reward hacking, evaluation awareness, contamination, refusals, sandbagging, and other behaviors that could undermine the result, including how confirmed cases affected scoring or interpretation.

By making harness choices and validity checks explicit, evaluation reports can avoid understating a system’s capabilities or overstating confidence in safety claims, and they provide a foundation for continued research into robust elicitation methods.