OpenAI HealthBench Release

OpenAI has introduced HealthBench, a new evaluation framework designed to measure the capabilities of AI systems in realistic health scenarios. Developed in partnership with 262 physicians across 60 countries, HealthBench provides a rigorous, physician-grounded method for assessing whether AI models can provide safe and useful health information.

HealthBench Dataset and Methodology

HealthBench consists of 5,000 realistic, multi-turn, and multilingual health conversations. These scenarios simulate interactions between AI models and either individual users or clinicians, spanning various medical specialties and contexts. The dataset was developed using a combination of synthetic generation and human adversarial testing to ensure the examples are sufficiently difficult and representative of real-world use.

Rubric-Based Evaluation

Unlike traditional exam-style benchmarks, HealthBench uses a rubric-based grading system. Each conversation is paired with a custom physician-created rubric containing specific criteria that a model response must meet or avoid.

• Scale and Detail: The benchmark contains 48,562 unique rubric criteria.
• Weighting: Each criterion is assigned a point value based on the physician's judgment of its importance.
• Grading Process: A model-based grader (GPT-4.1) assesses whether each rubric criterion is met, and the model receives an overall score based on the total points earned relative to the maximum possible score.

Organizational Structure

HealthBench organizes its evaluations into seven themes (such as emergency situations and global health) and various axes that define the aspect of model behavior being graded, such as accuracy, communication quality, or context seeking.

Model Performance and Baselines

OpenAI evaluated several generations of models, finding that recent frontier models have shown rapid improvement in performance, cost, and reliability.

Frontier Performance

Recent OpenAI models, specifically o3, outperform other frontier models including Claude 3.7 Sonnet and Gemini 2.5 Pro (March 2025). OpenAI reports that its frontier models have improved by 28% on HealthBench in recent months.

Cost and Accessibility

Newer models have shifted the performance-cost frontier. GPT-4.1 nano, for example, outperforms the August 2024 GPT-4o model while being 25x cheaper. The data suggests that reasoning models (o3, o4-mini, o1) continue to improve as test-time compute increases.

Reliability and Error Rates

To address the critical need for reliability in healthcare, OpenAI introduced two specialized variations of the benchmark:

• HealthBench Consensus: A subset of 3,671 examples with criteria multiply validated by physician consensus. o3 and GPT-4.1 show a significant reduction in error rates on this subset compared to GPT-4o.
• HealthBench Hard: A subset of 1,000 examples where current frontier models struggle, intended to provide a target for future development.

Comparison to Human Physician Baselines

OpenAI compared model responses against expert responses written by physicians to establish clinical baselines.

• September 2024 Models: Physicians with access to responses from o1-preview and GPT-4o were able to improve upon those responses, indicating that physicians could still add value over these models.
• April 2025 Models: In experiments with o3 and GPT-4.1, physicians with access to the model responses were no longer able to improve the quality of the responses, suggesting these newer models have reached a level of performance that matches or exceeds expert clinical judgment in these specific benchmark scenarios.

Trustworthiness and Validation

To validate the model-based grader, OpenAI conducted "meta-evaluations" by comparing the grader's results with physician reviews of responses in HealthBench Consensus. The findings indicated that the pairwise agreement between the model-based grader and physicians was similar to the agreement found between individual physicians themselves.

Availability

HealthBench, including the evaluation suite and underlying data, is openly available in OpenAI's GitHub repository to support the broader research community in developing AI systems that benefit human health.