OpenAI Introduces GeneBench-Pro for Computational Biology Reasoning

OpenAI has introduced GeneBench-Pro, a benchmark designed to test whether AI models can handle the judgment-heavy, iterative analysis required for real-world computational biology. Unlike traditional benchmarks that test fact recall or predefined workflows, GeneBench-Pro measures "research taste"—the ability to handle ambiguity, revise assumptions, and choose the correct analytical path to reach a decision-ready result.

Measuring "Research Taste" and Scientific Judgment

GeneBench-Pro focuses on the system-level judgment calls that constrain AI performance in scientific research. The benchmark defines "research taste" as the chain of judgment calls that shape an analysis, including determining which questions the data can support and knowing when to revise an initial plan based on early diagnostics.

To test these capabilities, each problem provides the model with:

• A realistic and messy dataset.
• Brief experimental context.
• A target estimand tied to a downstream decision.

Models must explore the data, select an appropriate analytical approach, and engage in an iterative process of experimentation to supply a final answer.

Dataset Construction and Domain Coverage

GeneBench-Pro consists of 129 questions spanning 10 domains and 21 sub-domains across genomics, quantitative biology, and translational medicine. To avoid common benchmark failures—such as arbitrary author preferences or numerical insensitivity—OpenAI utilized synthetic data generation.

By simulating the data-generating process and knowing the full causal structure, OpenAI ensured that:

• Reasonable differences in subjective analytical choices still produce accepted numerical results.
• Plausible but incorrect analyses fail.
• Information leakage and unintended solution pathways are minimized through detailed trace analyses.

Additionally, 82 of the 129 questions were audited by external domain experts (including professors, industry scientists, and postdoctoral researchers) to verify realism, identifiability of the target answer, and the appropriateness of methods.

Evaluation Framework and Grading

Agents are evaluated in an isolated workspace equipped with a standard bioinformatics stack, including Python, scientific computing libraries, and basic genomics packages like PLINK 2.0.

Because the data is synthetically generated, correctness is graded deterministically against known targets. This removes the variability and verbosity effects often associated with rubric-based evaluations.

OpenAI is open-sourcing 10 representative questions on Hugging Face and will provide a 50-question subset to Artificial Analysis for independent third-party benchmarking.

Performance Results and Model Comparisons

GPT-5.6 Sol is the strongest performing model on the benchmark, achieving a pass rate of 31.5% with Pro mode enabled (28.7% at the highest reasoning level). This represents a significant increase over the original GeneBench, where the best frontier model, GPT-5, scored below 5%.

Key findings from the results include:

• Scaling Test-Time Compute: At the highest reasoning level, GPT-5.6 Sol solves nearly six times as many questions as GPT-5.2 while using approximately two-thirds as many tokens.
• Model Family Gaps: GPT models outperformed leading open-source models like GLM 5.2. OpenAI notes that the performance gap in high-level scientific reasoning is significantly larger than the gap seen in coding benchmarks, suggesting open-source models may be more specialized for coding than for general reasoning.
• Comparison to Human Labor: Reviewers estimated that a typical GeneBench-Pro problem would take a human expert 20–40 hours to complete. While AI agents are currently too unreliable to replace experts, the inference cost (several dollars) is orders of magnitude lower than the estimated human labor cost (thousands of dollars).

Implications for Scientific Discovery

While frontier models still solve fewer than one-third of the problems, they demonstrate the ability to make partial progress on challenging tasks. OpenAI suggests that the current failure pattern—making observations but struggling to integrate them into a broader context—mirrors the difference between human novices and experts.

If AI agents can reliably automate these analyses, it could transform industrial research by accelerating hypothesis triage, target prioritization, and the iteration cycle between data generation and decision-making, shifting the bottleneck of scientific discovery from data generation to the extraction of actionable insights.