OpenAI BioBasicsBench: Measuring AI Agent Performance in Wet-Lab Science

Overview of BioBasicsBench

BioBasicsBench is a new benchmark designed to measure the ability of AI agents to perform routine wet-lab analysis, such as interpreting raw experimental data, identifying errors, and recognizing when evidence is insufficient to support a conclusion. Developed in collaboration with Ligo, a biotech company specializing in enzyme design, the benchmark addresses the limitations of existing datasets that often rely on clean, published figures rather than the ambiguous, messy reality of daily laboratory work.

Benchmark Structure and Methodology

BioBasicsBench consists of 100 tasks spanning five core areas of wet-lab science: protein methods, imaging and cytometry, quantification and assays, nucleic acid methods, and molecular cloning. The tasks utilize raw experimental evidence, including:

  • Visual Data: Gel and blot images, plate photographs, microscopy stacks, and videos of bench work.
  • Instrument Data: Raw exports from plate readers, flow cytometry (FCS) files, and instrument screens.
  • Documentation: Protocols, lab notebooks, handwritten notes, and sequencing files.

Each task is paired with expert-written grading criteria. The tasks were subjected to a rigorous multi-stage review process involving internal Ligo scientists, external PhD-level wet-lab researchers, and final internal verification to ensure accuracy and relevance.

Performance Results

OpenAI evaluated seven models across 37 reasoning-effort settings, totaling 18,500 results. The results demonstrate that stronger native multimodal capabilities correlate with improved performance in lab-based tasks.

At the 'xhigh' reasoning-effort setting, GPT-6 Astra achieved a mean reward of 66.7%, outperforming GPT-5.6 Sol, which scored 43.8%. While these results show progress in handling routine analysis, the data indicates that models still struggle with complex visual interpretation, control verification, and maintaining experimental context.

Key Capabilities and Limitations

Advancements in Reasoning

GPT-6 Astra shows improved performance in:

  • Discerning Data Limits: Better ability to recognize when evidence is insufficient to draw a firm conclusion, avoiding over-interpretation of ambiguous signals.
  • Multimodal Integration: Enhanced capacity to synthesize information from diverse sources, such as correlating plate labels with experimental protocols to identify faulty controls.
  • Context Maintenance: Improved ability to interpret experimental context, such as recognizing z-stacks in microscopy images rather than misinterpreting focus changes as biological phenomena.

Remaining Challenges

Despite these gains, models still exhibit significant failures in reliable lab assistance:

  • Assumption Bias: Models may allow familiar assumptions to override clear evidence, such as misidentifying colonies on a PCR gel because the results did not match expected patterns.
  • Confirmation Bias: Models sometimes discount contradictory data points that conflict with their preferred interpretation of an image or assay.
  • Precision Errors: In tasks like colony counting, models struggle with spatial resolution, frequently missing colonies, counting empty agar, or failing to distinguish between closely spaced samples.

Implications for Biological Research

BioBasicsBench serves as a tool to drive improvements in AI reliability for laboratory settings. By focusing on the routine, error-prone tasks that occupy much of a scientist's time, the benchmark aims to help developers build agents that can reliably assist in experimental workflows. Furthermore, tracking these capabilities is essential for understanding the broader implications of AI in biology, including critical considerations for biosafety and biosecurity.

Sources