OpenAI BrowseComp Benchmark Release
OpenAI has released BrowseComp, a new open-source benchmark consisting of 1,266 challenging problems designed to evaluate the ability of AI agents to locate hard-to-find, entangled information on the internet. This benchmark addresses the saturation of existing tools like SimpleQA, which measure basic isolated fact retrieval, by focusing on tasks that require persistence, strategic reasoning, and depth of browsing.
BrowseComp Design and Methodology
BrowseComp focuses on questions with short, single, indisputable answers that are easy to verify but difficult to find. This "asymmetry of verification" allows for simple, reliable grading while maintaining a high level of difficulty.
To ensure the benchmark remains challenging, OpenAI employed three specific verification checks during the creation process:
- Model Failure: Trainers verified that GPT-4o (with and without browsing), o1, and an early version of the deep research model could not solve the problem.
- Search Engine Resistance: Trainers confirmed that the answer was not available on the first pages of search engine results after five simple searches.
- Human Difficulty: Tasks were designed so that a human would struggle to solve them within ten minutes. Tasks solved more than 40% of the time by a second trainer were revised.
Questions were often created using an "inverted" approach: trainers started with a known fact (a seed) and created a question based on characteristics with a large search space. For example, a question might ask for a specific scientific paper based on the authors' undergraduate institutions and the conference year, requiring the agent to examine numerous papers and author backgrounds to find the match.
Human Performance and Dataset Difficulty
Human trainers, without the aid of AI assistants, attempted to solve the BrowseComp questions. The results highlight the extreme difficulty of the dataset:
- Success Rate: Trainers solved only 29.2% of the problems.
- Accuracy: Of the solved problems, the trainer's answer matched the reference answer 86.4% of the time.
- Time Investment: Many solvable problems required two to three hours of research. For unsolvable problems, most trainers gave up after approximately two hours of searching.
Model Performance Comparison
OpenAI evaluated several models on BrowseComp, demonstrating that basic browsing capabilities are insufficient for these tasks. The results show a significant gap between standard LLMs and specialized agent models:
| Model | Accuracy (%) |
|---|---|
| GPT-4o | 0.6 |
| GPT-4o w/ browsing | 1.9 |
| GPT-4.5 | 0.9 |
| OpenAI o1 | 9.9 |
| Deep Research* | 51.5 |
Note: The Deep Research model was trained on data specifically designed for BrowseComp tasks.
Key insights from these results include:
- Browsing vs. Reasoning: Enabling browsing for GPT-4o only marginally improved accuracy (0.6% to 1.9%), indicating that tool use alone is not enough.
- Internal Knowledge: OpenAI o1's higher accuracy (9.9%) despite lacking browsing capabilities suggests some answers can be found through inference over internal knowledge.
- Agentic Capability: The Deep Research model's 51.5% accuracy demonstrates the effectiveness of autonomous searching, synthesizing information from multiple sources, and adapting search strategies.
Scaling and Optimization Strategies
Test-Time Compute Scaling
Performance on BrowseComp scales smoothly as a function of the amount of test-time compute used. Because these tasks require iterative browsing and combining information, additional inference-time compute directly translates to better results.
Aggregation Strategies
OpenAI tested three methods for combining 64 sampled outputs per question to improve accuracy beyond a single attempt:
- Majority Voting: Selecting the most common answer.
- Weighted Voting: Weighting votes by a model-generated confidence score.
- Best-of-N: Selecting the output with the highest confidence score.
Best-of-N consistently achieved the highest accuracy, improving performance by 15% to 25% over single attempts. This suggests the Deep Research model can often identify when its own answer is correct.
Analysis of Task Difficulty
An analysis of pass rates for Deep Research and OpenAI o1 revealed a wide gap in task difficulty. Deep Research perfectly solved 16% of the tasks (100% pass rate) but failed entirely on 14% of the tasks (0% pass rate).
For the subset of questions where Deep Research failed, OpenAI found that providing the model with the ground-truth answer allowed it to locate supporting evidence. This confirms that the questions were not unsolvable, but rather required strategic perseverance and flexible search reformulation that the model could not achieve autonomously.