OpenAI MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

OpenAI has introduced MLE-bench, a benchmark designed to measure the performance of AI agents in machine learning (ML) engineering. By utilizing 75 curated Kaggle competitions, the benchmark tests an agent's ability to train models, prepare datasets, and run experiments in real-world ML engineering scenarios.

Benchmark Design and Methodology

MLE-bench evaluates AI agents on their ability to solve complex ML engineering tasks. The benchmark consists of 75 ML engineering-related competitions sourced from Kaggle. These tasks are designed to test a diverse set of real-world skills, including:

  • Model Training: The ability to implement and train machine learning models.

  • Dataset Preparation: The ability to handle and prepare data for model training.

  • Experimentation: The run and manage experiments to iteratively improve model performance.

To establish a baseline for human performance, OpenAI uses Kaggle's publicly available leaderboards.

Performance Results

OpenAI evaluated several frontier language models using open-source agent scaffolds. The highest performing configuration—OpenAI's o1-preview model combined with AIDE scaffolding—reached the performance level of a Kaggle bronze medal in 16.9% of the competitions.

Research Focus and Resource Scaling

Beyond the primary performance metrics, the research accompanying MLE-bench MLE-bench explores several critical factors affecting agent performance:

  • Resource Scaling: The investigation into various forms of resource-scaling for AI agents to determine how additional resources impact their success rate.

  • Pre-training Contamination: The investigation into the impact of contamination from the models' pre-training data, which may influence the results.

Availability

To facilitate further research into the ML engineering capabilities of AI agents, OpenAI has open-sourced the benchmark code on GitHub.

Sources