openml/automlbenchmark

What it solves

It addresses the lack of standardized benchmarking suites and the inconsistency in problem definitions (such as hyperparameter search spaces and time budgets) that make it difficult to reproduce research and compare the performance of different Automatic Machine Learning (AutoML) systems.

How it works

The project provides a framework that sets up standardized environments for experimentation. It includes curated suites of regression and classification datasets from OpenML and provides the code necessary to benchmark various popular AutoML systems. The system is extensible, allowing users to add their own AutoML frameworks and datasets.

Who it’s for

Researchers and developers building or evaluating AutoML systems who need a rigorous, reproducible, and standardized way to compare their tools against existing frameworks.

Highlights

• Curated benchmarking datasets for regression and classification from OpenML.
• Support for multiple popular AutoML systems.
• Extensible architecture for adding new frameworks and datasets.
• Flexible execution options via Docker or Singularity containers, locally or on AWS.