autogluon/tabarena

What it solves

TabArena provides a reliable, standardized benchmarking system for machine learning models and systems designed for tabular data. It addresses the lack ofconsistency in how tabular ML models are tuned and evaluated, ensuring that methods are tested at their peak potential using best practices like cross-validated ensembles and strong hyperparameter search spaces.

How it works

The project implements a single codebase that powers two distinct benchmarks:

• TabArena-v0.1: A living benchmark focused on curated, Independent and Identically Distributed (IID) tabular datasets.
• BeyondArena: A more comprehensive benchmark that tests generalization beyond IID, covering temporal and grouped tasks across various dataset sizes and feature dimensionalities.

It distinguishes between models (single methods tuned by the arena's shared protocol) and systems (complete pipelines like AutoML frameworks that handle their own tuning and ensembling).

Who it’s for

• ML Researchers: Those developing new tabular ML models or foundation models to see how they compare against existing state-of-the-art methods.
• AutoML Developers: Creators of AutoML systems (like AutoGluon) that need a rigorous stress-test for their pipelines.
• Data Scientists: Users who want to analyze metadata or use TabArena-tuned models on their own data.

Highlights

• Comprehensive Coverage: Includes 51 curated IID datasets and extends to 142 datasets for BeyondArena.
• Standardized Protocol: Implements early stopping, model refitting, parallel bagging, and memory usage estimation.
• Detailed Artifacts: Caches predictions, results, and leaderboards to allow for reproduction and post-hoc analysis without re-running full benchmarks.
• Flexible Installation: Offers different installation paths for evaluation-only, core benchmarking, or extended model sets.