stanford-star/relbench

What it solves

RelBench provides a standardized benchmark for predictive machine learning on relational databases. It addresses the challenge of applying deep learning to structured, multi-table data without requiring per-dataset custom code for each single task.

How it works

RelBench uses a self-describing format for datasets and tasks. It allows users to load datasets and tasks directly from the Hugging Face Hub or local paths using a unified API. The framework is agnostic to the modeling stack, meaning users can use any ML model (such as Graph Neural Networks) to train on the database and predict outcomes on test entities. It handles temporal splits (train/val/test) to ensure predictions are made based on data available before a specific timestamp.

Who it’s for

It is designed for ML researchers and practitioners who are developing predictive models for relational data, specifically those working on classification, regression, and recommendation tasks.

Highlights

• Unified API: Load datasets and tasks with no per-dataset code required.
• Extensive Library: Includes a wide range of databases (e.g., SALT, RateBeer, arXiv, MIMIC-IV) and dozens of tasks across classification, regression, and recommendation.
• Hugging Face Integration: Datasets and tasks are hosted on the Hugging Face Hub for easy access and loadable via relbench.load_dataset.
• BYOD Support: Users can easily package their own relational databases and tasks in the RelBench format and publish them to the Hub.
• Leaderboard: A centralized system for ranking methods based on test-set performance across multiple tasks.