stanford-star/relbench

RelBench – Benchmarking Deep Learning on Relational Databases

What it is – RelBench is an open‑source benchmark suite that lets researchers and engineers train and evaluate machine‑learning models (especially graph neural networks) directly on relational databases. It provides a collection of real‑world relational datasets, a unified task API, and a public leaderboard.

Why it matters – Most ML work focuses on flat tables or single‑table prediction. Relational data (multiple linked tables) is common in industry but hard to feed into neural nets. RelBench standardises the problem, offering ready‑to‑load datasets, temporal train/val/test splits, and a common evaluation protocol so that new methods can be compared fairly.

Key components

• Python package (relbench) – Install via PyPI. It pulls datasets and tasks from the Hugging Face hub, returns a Database object (the raw tables) and a Task object (train/val/test label tables, metric calculators). Works with any ML stack; examples use PyTorch Geometric + PyTorch Frame.
• Datasets & tasks – Starts with a core set (e.g., rel-f1) and has grown to include SALT, RateBeer, arXiv, MIMIC‑IV, plus >70 CTU databases via the ReDeLEx integration and 4DBInfer tables. Each dataset ships as a self‑describing folder (manifest.yaml + parquet tables + task definitions).
• Leaderboard – Three boards (classification, regression, recommendation). Submissions are simple CSV files per task; the CLI python -m relbench.submit validates, packages, and opens a GitHub issue, with optional automated agent‑driven submissions.
• BYOD (Bring‑Your‑Own‑Data) – Users can create their own RelBench‑formatted folders and publish them to Hugging Face. No central registry is required; the library can load them directly via the org/repo[/subdir] address.
• Tutorials & examples – Colab notebooks for a quick start and a full GNN baseline, plus runnable scripts in the examples/ directory.

Typical workflow

import relbench

ds = relbench.load_dataset("rel-f1")          # pulls from HF hub
db = ds.get_db()                               # raw relational tables
task = ds.load_task("driver-position")        # e.g., an entity‑prediction task
train = task.get_table("train")
# train any model on `db` + `train`
pred = model.predict(db, test_entities)
score = task.evaluate(pred)                    # uses the task‑specific metric

The library also exposes ds.val_timestamp and ds.test_timestamp for temporal splits.

Who should use it – Researchers developing relational deep learning models, data scientists who want a plug‑and‑play benchmark for multi‑table prediction, and anyone interested in comparing GNN‑based or other relational ML approaches on a common set of real‑world tasks.

Where to find more – Project website, Hugging Face dataset hub, the arXiv papers (NeurIPS 2024 and ICLR 2026), and the GitHub repository for code, leaderboards, and contribution guides.