soda-inria/tabicl

TabICLv2: A state-of-the-art tabular foundation model

What it solves

TabICL is designed to solve the challenge of tabular data classification and regression without the need for extensive hyperparameter tuning. It aims to provide a state-of-the-art alternative to traditional gradient-boosted decision trees (GBDTs) like XGBoost, CatBoost, or LightGBM, particularly for datasets ranging from 300 to 100,000 training samples.

How it works

TabICL is a tabular foundation model based on a Transformer architecture. It uses in-context learning (ICL), where the model performs a single forward pass through a pre-trained transformer to jointly fit and predict on new data. It is pre-trained on millions of synthetic datasets to acquire general learning capabilities. To improve performance on specific datasets, it also supports full PyTorch training loops for fine-tuning.

Who it’s for

It is intended for data scientists and ML engineers who want high-accuracy tabular models that are fast to deploy and scikit-learn compliant, as well as researchers interested in tabular foundation models and in-context learning.

Highlights

  • Zero-shot performance: Outperforms heavily tuned GBDTs on approximately 80% of TabArena datasets without tuning.
  • Scalability: Supports datasets with up to 100,000 training samples and 2,000 features, with CPU and disk offloading for even larger sets.
  • Speed: Up to 10x faster than TabPFN-2.5 on certain benchmarks, with KV caching for faster repeated inference.
  • Versatility: Supports classification, regression, and zero-shot time series forecasting.
  • Explainability: Integrated with SHAP for feature importance analysis.
  • Open Source: Includes pre-training code and synthetic data generation recipes.