google/uncertainty-baselines

What it solves

Uncertainty Baselines provides a standardized template and high-quality implementations of state-of-the-art methods for researchers focusing on uncertainty and robustness in deep learning. It addresses the problem of fragmented, one-off experimental codebases that make it difficult for researchers to quickly prototype new ideas or accurately compare results across different projects due to inconsistent architectures, hyperparameters, and preprocessing.

How it works

The project is organized into three main components:

• Baselines: A collection of scripts organized by training dataset (e.g., CIFAR-10) that implement standard and state-of-the-art uncertainty methods. These are designed to be minimal and easily forkable to reduce dependencies.
• Datasets: A module (ub.datasets) that wraps TensorFlow Datasets (TFDS) to provide default data preprocessing and easy loading for TensorFlow, Jax, and PyTorch.
• Models: A module (ub.models) containing model architectures following the tf.keras.Model API.

Researchers can launch these baselines using TPUs (via Colab or Google Cloud) or GPUs, adjusting flags to control the number of cores and hardware accelerator type.

Who it’s for

AI researchers and developers specializing in uncertainty quantification, robustness benchmarking, and out-of-distribution detection in deep learning.

Highlights

• Standardized Benchmarking: Prescribes best practices for uncertainty and robustness evaluation.
• High-Quality Implementations: Provides a library of state-of-the-art methods on standard tasks.
• Detailed Metrics: Includes built-in support for measuring Test Accuracy, Expected Calibration Error (ECE), Negative Log-Likelihood (NLL), and precise training/test runtimes.
• Hardware Flexibility: Native support for TPU and GPU training with clear configuration paths.
• TFDS Integration: Seamless integration with TensorFlow Datasets for consistent data handling.