scikit-learn-contrib/MAPIE

A scikit-learn-compatible library for estimating prediction intervals and controlling risks, based on conformal predictions.

What it solves

MAPIE provides a way to quantify uncertainty and control risks in machine learning models. Instead of providing a single point prediction, it allows users to generate prediction intervals (for regression) or prediction sets (for classification) with theoretical guarantees, ensuring that the true value falls within the predicted range or set with a specified probability.

How it works

MAPIE uses techniques from Conformal Prediction and Distribution-Free Inference. It is model-agnostic, meaning it can be wrapped around any model from scikit-learn, TensorFlow, or PyTorch. By estimating uncertainty on a separate conformalization dataset, it transforms a standard model's output into a probabilistic guarantee.

Who it’s for

Data scientists and ML engineers who need reliable uncertainty quantification for regression, classification, time series, multi-label classification, and computer vision tasks like semantic segmentation.

Highlights

  • Model Agnostic: Works with any scikit-learn compatible model, including those from PyTorch and TensorFlow.
  • Theoretical Guarantees: Implements peer-reviewed algorithms with distribution-free guarantees under minimal assumptions.
  • Broad Application: Supports regression, classification, and time series, as well as complex tasks like image segmentation and LLM-as-Judge risk control.
  • Adaptive Methods: Includes new adaptive conformal prediction methods and exchangeability tests to verify application legitimacy.

Related

  • Project
    deel-ai/puncc

    A Python library for predictive uncertainty calibration and conformalization, providing rigorous uncertainty estimations for any machine learning model across various tasks.

  • Project
    ml-stat-Sustech/TorchCP

    A PyTorch-based toolbox for conformal prediction in deep learning, providing guaranteed uncertainty quantification for classification, regression, GNNs, and LLMs.

  • Project
    torch-uncertainty/torch-uncertainty

    A deep learning framework for uncertainty quantification that provides tools to make neural networks more reliable through uncertainty-aware training, evaluation, and post-processing.

  • Project
    sktime/skpro

    A Python library for supervised probabilistic prediction that enables users to predict intervals, quantiles, and distributions for tabular data and survival analysis.

  • Project
    pymc-devs/pymc

    PyMC is a Python package for Bayesian statistical modeling that uses MCMC and variational inference algorithms to estimate parameters in complex probabilistic models.