Kaleidophon/deep-significance

deep-significance

What it is: A Python library that helps researchers and engineers properly compare the performance of deep learning models using statistical significance tests, instead of just comparing average scores.

Why it matters: Deep neural networks are highly sensitive to random seeds, hyperparameters, and other stochastic factors. A single run's score can be misleading. This package provides rigorous statistical methods to determine whether one model is actually better than another, or if the difference is just due to chance.

Key features:

• Almost Stochastic Order (ASO): A modern significance test that compares entire score distributions, not just means. It returns a confidence score (eps_min) indicating how strongly one model dominates another.
• Classic tests: Bootstrap and permutation-randomization tests for p-value based hypothesis testing.
• Multiple comparison correction: Bonferroni correction to handle testing across many datasets or model pairs.
• Sample size analysis: Tools to estimate how many runs you need for reliable conclusions.
• Compatibility: Works with NumPy, PyTorch, TensorFlow, and JAX arrays.

How it works (simple version):

• You provide two sets of scores (e.g., from running model A and model B multiple times).
• The aso() function compares the distributions and returns a value eps_min. If eps_min < 0.5, model A is likely better; the lower the value, the more confident you can be.
• For multiple models, multi_aso() produces a matrix of pairwise comparisons.

Use cases:

• Comparing a new model against a baseline across multiple random seeds.
• Evaluating models on multiple datasets while avoiding false positives.
• Comparing models based on per-sample scores (e.g., per-test-example predictions).
• Determining how many experimental runs are needed for a given confidence level.

Target audience: Machine learning researchers, practitioners, and anyone who needs to rigorously evaluate and compare neural network models.

Limitations (from README): The package does not guarantee superiority in all settings; it's a tool to support conclusions, not a magic bullet. It also assumes you have multiple runs or samples to compare.

Example usage:

from deepsig import aso
import numpy as np

# Simulate scores from two models
model_a_scores = np.random.normal(0.9, 0.8, 5)
model_b_scores = np.random.normal(0, 1, 5)

# Compare
eps_min = aso(model_a_scores, model_b_scores)
# If eps_min < 0.5, model A is better

In short: deep-significance gives you statistically sound ways to answer the question: "Is my model really better?"