noahgolmant/pytorch-hessian-eigenthings

What it solves

Computing the full Hessian matrix (the second-order derivatives of a loss function) is computationally impossible for large neural networks because the memory requirements grow quadratically with the number of parameters. This library provides a scalable way to analyze the curvature of a model's loss landscape without ever explicitly constructing the full matrix.

How it works

The project uses Hessian-vector products (HVPs), which only require linear memory. By combining HVPs with iterative algorithms, it can compute specific properties of the Hessian and other curvature matrices (like the Generalized Gauss-Newton and empirical Fisher) without the memory bottleneck.

Key algorithms included are:

• Lanczos and stochastic power iteration for finding the top eigenvalues and eigenvectors.
• Hutch++ for estimating the trace of the matrix.
• Stochastic Lanczos Quadrature for estimating the spectral density.

For large language models, it includes optimized kernels (using Triton or torch.compile) to speed up the cross-entropy Hessian-vector products.

Who it’s for

Researchers and developers working on neural network optimization, generalization analysis, and those studying the geometry of the loss landscape (e.g., analyzing "flat minima").

Highlights

• Scalable Analysis: Computes eigendecomposition and spectral density for real-world models, including those from HuggingFace and TransformerLens.
• Flexible Operators: Supports Hessian, Generalized Gauss-Newton (GGN), and Empirical Fisher operators.
• Memory Efficient: Uses linear memory via HVPs instead of quadratic memory.
• Performance Optimizations: Features fused kernels for LM-scale work to reduce peak memory and increase speed.
• Parameter Filtering: Allows analysis of specific parameter subsets (e.g., specific blocks in a transformer) using name-based filters.