f-dangel/backpack

What it solves

BackPACK simplifies the process of extracting additional information from the neural network training process. While calculating gradients is standard, researchers often need other metrics—such as individual gradients from a mini-batch or gradient variance—that are difficult to implement in standard PyTorch without significant overhead or manual effort.

How it works

Built as an extension for PyTorch, BackPACK modifies the backward pass of the training process. It reuses the information already being backpropagated to efficiently compute quantities beyond the standard gradient, such as second-order information (diagonal and Kronecker approximations) and second moments.

Who it’s for

Machine learning researchers and developers who need deeper insights into the gradient flow and training dynamics of their PyTorch models to implement advanced optimization or analysis tools.

Highlights

• Efficient computation: Reuses existing backpropagation data to minimize overhead.
• Extended metrics: Provides individual gradients per sample in a mini-batch.
• Detailed analysis: Computes estimates of gradient variance and second moments.
• Second-order information: Supports diagonal and Kronecker approximations for approximate second-order information.