mapillary/inplace_abn

What it solves

In-Place Activated BatchNorm (InPlace-ABN) reduces the GPU memory required to train deep neural networks. In standard training, frameworks must store large buffers for both the input to Batch Normalization (BN) and the input to the subsequent Convolution layer to calculate gradients during the backward pass. InPlace-ABN eliminates the need to store the BN input buffer, potentially saving up to 50% of training memory.

How it works

It redefines the Batch Normalization and non-linear activation sequence as a single in-place operation. To avoid storing the input buffer, the backward pass of BN is rewritten to use its output, which is reconstructed from the subsequent layer's input by inverting the activation function. This requires a specific change to the parametrization of the BN scaling factor to ensure the transformation remains invertible.

Who it’s for

Researchers and developers training large-scale deep learning models (such as ResNet, ResNeXt, and Wide ResNet) who are constrained by GPU memory or wish to increase batch sizes to improve performance.

Highlights

• Significant Memory Savings: Reduces training memory usage by up to 50% in modern architectures.
• Improved Performance: Enables larger batch sizes, contributing to top-tier results on benchmarks like ImageNet, Cityscapes, and Mapillary Vistas.
• PyTorch Implementation: Provides a native C++/CUDA implementation for efficiency.
• Broad Architecture Support: Compatible with various models including ResNet, ResNeXt, DenseNet, and WideResNet.