Weight Normalization: Accelerating Deep Neural Network Training

TL;DR

Weight normalization is a reparameterization of neural network weight vectors that decouples their length from their direction. This technique improves the conditioning of the optimization problem, speeding up the convergence of stochastic gradient descent (SGD) while avoiding the minibatch dependencies inherent in batch normalization.

The Core Mechanism: Weight Reparameterization

Weight normalization accelerates training by separating the magnitude of a weight vector from its direction. By decoupling these two components, the optimization process becomes more efficient, improving the conditioning of the optimization problem and allowing for faster convergence during training.

Advantages Over Batch Normalization

While inspired by batch normalization, weight normalization offers several distinct technical advantages:

  • No Minibatch Dependencies: Unlike batch normalization, weight normalization does not introduce dependencies between examples within a minibatch. This makes it a more stable and flexible alternative for specific architectures.
  • Lower Computational Overhead: The method is simpler to implement and requires less computational power, which allows for more optimization steps to be performed within the same timeframe.
  • Broad Applicability: Because it lacks minibatch dependencies, weight normalization can be successfully applied to models and applications where batch normalization is less suited, including:
    • Recurrent models such as Long Short-Term Memory (LSTM) networks.
    • Noise-sensitive applications, such as generative models.
    • Deep reinforcement learning.

Demonstrated Applications

OpenAI has demonstrated the effectiveness of weight normalization across three primary domains of machine learning:

  1. Supervised Image Recognition: Improving the speed and stability of training for standard classification tasks.
  2. Generative Modelling: Enhancing the stability of models that create new data samples.
  3. Deep Reinforcement Learning: Accelerating the training of agents in complex environments.

Sources