lucidrains/ema-pytorch

A simple way to keep track of an Exponential Moving Average (EMA) version of your Pytorch model

What it solves

It provides a simple way to maintain an Exponential Moving Average (EMA) version of a PyTorch model. EMA is often used to improve model stability and performance by keeping a weighted average of model weights during training, rather than relying on the latest iteration.

How it works

The library wraps a PyTorch module and tracks a separate set of weights that are updated based on a decay factor (beta). It includes several advanced techniques:

  • Standard EMA: Updates a shadow copy of the model weights periodically.
  • Post-Hoc EMA: Implements a method to synthesize a new EMA model after training by using multiple checkpoints with different hyperparameters.
  • Switch EMA: Allows updating the main model with the EMA weights periodically to improve flatness and sharpness.
  • Target Representation Routing: Using EMAModuleWrapper, the library can route outputs from EMA submodules directly into the forward pass of other submodules, which is particularly useful for self-supervised learning (SSL) architectures.

Who it’s for

Machine learning engineers and researchers using PyTorch who want to implement weight averaging techniques to stabilize training or implement teacher-student frameworks in self-supervised learning.

Highlights

  • Simple wrapper for any PyTorch neural network.
  • Support for post-hoc synthesis of EMA models.
  • Integration of "Switch EMA" for periodic model updates.
  • Advanced routing for nested module trees in SSL setups.

Related

  • Project
    facebookresearch/multimodal

    A PyTorch library for training state-of-the-art multimodal models at scale, providing modular building blocks and pretrained weights for models like CLIP and BLIP-2.

  • Project
    naver/roma

    A lightweight PyTorch library for differentiable 3D rotation manipulation, providing mappings between rotation representations and utilities for rotation regression.

  • Project
    nschaetti/EchoTorch

    A PyTorch-based module for implementing and testing Echo State Networks and Reservoir Computing, designed for research purposes and integration into deep architectures.

  • Project
    open-mmlab/mmengine

    A foundational PyTorch library that serves as a training engine for deep learning models, providing standardized components for training loops, configuration, and monitoring.

  • Project
    sktime/pytorch-forecasting

    A PyTorch-based library for time series forecasting that provides a high-level API and state-of-the-art deep learning architectures like Temporal Fusion Transformers and N-BEATS.