facebookresearch/optimizers

For optimization algorithm research and development.

What it solves

It provides a collection of advanced PyTorch optimization algorithms designed for high-performance machine learning training, specifically focusing on distributed environments and efficient gradient averaging.

How it works

The repository implements specialized optimizers that can be integrated into PyTorch workflows. It currently features Distributed Shampoo, which uses preconditioning to improve convergence, and GPA-AdamW (Generalized Primal Averaging), which utilizes a switching mechanism between training and evaluation modes to handle gradient computations across different sequences.

Who it’s for

Machine learning researchers and developers who need high-efficiency optimization algorithms for training large-scale models in PyTorch.

Highlights

  • Implements Distributed Shampoo for optimized preconditioning.
  • Includes GPA-AdamW for generalized primal averaging.
  • Designed for external collaboration and development.
  • Fully compatible with PyTorch.

Related

  • Project
    mosaicml/composer

    A PyTorch-based deep learning library that simplifies distributed training at scale, providing tools for memory optimization and algorithmic speedups to reduce training time and cost.

  • Project
    pytorch/ao

    A PyTorch-native library for model optimization that provides quantization and sparsity tools to accelerate training and inference while reducing memory usage.

  • Project
    pypose/bae

    A PyTorch‑based library that provides exact second‑order (Levenberg‑Marquardt, Schur‑complement) optimizers for sparse robotics problems such as Bundle Adjustment and Pose‑Graph Optimization, with CUDA acceleration, PyPose integration, and support for torch.compile.

  • Project
    HomebrewML/HeavyBall

    A high-performance PyTorch optimizer library that uses compiled, composable building blocks to fuse operations into Triton kernels for faster training and lower memory usage.

  • Project
    kozistr/pytorch_optimizer

    A PyTorch‑compatible library that aggregates 100+ research optimizers, several LR schedulers and loss functions behind a single, consistent API, with optional integrations for low‑precision training.