kozistr/pytorch_optimizer

What it solves

It provides a unified, production-ready toolkit for PyTorch users who want to experiment with a vast array of modern optimization algorithms, learning rate schedulers, and loss functions without having to write repetitive boilerplate code or manually integrate multiple research repositories.

How it works

The library offers a consistent API to load and create optimizers, schedulers, and losses. Users can instantiate optimizers either by calling the class directly, using a load_optimizer function, or through a create_optimizer helper that supports advanced features like Gradient Centralization and Lookahead. It also supports PyTorch's torch.compile to fuse optimizer updates for better performance, with dedicated compiled paths for specific optimizers like Lion and Tiger.

Who it’s for

Machine learning engineers and researchers using PyTorch who need fast experimentation with diverse training methods and a standardized way to integrate advanced optimizers into their training pipelines.

Highlights

• 100+ optimizers, 10+ learning rate schedulers, and 10+ loss functions.
• Support for torch.compile to automatically fuse optimizer updates.
• Integration with ecosystem tools like bitsandbytes, q-galore-torch, and torchao.
• Consistent loader APIs for easy switching between different optimization strategies.