kozistr/pytorch_optimizer

pytorch‑optimizer – a toolbox of ready‑to‑use PyTorch optimizers, schedulers & loss functions

What it is – A Python package that bundles more than a hundred research‑grade optimizers (e.g., AdamP, Ranger21, Lion, SAM), a handful of learning‑rate schedulers and loss functions behind a single, consistent API. It is meant for anyone training deep‑learning models with PyTorch who wants to try newer optimisation tricks without writing custom boilerplate.

Why you might care

• Broad coverage – the library collects many recent variants that are otherwise scattered across individual repos or papers.
• Uniform interface – all components can be instantiated directly, looked‑up by name (load_optimizer('adamp')), or built with the helper create_optimizer() that also wires optional tricks like Gradient Centralization, Lookahead, or foreach for speed.
• Optional ecosystem hooks – works with bitsandbytes, q‑galore‑torch and torchao when those packages are installed, giving you low‑precision or quantised optimisation without extra code.
• Production‑ready – CI‑tested, type‑annotated, and published on PyPI (Apache‑2.0 licence).

Installation

pip install pytorch-optimizer   # requires Python ≥3.8 and PyTorch ≥1.10

If you need the optional integrations, install them separately (e.g., pip install bitsandbytes).

Quick start

from pytorch_optimizer import AdamP, create_optimizer, load_optimizer

model = MyNet()
# 1️⃣ Direct class use
opt = AdamP(model.parameters(), lr=1e-3)

# 2️⃣ Load by string name
opt = load_optimizer('adamp')(model.parameters(), lr=1e-3)

# 3️⃣ Helper that also adds common tricks
opt = create_optimizer(
    model,
    optimizer_name='adamp',
    lr=1e-3,
    weight_decay=1e-3,
    use_gc=True,          # Gradient Centralization
    use_lookahead=True,   # Lookahead wrapper
)

You can also fetch the class via torch.hub:

opt_cls = torch.hub.load('kozistr/pytorch_optimizer', 'adamp')
opt = opt_cls(model.parameters(), lr=1e-3)

Discover what’s available

from pytorch_optimizer import (
    get_supported_optimizers,
    get_supported_lr_schedulers,
    get_supported_loss_functions,
)

print(get_supported_optimizers())               # list of all 100+ optimizers
print(get_supported_optimizers('adam*'))        # filter by pattern
print(get_supported_lr_schedulers('cosine*'))   # scheduler names
print(get_supported_loss_functions('*focal*')) # loss names

The README includes a long table linking each optimizer to its original code repo and paper, so you can trace back to the research source.

Typical use‑case workflow

1. Pick an optimizer (e.g., Ranger21) that matches your training regime.
2. Optionally enable use_gc or use_lookahead for extra stability.
3. Plug the optimizer into your usual PyTorch training loop – no other changes needed.
4. If you need a learning‑rate schedule, retrieve one with get_supported_lr_schedulers() and pass it to torch.optim.lr_scheduler or use the library’s built‑in wrappers.

Who maintains it – Actively maintained by the kozistr team, with CI badges, code‑coverage reporting and regular releases on PyPI.

---

Bottom line – pytorch‑optimizer is a practical, production‑oriented collection of modern optimisers and related utilities that lets PyTorch users experiment with the latest training algorithms quickly and safely.