HomebrewML/HeavyBall

What it solves

HeavyBall provides a high-performance, composable library of optimizers for PyTorch. It addresses the inefficiency of standard optimizer implementations by using compiled building blocks that fuse into Triton kernels, reducing memory traffic and increasing execution speed. It also simplifies the integration of advanced optimization techniques (like second-order methods and state compression) that are typically difficult to implement or combine.

How it works

The library is built from over 100 compiled functions that are assembled into chains of transforms. By using torch.compile(fullgraph=True), these functions fuse into minimal kernels, significantly reducing the number of reads and writes to memory. For example, a standard Adam update can be reduced from 14 reads and 9 writes to 4 reads and 3 writes.

It supports a wide array of optimizer types, including first-order (AdamW, SGD), orthogonal (Muon), Shampoo-based (SOAP), and Kronecker-factored (PSGD). It also features a "SplitOpt" for applying different optimizers to different parameter groups. For distributed training, it automatically handles repartitioning for second-order methods when using FSDP.

Who it’s for

Machine learning engineers and researchers who need faster optimizer steps, lower memory overhead for optimizer states, or access to a variety of advanced second-order and orthogonal optimization algorithms within a PyTorch ecosystem.

Highlights

• Extensive Optimizer Suite: Includes API-compatible replacements for AdamW, SGD, and RMSprop, plus Muon, SOAP, LATHER, and ADOPT.
• Composable Features: Chainable flags for MARS variance reduction, cautious updates, and PaLM-style beta2 scheduling.
• ECC State Compression: Reduces optimizer state memory usage (e.g., bf16 + int8 correction) to save space without sacrificing precision.
• High Performance: Fuses operations into Triton kernels via torch.compile for significant speedups in step latency.
• Distributed Support: Native compatibility with DDP and FSDP, including automatic repartitioning for complex second-order methods.