microsoft/dion

What it solves

Training large neural networks typically relies on optimizers like AdamW, but orthonormal optimizers can significantly accelerate model convergence. However, standard orthonormalization (like Newton-Schulz iterations) is communication-intensive in distributed training because it requires reconstructing full matrices from sharded shards across multiple GPUs.

How it works

This project provides a suite of scalable, communication-efficient orthonormal optimizers. While they share the benefits of Muon, they reduce costs by shrinking the matrix before orthonormalization:

• Dion, Dion2, and Dion3: These methods compress the matrix to reduce computational and communication overhead. Dion uses power iteration for low-rank approximation, Dion2 uses submatrix selection, and Dion3 combines submatrix selection with per-neuron normalization. All three use an error-feedback mechanism to track information loss during compression.
• Muon and NorMuon: High-performance implementations of Muon and its variant NorMuon, which adds neuron-wise normalization for better stability.

To handle distributed training, the library uses DTensor-based parallelism, supporting PyTorch DDP, FSDP2, and (for the legacy Dion) Tensor Parallelism. It optimizes performance by processing parameters in batches and interleaving compute with communication.

Who it’s for

ML engineers and researchers training large-scale models (like Transformers) in distributed environments who want to accelerate convergence using orthonormal optimization without the massive communication overhead of standard methods.

Highlights

• Communication Efficiency: Reduces overhead by communicating only a fraction of the momentum matrix (in Dion2/3).
• Distributed Support: Native integration with PyTorch FSDP2, DDP, and Tensor Parallelism.
• Advanced Parameter Grouping: Supports specialized handling for attention projections (per-head or per-block Newton-Schulz) to prevent information blending across heads.
• Flexible Optimizer Suite: Includes Dion, Dion2, Dion3, Muon, and NorMuon, allowing users to choose the best trade-off between efficiency and stability.