google-deepmind/optax

What it solves

Optax is a gradient processing and optimization library for JAX. It aims to simplify the process of implementing and experimenting with optimizers by providing a set of small, composable building blocks that can be recombined to create custom optimization solutions.

How it works

Optax provides implementations of popular optimizers (such as Adam) and loss functions (such as L2 loss). It uses a functional approach where an optimizer is initialized with model parameters to create an optimizer state. Gradients are then computed via JAX, processed through the optimizer's update function to generate updates, and finally applied to the parameters using utility functions like apply_updates.

Who it’s for

This library is designed for researchers and developers working with JAX who need efficient, well-tested implementations of core optimization components or the flexibility to build their own custom optimizers.

Highlights

• Composable Design: Focuses on small building blocks that can be easily combined into custom optimizers.
• JAX Integration: Built specifically for the JAX ecosystem to facilitate gradient processing.
• Extensive Library: Includes a wide array of popular optimizers and loss functions.
• Readability: Prioritizes code structure that matches standard mathematical equations over code reuse.