google-deepmind/kfac-jax

What it solves

KFAC-JAX provides an easy-to-use implementation of the K-FAC (Kronecker-factored Approximate Curvature) optimizer and curvature estimator. It addresses the challenge of performing second-order optimization of neural networks, which is typically more computationally expensive than first-order methods, by providing scalable approximations of the curvature matrix.

How it works

Built on top of JAX, the library allows researchers to optimize neural networks by approximating the curvature of the loss surface. It requires the user to register the specific loss function (e.g., via register_softmax_cross_entropy_loss) so the library can correctly approximate the curvature matrix. The optimizer handles internal staging (JIT and PMAP) automatically, meaning users should not manually apply jax.jit or jax.pmap to the optimizer's step function or the loss function.

Who it’s for

AI researchers who are using JAX and want to implement second-order optimization techniques to improve the training of their neural networks.

Highlights

• JAX-native: Built specifically for the library for high performance on accelerators.
• Automatic Staging: Automatically handles jit and pmap for the optimizer step.
• Adaptive Parameters: Supports adaptive learning rate, adaptive momentum, and adaptive damping.
• Curvature Estimation: Provides scalable curvature approximations for neural network training.