google-deepmind/dm-haiku

What it solves

Haiku is a neural network library for JAX that allows developers to use familiar object-oriented programming models (like those found in Sonnet or TensorFlow) while maintaining full access to JAX's pure function transformations. It simplifies the management of model parameters and internal state without requiring the user to write extensive boilerplate for initialization.

How it works

Haiku provides two primary tools to bridge the gap between object-oriented design and functional purity:

• hk.Module: A Python object used to define neural network layers and components. These modules hold references to parameters and methods, but are treated as functionally "impure" during definition.
• hk.transform: A function transformation that converts these impure module-based functions into a pair of pure functions: init (which collects initial parameter values) and apply (which injects those parameters back into the function for computation).

For models requiring mutable state (like Batch Normalization moving averages), Haiku provides hk.transform_with_state, which manages both parameters and state separately.

Who it’s for

Researchers and developers who want the productivity of an object-oriented API for building neural networks but need the performance and transformation capabilities of JAX (such as jax.jit, jax.grad, and jax.pmap).

Highlights

• JAX Integration: Fully compatible with JAX's automatic differentiation and GPU/TPU support.
• Sonnet-like API: Designed to be a near-match for the Sonnet 2 API, making transitions from TensorFlow/Sonnet easy.
• Simplified RNG Management: Provides hk.next_rng_key() to handle random number generation deterministically within transformed functions.
• Scalable: Tested by DeepMind researchers at scale for image, language, and reinforcement learning tasks.
• Library, not Framework: Focuses strictly on parameter and state management, leaving optimizers and checkpointing to other libraries.