google/orbax

Orbax provides common checkpointing and persistence utilities for JAX users

What it solves

Orbax provides a standardized way to save and restore the state of JAX models (checkpointing) and handle model persistence. It addresses the complexity of managing model weights and optimizer states across different frameworks and storage formats, particularly for large-scale distributed training.

How it works

Orbax offers a composable API that allows users to save (checkpoint) and load (restore) JAX pytrees—such as model weights and optimizer states—to and from storage. It supports asynchronous checkpointing to minimize training interruptions and provides flexibility in handling custom types and various storage formats.

Who it’s for

It is designed for machine learning practitioners and researchers using JAX for training and evaluating large-scale models, including those building foundation models or high-performance sequence models.

Highlights

  • Asynchronous Checkpointing: Reduces overhead by saving state without blocking training.
  • Distributed Support: Used extensively in high-performance JAX frameworks like MaxText and PaxML.
  • Flexible Storage: Supports various storage formats and custom types.
  • Broad Integration: Integrated into major JAX ecosystems including Flax, Gemma, and AXLearn.

Related

  • Project
    google-deepmind/optax

    A gradient processing and optimization library for JAX that provides composable building blocks for creating custom optimizers and loss functions.

  • Project
    probml/dynamax

    A JAX-based library for probabilistic state space models that provides tools for inference and learning across various model types like HMMs and Gaussian SSMs.

  • Project
    google/tunix

    A JAX-based post-training library for LLMs that provides scalable support for supervised fine-tuning and reinforcement learning on TPUs.

  • Project
    AI-Hypercomputer/maxtext

    A high-performance, scalable LLM library written in JAX and Python for pre-training and post-training large models on Google Cloud TPUs and GPUs.

  • Project
    RobertTLange/evosax

    A high-performance JAX library implementing over 30 Evolution Strategies for efficient, vectorized neuroevolution on hardware accelerators.