google/paxml

What it solves

Paxml (or Pax) provides a framework for configuring and running large-scale machine learning experiments, specifically designed to work on top of Jax. It simplifies the process of managing complex model configurations and deploying them across distributed hardware, such as Cloud TPU VMs and TPU Pod slices.

How it works

Pax leverages Jax to handle computation and provides a structured way to define experiments via configuration files. It supports various parallelism strategies, including SPMD (Single Program, Multiple Data) using pjit and pmap, as well as pipeline parallelism. The framework handles multihost data infeed, ensuring that data is sharded across different Jax processes to avoid duplicate batches during training. It also integrates with different data input types, including SeqIO, Lingvo, and custom implementations, to feed data into models for training, evaluation, and decoding.

Who it’s for

It is designed for ML researchers and engineers who are training very large models (ranging from billions to trillions of parameters) and need a robust system to manage these experiments on TPU or GPU infrastructure.

Highlights

• TPU Optimization: Deeply integrated with Cloud TPU VMs and Pod slices for high-performance training.
• Scalability: Supports weak scaling for large language models, allowing model size to grow in proportion to the number of chips used.
• Flexible Parallelism: Supports pjit (SPMD) and pmap for distributed execution.
• Data Management: Built-in support for multihost infeed and automatic sharding/padding for evaluation data via SeqIO.
• Performance Metrics: Utilizes Model FLOPs Utilization (MFU) to measure end-to-end training efficiency.