AI-Hypercomputer/maxtext

What it solves

MaxText is designed to provide a high-performance, scalable reference implementation for training Large Language Models (LLMs). It addresses the challenge of achieving high Model FLOPs Utilization (MFU) and throughput (tokens per second) when scaling from a single host to very large clusters of TPUs and GPUs, while keeping the implementation simple and largely "optimization-free."

How it works

Written in pure Python and JAX, MaxText leverages the XLA compiler to optimize performance. It integrates with a suite of JAX AI libraries including Flax (neural networks), Tunix (post-training), Orbax (checkpointing), Optax (optimization), and Grain (dataloading). It supports both pre-training from scratch and scalable post-training using techniques like Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) methods such as Group Relative Policy Optimization (GRPO) and Group Sequence Policy Optimization (GSPO).

Who it’s for

It is intended for researchers and developers building ambitious LLM projects in both research and production environments who need a scalable framework for pre-training or post-training popular open-source models.

Highlights

• Broad Model Support: Supports a wide array of models including Gemma, Llama, DeepSeek, Qwen, and Mistral, including both dense and Mixture-of-Experts (MoE) architectures.
• High Scalability: Capable of pre-training across tens of thousands of chips.
• Multimodal Capabilities: Supports multimodal training for Gemma 3, Gemma 4, and Llama 4 VLMs.
• Diverse Training Techniques: Provides implementations for SFT and RL (GRPO, GSPO) and incorporates advanced features like Multi-Token Prediction (MTP) and Streaming DiLoCo for distributed low-communication training.
• Hardware Target: Specifically optimized for Google Cloud TPUs and GPUs.