AI-Hypercomputer/maxtext
A simple, performant, and scalable Jax LLM!
What it solves
MaxText provides a high-performance, scalable reference implementation for training Large Language Models (LLMs). It eliminates the need for complex manual optimization by leveraging JAX and the XLA compiler to achieve high Model FLOPs Utilization (MFU) and throughput across single hosts and massive clusters (up to tens of thousands of chips).
How it works
Written in pure Python and JAX, MaxText targets Google Cloud TPUs and GPUs. It integrates a suite of JAX AI libraries for a complete training stack: Flax for neural networks, Tunix for post-training, Orbax for checkpointing, Optax for optimization, and Grain for data loading. It supports both pre-training from scratch and scalable post-training using techniques like Supervised Fine-Tuning (SFT) and Reinforcement Learning (GRPO and GSPO).
Who it’s for
It is designed for AI researchers and production engineers building ambitious LLM projects who need a high-performance starting point for experimentation, ideation, and large-scale model training.
Highlights
- Broad Model Support: Includes reference implementations for Gemma, Llama, DeepSeek, Qwen, Mistral, and GPT-OSS.
- Scalable Training: Supports pre-training on massive clusters and scalable post-training via Tunix.
- Multi-modal Capability: Supports training for multi-modal models like Gemma 3, Gemma 4, and Llama 4 VLMs.
- Optimization-Free: Achieves high efficiency and throughput automatically via JAX and XLA.
- Decoupled Mode: Can be run without Google Cloud Platform (GCP) dependencies.