Hugging Face TorchTPU Native Integration

Native TPU Support in PyTorch

TorchTPU enables native PyTorch support for Google's Tensor Processing Units (TPUs), eliminating the need for complex workarounds like torch_xla or JAX. By integrating TPU as a standard device backend, developers can now run Hugging Face Transformers and Diffusers models directly on TPU hardware using familiar PyTorch syntax. Simply setting the device to "tpu" allows models to execute in eager mode, with optional torch.compile support for enhanced performance.

Moving Beyond Legacy Workflows

Historically, running PyTorch on TPUs required torch_xla, which introduced challenges such as lazy tensor evaluation, complex debugging, and frequent graph recompilations. The introduction of TorchTPU replaces the need for the previous Optimum TPU library, providing a more seamless experience. Installation is performed via a single command:

pip install torch-tpu --extra-index-url https://download.pytorch.org/whl/cpu

This approach uses the CPU build of torch to avoid unnecessary CUDA dependencies, optimizing the environment for TPU hardware.

Integration with Transformers and Diffusers

Transformers v5 now treats TPU as a first-class citizen. Users can load models directly onto TPU hardware by passing device_map="tpu" to from_pretrained. For Diffusers, pipelines support TPU through the standard .to("tpu") method.

For large-scale models exceeding the memory capacity of a single chip, users can implement tensor parallelism by passing a TensorParallelConfig. This shards the model across multiple chips during the loading process, enabling the execution of large-scale architectures on TPU pods.

High-Performance Kernels

Users can further optimize TPU performance by utilizing specialized kernels from the Hugging Face Kernels Hub. These kernels are written in Pallas, providing low-level hardware control. For example, using the ragged-dot kernel from Tokamax for Mixture-of-Experts (MoE) models has demonstrated significant throughput improvements. At a batch size of 64, decoding speeds for models like Qwen3-30B-A3B and Gemma 4 26B-A4B saw performance gains of 21.5x and 12.8x, respectively, compared to default implementations.

Continuous Testing and Reliability

To ensure stability, Hugging Face is integrating TPU support into its continuous integration (CI) pipeline. Currently, 96.5% of the tested library suite passes on TPU hardware. The team is actively tracking failures, reporting issues to the TorchTPU repository, and upstreaming fixes to the Transformers library to improve overall compatibility. This ongoing effort aims to make TPU deployment as reliable as traditional GPU-based workflows.