Training Language Models with Hugging Face Transformers, TensorFlow, and TPUs

Hugging Face has detailed a scalable workflow for training masked language models (MLM) from scratch using TensorFlow and Tensor Processing Units (TPUs). By leveraging XLA (Accelerated Linear Algebra) and TPUStrategy, developers can train models ranging from a few million parameters to massive scales, such as Google’s PaLM model, which utilized TPU pods for its 500 billion parameters.

XLA Compatibility and TPU Accessibility

Training TensorFlow models on TPUs has historically been difficult due to XLA incompatibility and data collators that relied on non-native TensorFlow operations. Hugging Face has updated its codebase to ensure most TensorFlow models are XLA-compatible, removing these roadblocks to make TPU training more accessible.

This shift is critical given the ongoing shortage of high-end GPUs. TPUs offer a high-performance alternative for accessing ultra-high-performance compute hardware, providing a scalable path for training large-scale generative AI and LLMs without relying solely on GPU availability.

End-to-End Training Workflow

To demonstrate the scalability of this approach, Hugging Face trained a RoBERTa-base model from scratch using the WikiText (v1) dataset. The process follows a specific pipeline to ensure efficiency on TPU hardware:

1. Tokenizer Training and Data Preparation

Because the model is trained from scratch, a custom tokenizer is required. The workflow involves:

  • Loading the train split of the WikiText dataset via ‚datasets.
  • Training a Unigram model using ‚tokenizers.
  • Uploading the resulting tokenizer to the Hugging Face Hub.

2. Creating TFRecord Shards

To enable massively parallel processing, data is converted into TFRecord shards rather than single files. The tokenization strategy involves concatenating samples and splitting them into fixed-size chunks (128 tokens) to prevent the aggressive loss of text content caused by truncation.

These shards are then uploaded to a Google Cloud Storage (GCS) bucket. This is necessary for TPU nodes, which have limited host memory and must stream data directly from GCS. (Note: TPU VMs can use local datasets or persistent storage).

3. Model Initialization and Distributed Training

To distribute training across TPU workers using data-parallelism, the model and optimizer must be initialized within a TPUStrategy scope:

import tensorflow as tf

tpu = tf.distribute.cluster_resolver.TPUClusterResolver(...)
strategy = tf.distribute.TPUStrategy(tpu)

with strategy.scope():
    # Model and tokenizer initialization happens here
    model = TFAutoModelForMaskedLM.from_config(config)

Key Technical Requirements for TPU Training

Successful TPU integration requires specific configurations in the data pipeline and model setup:

  • TensorFlow-Native Data Collators: The DataCollatorForLanguageModeling must be configured with return_tensor="tf". This ensures the collator returns TensorFlow tensors instead of NumPy arrays, which is essential for TPU compatibility.
  • GCS Integration: TensorFlow's tf.io.gfile.glob allows seamless reading of TFRecord shards from GCS buckets using the gs:// identifier.
  • Model Checkpointing: The PushToHubCallback is used to sync model checkpoints directly to the Hugging Face Hub during training.

Inference and Results

Once trained, the model can be deployed for inference using the standard Hugging Face pipeline API with the framework="tf" argument. The trained RoBERTa-base model was trained for a longer duration with a learning rate of 1e-4, and the resulting weights are available on the Hugging Face Hub.

Sources