Pre-training BERT with Hugging Face Transformers and Habana Gaudi

TL;DR

Hugging Face has provided a technical guide for pre-training BERT-base from scratch using Habana Gaudi-based DL1 instances on AWS. By leveraging the optimum-habana library, the process achieves a 25% cost reduction compared to equivalent NVIDIA V100 GPU training setups.

Hardware and Software Stack

Pre-training is executed on AWS DL1 instances, which feature 8 HPU-cores. The software stack integrates the following Hugging Face libraries:

  • Transformers: The core library for model architecture.
  • Optimum Habana: Provides the GaudiTrainer and GaudiTrainingArguments, which are wrappers around the standard Hugging Face Trainer designed specifically for Habana Gaudi hardware.
  • Datasets: Used for loading and managing the pre-training corpora.

Pre-training Workflow

The pre-training process is divided into CPU-intensive data preparation and HPU-intensive model training.

1. Data Preparation and Tokenization

Data preparation is performed on CPU-optimized instances (such as AWS c6i.12xlarge) before moving to the DL1 instance. The workflow includes:

  • Dataset Merging: Combining Wikipedia (20220301.en split) and BookCorpus into a single dataset.
  • Tokenizer Training: Training a new BertTokenizerFast from scratch with a vocabulary size of 32,000 tokens.
  • Preprocessing: Tokenizing the text and grouping it into chunks of 512 tokens (the maximum sequence length for BERT), truncating documents that exceed this limit.

2. Model Training via Masked Language Modeling (MLM)

BERT is pre-trained using Masked Language Modeling (MLM), a task where the model predicts hidden words in a sentence using bidirectional context.

To leverage the 8 HPU-cores of the DL1 instance, the training is implemented as distributed data-parallel training using a script (run_mlm.py) and the DistributedRunner from optimum-habana (or gaudi_spawn.py).

Performance and Cost Analysis

Training Time and Cost

Hugging Face benchmarked a run of 100,000 steps with a global batch size of 256, which completed in approximately 12.5 hours. Extrapolating this to the original BERT pre-training requirement of 1 million steps:

  • Total Estimated Time: 125 hours.
  • Total Estimated Cost: ~$1,650 using Habana Gaudi on AWS.

Comparison with GPU Baselines

When compared to the DeepSpeed record for the fastest BERT pre-training on a DGX-2 (16 NVIDIA V100 GPUs), the Habana Gaudi approach offers significant savings:

  • GPU Cost Estimate: Using AWS p3dn.24xlarge instances (8x V100 32GB) as a reference, a similar setup would cost approximately $2,075.
  • Cost Reduction: Habana Gaudi delivers a 25% cost reduction over the V100-based setup.

Notably, the author observes that without DeepSpeed optimization, GPU training costs would likely increase further to approximately $3,000–$4,000.

Implications for Domain Adaptation

Pre-training from scratch allows organizations to adapt models to specific languages or specialized domains. According to the source, this process can improve model accuracy by up to 10% compared to using general-purpose BERT models.

Sources