Distributed Training of BART and T5 for Summarization via Hugging Face and Amazon SageMaker

Hugging Face and Amazon SageMaker have integrated to provide optimized Deep Learning Containers (DLCs) and a dedicated HuggingFace estimator in the SageMaker Python SDK. This collaboration allows developers to train state-of-the-art NLP models, such as BART and T5, using distributed training strategies to accelerate performance and reduce training time.

Optimized Infrastructure for Transformers

Amazon SageMaker now features Hugging Face-optimized Deep Learning Containers to accelerate the training of Transformers-based models. To simplify the deployment process, the SageMaker Python SDK includes a HuggingFace estimator, which enables users to initiate training jobs with minimal code.

Distributed Training with SageMaker Data Parallelism

Distributed training is implemented via SageMaker Data Parallelism, which is integrated into the Hugging Face Trainer API. By defining the distribution parameter within the HuggingFace estimator, users can scale training across multiple GPUs and instances.

Configuration Example:

distribution = {'smdistributed':{'dataparallel':{ 'enabled': True }}}

In a distributed setup, the total batch size is calculated as the per_device_train_batch_size multiplied by the number of GPUs used across all instances.

Technical Implementation: Fine-Tuning BART for Summarization

To demonstrate the integration, a BART-large-cnn model was fine-tuned on the samsum dataset, which consists of approximately 16,000 messenger-like conversations and their corresponding summaries.

Hardware and Hyperparameters

Training was conducted using the following configuration:

  • Instance Type: ml.p3dn.24xlarge (containing 8x NVIDIA V100 GPUs)
  • Instance Count: 2 (Total of 16 GPUs)
  • Total Batch Size: 64 (4 per device * 16 GPUs)
  • Model Size: 400 Million Parameters
  • Training Epochs: 3
  • Learning Rate: 5e-5
  • Precision: FP16 enabled

Performance and Cost

The training job completed in 2,882 billable seconds. For this specific configuration using 16 NVIDIA Tesla V100 GPUs, the cost was approximately $28.

Model Deployment and Hub Integration

Once training is complete, the model artifacts are stored in Amazon S3. The workflow for sharing the model includes:

  1. Downloading Artifacts: Using the S3Downloader to retrieve the trained model from S3.
  2. Model Card Creation: Generating a README.md that includes hyperparameters and evaluation metrics (such as ROUGE-1, ROUGE-2, and ROUGE-L).
  3. Hub Upload: Utilizing the huggingface_hub SDK to create a repository and push the fine-tuned model to huggingface.co for public access and hosted inference testing.

Sources