Distributed Training with PyTorch DDP, Accelerate, and Transformers Trainer

Hugging Face has detailed a progression of three abstraction levels for implementing distributed training in PyTorch, moving from native Distributed Data Parallelism (DDP) to the Accelerate library and finally the Transformers Trainer API. This progression allows developers to choose between granular control over the training loop and high-level automation of distributed setups.

Native PyTorch Distributed Data Parallelism (DDP)

PyTorch DDP enables training across multiple GPUs, whether they are on a single machine (multi-GPU) or spread across multiple machines (multi-node). It works by copying the model onto each GPU and averaging the resulting gradients across all copies during the loss.backward() call to ensure weight consistency across devices.

Implementing native DDP requires several manual configuration steps:

• Process Group Setup: Developers must define setup and cleanup functions to initialize and destroy the process group, specifying the MASTER_ADDR and MASTER_PORT for communication.
• Model Wrapping: The model must be wrapped in the DistributedDataParallel (DDP) module, and the optimizer must be declared based on this wrapped model to ensure gradients are calculated correctly.
• Execution: Scripts are typically launched using the torchrun command-line module, specifying the number of nodes and processes per node.

Simplifying Distribution with 🤗 Accelerate

Accelerate is a light wrapper around pytorch.distributed that allows the same code to run on a single GPU, multiple GPUs, or TPUs with minimal changes. It removes the need for manual process group setup and simplifies device placement.

Technical Improvements

Accelerate introduces the Accelerator class, which handles the distribution of the model, optimizer, and data loaders via a single call: accelerator.prepare(). This replaces manual .to(rank) calls and DDP wrapping.

Beyond simplification, Accelerate improves memory efficiency through custom samplers. Instead of creating multiple full copies of the data loaders across devices, Accelerate ensures that only one full copy of the original dataset exists in memory, splitting subsets of the data across the available nodes. This prevents memory explosions when training on very large datasets.

Notebook Integration

For users working in Jupyter Notebooks, Accelerate provides the notebook_launcher utility. This allows multi-GPU training to be triggered directly from a notebook by passing the training function and the number of processes to use.

High-Level Abstraction via 🤗 Trainer

The Transformers Trainer API provides the highest level of abstraction, removing almost all boilerplate code associated with distributed training. It automatically handles the underlying distributed logic without requiring the user to write explicit training loops.

To use the Trainer, developers define TrainingArguments to manage hyperparameters and can subclass the Trainer to implement a custom compute_loss function. The Trainer then manages the training and evaluation process across distributed systems automatically. Like the Accelerate examples, the Trainer can also be integrated with notebook_launcher for rapid experimentation in interactive environments.