Accelerate Large Model Training using PyTorch Fully Sharded Data Parallel

Hugging Face has integrated PyTorch Fully Sharded Data Parallel (FSDP) into the Accelerate library, allowing practitioners to train significantly larger models by sharding optimizer states, gradients, and parameters across data parallel workers. This integration democratizes large model training by enabling the use of larger batch sizes and the ability to train models that would otherwise exceed GPU memory limits through CPU offloading.

FSDP vs. Distributed Data Parallel (DDP)

PyTorch FSDP improves upon Distributed Data Parallel (DDP) by eliminating the redundant memory consumption associated with full model replicas on every GPU.

In DDP, every worker maintains a complete copy of the model parameters, gradients, and optimizer states. While each worker processes a different data batch, they must perform an all-reduce operation to average gradients across all workers before updating the model.

In FSDP, optimizer states, gradients, and model parameters are sharded across workers. During the forward and backward passes, FSDP uses all-gather operations to retrieve only the necessary parameters for a specific layer or wrapped module, releasing them immediately after computation. Local gradients are then averaged and distributed via a reduce-scatter operation, allowing each worker to update only its local shard of parameters. This approach drastically reduces the memory footprint per GPU.

Performance Benchmarks on GPT-2

Hugging Face benchmarked FSDP against DDP using two NVIDIA Titan RTX GPUs (24GB each) on Causal Language Modeling tasks.

GPT-2 Large (762M Parameters)

FSDP enables significantly larger batch sizes compared to DDP. Without CPU offload, FSDP allows batch sizes up to 15 (compared to 7 for DDP). With CPU offload enabled, batch sizes increase further to 22. While DDP with mixed precision (FP16) was the fastest in terms of raw training time, FSDP provides the memory efficiency required for larger batches, which is particularly beneficial for applications with dynamic batching.

GPT-2 XL (1.5B Parameters)

For the GPT-2 XL model, DDP failed with CUDA Out of Memory (OOM) errors even at a batch size of 1. In contrast, FSDP enabled successful training:

  • FSDP (Zero-Stage 3): Supported a batch size of 5 per GPU on 2 GPUs.
  • FSDP with CPU Offload: Enabled training on a single GPU with a batch size of 10, and on 2 GPUs with a batch size of 14 per GPU.

Technical Implementation and Configuration

Integration via Accelerate

Users can leverage FSDP through the accelerate config CLI or for more granular control via the FullyShardedDataParallelPlugin. Key configuration options include:

  • Sharding Strategy: Choices between FULL_SHARD and SHARD_GRAD_OP.
  • Min Num Params: The minimum number of parameters required for a layer to be wrapped by the default auto-wrap policy.
  • Offload Params: A boolean to determine if parameters and gradients should be offloaded to the CPU.

The Role of Auto Wrap Policy

The min_num_params setting is critical for memory optimization. When using the default_auto_wrap_policy, FSDP wraps a layer if its parameter count exceeds the specified threshold. Benchmarks on BERT-Large (330M) show that FSDP with auto-wrap consumes roughly half the memory of DDP. Lowering min_num_params (e.g., to 2k) can marginally reduce memory usage further compared to higher thresholds (e.g., 1M).

Critical Caveats and Limitations

Practitioners using FSDP must be aware of several technical constraints:

  • Optimizer Initialization: FSDP flattens parameters and shards them in place. Therefore, the model must be prepared via accelerator.prepare(model) before the optimizer is created. Creating an optimizer before wrapping the model can break the optimizer or lead to increased memory usage.
  • Parameter Groups: Because FSDP flattens nested modules into 1D arrays, parameter groups created before wrapping (e.g., applying different weight decay to biases) are conflated into a single group and lost.
  • Multiple Models: When training multiple models, preparing the models before creating their respective optimizers is mandatory to avoid errors.
  • Mixed Precision: At the time of this publication, mixed precision was not supported with FSDP due to pending PyTorch fixes.

Summary of Distributed Training Approaches

FSDP is part of a broader ecosystem of distributed training strategies designed to handle massive models:

  • ZeRO (Zero Redundancy Optimizer): The foundation for FSDP, sharding optimizer states (Stage 1), gradients (Stage 2), and parameters (Stage 3).
  • Tensor Parallelism: Sharding parameters of individual large layers across GPUs.
  • Pipeline Parallelism: Distributing different layers across different GPUs and pipelining micro-batches.
  • 3D Parallelism: A combination of ZeRO, Tensor, and Pipeline parallelism, used for models with hundreds of billions of parameters.

Sources