Accelerate ND-Parallel: Efficient Multi-GPU Training Guide

Hugging Face has introduced ND-Parallelism in Accelerate and Axolotl, providing a streamlined way to combine multiple parallelism strategies—Data Parallelism (DP), Fully Sharded Data Parallelism (FSDP), Tensor Parallelism (TP), and Context Parallelism (CP)—within a single training script. This integration allows developers to optimize the trade-off between memory usage and communication overhead when training models with tens or hundreds of billions of parameters across multi-node GPU clusters.

Core Parallelism Strategies

Data Parallelism (DP)

Data Parallelism replicates the entire model, gradients, and optimizer states across every device. Each device processes a different sub-batch of data, and gradients are synchronized across all devices before updating parameters. This increases throughput but requires the entire model to fit on a single GPU.

Fully Sharded Data Parallelism (FSDP)

FSDP shards model weights, gradients, and optimizer states across GPUs, reducing the memory footprint per device. To perform a forward or backward pass, FSDP gathers the required weights for a specific layer (typically a transformer decoder block) and re-shards them afterward. This trades increased communication overhead for significantly lower peak memory usage.

Tensor Parallelism (TP)

Tensor Parallelism splits large linear layers (such as feed-forward layers or attention projections) across devices. Unlike FSDP's dynamic sharding, TP creates static memory partitions. Because TP requires frequent activation synchronization, it is most effective within a single node using high-bandwidth links (e.g., NVLink) and is not recommended for PCIe-linked GPUs.

Context Parallelism (CP)

Context Parallelism shards the input sequence across GPUs to handle extremely long sequence lengths that would otherwise exceed GPU memory due to the quadratic scaling of attention. Using RingAttention, each GPU holds a shard of the query, key, and value matrices and circulates the key-value shards around a ring of GPUs, ensuring each query computes attention scores against the entire sequence while distributing the compute and memory load.

ND-Parallelism: Composing Strategies for Multi-Node Scaling

Multi-node training often faces bottlenecks from inter-node latency and memory constraints. ND-Parallelism treats the cluster as a multi-dimensional topology to optimize communication.

Hybrid Sharded Data Parallelism (HSDP)

HSDP is a 2D parallelism approach that performs FSDP within a node (utilizing fast intra-node links) and DP across nodes. This minimizes slow inter-node communication to a single gradient synchronization step, increasing throughput at the cost of higher memory usage compared to pure FSDP.

FSDP + Tensor Parallelism

Combining FSDP and TP involves sharding the model across nodes via FSDP and splitting layers within a node via TP. This reduces FSDP latency, allows the training of layers too large for a single device, and enables a reduction in the global batch size.

FSDP + Context Parallelism

This 2D strategy is used when training with very large sequence lengths. While CP already integrates with FSDP, adding FSDP on top of CP further reduces the memory budget required for model weights and optimizer states.

Hybrid Sharded Data Parallelism + Tensor Parallelism

This 3D hierarchy uses DP to replicate the model across groups of nodes, FSDP to shard the model within those groups, and TP to split layers within each node. This configuration provides maximum flexibility for adapting to specific hardware and scaling constraints.

Implementation and Usage Notes

Configuration in Accelerate and Axolotl

Users can configure these strategies via the ParallelismConfig class in Accelerate or specific config fields in Axolotl:

  • dp_shard_size: Degree of FSDP
  • dp_replicate_size: Degree of DP
  • tp_size / tensor_parallel_size: Degree of TP
  • cp_size / context_parallel_size: Degree of CP

Memory and Stability Optimizations

  • CPU RAM Efficient Loading: For models too large for a single device, enabling cpu_ram_efficient_loading and SHARDED_STATE_DICT in the FullyShardedDataParallelPlugin is critical.
  • Effective Batch Size: The effective batch size is calculated as micro_batch_size * gradient_accumulation_steps * dp_world_size, where dp_world_size = (dp_shard_size * dp_replicate_size) / tp_size.
  • Learning Rate Scaling: As the effective batch size increases, the learning rate should be scaled linearly or via square root scaling to maintain stability.
  • Gradient Checkpointing: This trades compute for memory by recomputing intermediate activations during the backward pass. It can reduce activation memory by 60-80% while increasing training time by approximately 20-30%.

Sources