Hugging Face Transformers ZeRO Integration via DeepSpeed and FairScale
Hugging Face Transformers v4.2.0 introduces experimental support for DeepSpeed and FairScale, integrating the Zero Redundancy Optimizer (ZeRO) to allow users to train larger models and increase batch sizes by optimizing GPU memory usage. This integration allows these optimizations to be used directly via the Trainer API using the --sharded_ddp (FairScale) and --deepspeed (DeepSpeed) command line arguments.
ZeRO Memory Optimizations for Multi-GPU Training
Integrating ZeRO via DeepSpeed or FairScale significantly reduces training and evaluation time while increasing the maximum possible batch size (BS) compared to standard Distributed Data Parallel (DDP) baselines. In a benchmark using a t5-large model on two 24GB Titan RTX GPUs, the following performance improvements were observed:
| Method | Max Batch Size | Train Time | Eval Time |
|---|---|---|---|
| Baseline (DDP) | 16 | 30.9458 | 56.3310 |
| fp16 | 20 | 21.4943 | 53.4675 |
| sharded_ddp (FairScale) | 30 | 25.9085 | 47.5589 |
| sharded_ddp + fp16 | 30 | 17.3838 | 45.6593 |
| DeepSpeed (no CPU offload) | 40 | 10.4007 | 34.9289 |
| DeepSpeed (with CPU offload) | 50 | 20.9706 | 32.1409 |
DeepSpeed demonstrated the highest gains in batch size and training speed, while FairScale is noted as being easier to deploy as it only requires a single command line argument without a configuration file.
Single-GPU Training for Large Models
DeepSpeed enables the training of models that would otherwise exceed the memory capacity of a single GPU through CPU offloading. In a test using a t5-3b model on a single 24GB RTX-3090 card, a standard single-GPU setup failed with an Out of Memory (OOM) error even with a batch size of 1. By using DeepSpeed, the model was successfully trained with a batch size of 20, with the system hitting OOM only at a batch size of 30.
Technical Mechanics of ZeRO
ZeRO (Zero Redundancy Optimizer) optimizes memory by adding distributed data storage to data parallel training. Instead of replicating the entire model state across all GPUs, ZeRO partitions parameters, gradients, and optimizer states across the available GPUs.
Distributed Partitioning
Each GPU stores only a single shard of the parameters, gradients, and optimizer states. At runtime, each GPU fetches the required data for a specific layer from other participating GPUs on the fly, ensuring zero overlap in data storage.
ZeRO-Offload
ZeRO-Offload moves specific processing and memory requirements from the GPU to the host CPU, which is critical for fitting massive models (like t5-3b) onto limited GPU hardware.
Memory Fragmentation Management
DeepSpeed addresses GPU memory fragmentation—where OOM errors occur despite available total memory because no contiguous block is large enough—by managing GPU memory independently to separate long-term and short-term allocations.
Deployment and Integration
ZeRO optimizations require no modifications to the model architecture; changes are only required in the training code. Users of the Hugging Face Trainer can implement these features using the following flags:
--sharded_ddp: Activates FairScale integration.--deepspeed: Activates DeepSpeed integration (requires a JSON configuration file).
Future enhancements mentioned include DeepSpeed Sparse Attention, 1-bit Adam, and the anticipated support for model parameter sharding in both FairScale and DeepSpeed.