Hugging Face DeDLOC: Collaborative Training of Language Models over the Internet

Hugging Face has introduced DeDLOC (Distributed Deep Learning in Open Collaborations), a new method for collaborative distributed training that allows multiple participants to combine their computational resources over the internet to pretrain large language models. This approach overcomes the traditional bottleneck of slow internet connection speeds compared to high-performance GPU supercomputers, enabling the broader ML community to train high-quality models without needing centralized, expensive hardware.

DeDLOC: Enabling Distributed Training over the Internet

DeDLOC addresses the challenge of training Transformers on large datasets, which typically requires hardware resources beyond the reach of individuals or small organizations. While data-parallel distributed deep learning usually involves splitting data across workers and averaging gradients, this process typically fails over the internet due to unstable connections and limited bandwidth.

Fault-Tolerant Gradient Accumulation

To handle the instability of volunteer-based computing, DeDLOC accumulates one very large batch across all participating devices before performing an optimizer step. This method provides built-in fault tolerance:

  • Peer Disconnection: If a participant disconnects, their contribution is simply deducted from the current accumulated batch size, and other participants compensate.
  • Scalability: As more peers join, the target batch size is reached faster, naturally speeding up the training process.

Adaptive Averaging Strategy

To prevent central server overload and account for diverse hardware, DeDLOC uses an adaptive averaging algorithm based on the All-Reduce primitive. The system optimizes data transfer on the fly by splitting the gradient vector into parts based on each peer's internet speed:

  • High-speed peers aggregate the largest parts of the gradient.
  • Firewalled peers send their data for aggregation but do not compute the average themselves.

These core decentralized training techniques are implemented in the Hivemind library.

Case Study: Pretraining sahajBERT

To demonstrate the efficacy of DeDLOC, researchers pretrained sahajBERT, a masked language model for the Bengali language, using a collaborative event with 40 volunteers.

Model Architecture and Tokenization

Researchers chose the ALBERT (A Lite BERT) architecture because its weight-sharing mechanism is parameter-efficient, reducing the amount of data exchanged between peers. The model features approximately 18 million trainable parameters.

For the Bengali language, a custom tokenizer was developed using the Unigram Language Model approach with a 32k token vocabulary. The preprocessing pipeline included:

  • Normalization: NMT and NFKC normalization, removal of multiple spaces, and homogenization of recurring Unicode characters, while specifically preserving accents necessary for Bengali vowels.
  • Pretokenization: Isolation of punctuation and digits and the use of a special character (▁) to mark the beginning of words.

Dataset Streaming

To avoid requiring volunteers to have massive local storage, the team implemented dataset streaming. This allows participants to download and transform training examples from the Bengali Wikipedia dump and the OSCAR dataset in parallel with the training process, rather than downloading the entire corpus in advance.

Collaborative Execution and Results

The training event took place from May 12 to May 21, involving 40 participants (30 Bengali-speaking volunteers and 10 from the authors' organizations). The experiment utilized 600 different sessions and 16 preemptible T4 cloud instances for stability, totaling 234 days of cumulative runtime.

Performance Evaluation

sahajBERT was evaluated on two downstream tasks: Named Entity Recognition (NER) on WikiANN and News Category Classification (NCC) on the Soham articles dataset. Despite having only ~18M parameters, sahajBERT achieved results comparable to much larger models:

Model NER F1 (mean ± std) NCC Accuracy (mean ± std)
sahajBERT 95.45 ± 0.53 91.97 ± 0.47
XLM-R-large 96.48 ± 0.22 90.05 ± 0.38
IndicBert 92.52 ± 0.45 74.46 ± 1.91
bnRoBERTa 82.32 ± 0.67 80.94 ± 0.45

Notably, sahajBERT's performance is comparable to XLM-R-large, which has ~559M parameters and was trained on hundreds of V100 GPUs.

Sources