Google DeepMind Decoupled DiLoCo

Google DeepMind has announced Decoupled DiLoCo (Distributed Low-Communication), a new distributed architecture designed to train large language models (LLMs) across geographically distant data centers. By decoupling training runs into asynchronous "islands" of compute, the system isolates hardware failures and eliminates the communication bottlenecks associated with traditional tightly coupled synchronization.

Asynchronous Training and Hardware Resiliency

Decoupled DiLoCo enables training across separate compute islands, known as learner units, using an asynchronous data flow. This architecture prevents a chip failure in one region from interrupting the progress of other learner units, making the system significantly more fault-tolerant than traditional methods.

Key resiliency features include:

• Self-healing infrastructure: The system can continue training after the loss of entire learner units and seamlessly reintegrate them once they return online.
• Chaos engineering validation: Google DeepMind used artificial hardware failures to verify that the system maintains higher availability of learning clusters during failures compared to conventional training methods.
• ML Performance parity: Testing with Gemma 4 models demonstrated that Decoupled DiLoCo delivers the same benchmarked machine learning performance as traditional training approaches, despite the increased resiliency to hardware failures.

Low-Bandwidth, Global-Scale Pre-training

Decoupled DiLoCo is designed for production-level pre-training across wide-area networks (WAN), avoiding the "blocking" bottlenecks where compute units must wait for synchronization.

In a real-world experiment, Google DeepMind trained a 12 billion parameter model across four separate U.S. regions. The technical results included:

• Network Requirements: The training was executed using 2-5 Gbps of wide-area networking, a bandwidth level achievable with existing internet connectivity between data centers.
• Speed Improvements: The system achieved training results more than 20 times faster than conventional synchronization methods by incorporating communication into longer periods of computation.
• Bandwidth Efficiency: The architecture requires orders of magnitude less bandwidth than conventional training methods.

Support for Heterogeneous Hardware

The architecture allows for the mixing of different hardware generations within a single training run, which increases total available compute and extends the hardware lifecycle.

Google DeepMind demonstrated that mixing chips from different generations—specifically TPU v6e and TPU v5p—running at different speeds still matched the ML performance of single-chip-type training runs. This capability alleviates logistical bottlenecks caused by the staggered rollout of new hardware generations across different facilities.