PrimeIntellect-ai/OpenDiloco

OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training

What it solves

OpenDiLoCo provides a framework for globally distributed training of large models with low communication overhead. It addresses the bottleneck of high-bandwidth requirements typically associated with distributed training, allowing models to be trained across geographically dispersed machines.

How it works

The framework implements a low-communication training strategy (DiLoCo) that utilizes distributed weight averaging. It integrates with PyTorch FSDP (Fully Sharded Data Parallel) and the hivemind library, using a Distributed Hash Table (DHT) to manage peer connections and synchronize weights across workers. Users can implement this by using the DiLoCoOptimizer, which acts as a replacement for standard optimizers to handle the inner and outer optimization loops required for distributed averaging.

Who it’s for

It is designed for researchers and developers training large transformer models who have access to distributed compute resources across different locations and want to reduce the communication costs of synchronization.

Highlights

  • Low-Communication Training: Specifically designed for globally distributed environments.
  • Integration: Works with PyTorch FSDP and the hivemind library.
  • Drop-in Optimizer: Provides DiLoCoOptimizer for easy integration into existing training scripts.
  • Scalable Architecture: Supports distributed weight averaging via DHT.

Related

  • Project
    PrimeIntellect-ai/prime-diloco

    A framework for efficient, globally distributed AI model training over the internet, featuring fault-tolerant communication and optimized network bandwidth utilization.

  • Dispatch
    Google DeepMind Decoupled DiLoCo

    Google DeepMind introduced Decoupled DiLoCo, a distributed training architecture that enables resilient, asynchronous LLM training across distant data centers using low bandwidth and mixed hardware generations.

  • Project
    horovod/horovod

    A distributed deep learning training framework that makes it easy to scale single-GPU training scripts across multiple GPUs and hosts for TensorFlow, PyTorch, Keras, and MXNet.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Dispatch
    Hugging Face DeDLOC: Collaborative Training of Language Models over the Internet

    Hugging Face introduces DeDLOC, a distributed training method that enables volunteers to collaboratively pretrain large language models over the internet by adapting to varying network and hardware constraints.