horovod/horovod

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

What it solves

Horovod simplifies the process of scaling deep learning training from a single GPU to many GPUs across multiple hosts. It addresses the difficulty of modifying training scripts for distributed environments and aims to maximize training speed and scaling efficiency.

How it works

Horovod uses an MPI-based model (Message Passing Interface) to handle distributed training. It provides a DistributedOptimizer that wraps a standard optimizer to average gradients across workers using operations like allreduce or allgather. It also supports the Gloo collective communications library for environments without MPI. Key performance features include "Tensor Fusion," which batches small allreduce operations to interleave communication and computation, and an autotuning system to optimize these settings.

Who it’s for

It is designed for machine learning engineers and researchers who use frameworks like TensorFlow, Keras, PyTorch, or Apache MXNet and need to scale their models to large GPU clusters.

Highlights

  • Multi-framework support: Works with TensorFlow, Keras, PyTorch, and Apache MXNet.
  • High scaling efficiency: Achieves significant scaling efficiency for models like Inception V3 and ResNet-101.
  • Flexible deployment: Can be run on local machines, Docker, Kubernetes, Spark, Ray, and HPC clusters.
  • Performance tools: Includes a timeline tool for activity analysis and automated performance tuning.

Related

  • Project
    hpcaitech/ColossalAI

    A distributed deep learning framework that makes training and inference for large AI models faster and cheaper through advanced parallelism and memory management.

  • Project
    intelligent-machine-learning/dlrover

    An automatic distributed deep learning system that simplifies large-scale AI model training by providing automated fault tolerance, auto-scaling, and rapid checkpoint recovery on K8s and Ray.

  • Project
    kubeflow/trainer

    A Kubernetes-native distributed AI platform for scalable LLM fine-tuning and training across multiple frameworks like PyTorch, JAX, and DeepSpeed.

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.

  • Project
    mosaicml/composer

    A PyTorch-based deep learning library that simplifies distributed training at scale, providing tools for memory optimization and algorithmic speedups to reduce training time and cost.