horovod/horovod

What it solves

Horovod simplifies the process of scaling deep learning training from a single GPU to multiple GPUs and multiple hosts. It removes the complexity of managing parameter servers and reduces the amount of code changes required to make a training script distributed.

How it works

Horovod uses an MPI-based model (Message Passing Interface) and collective communication operations like allreduce, allgather, and broadcast. It integrates with major frameworks such as TensorFlow, Keras, PyTorch, and Apache MXNet.

Key technical features include:

• Tensor Fusion: Batches small allreduce operations together to interleave communication and computation for better performance.
• Distributed Optimizer: A wrapper that averages gradients across workers before applying them.
• Autotuning: An automated system to optimize performance settings for Tensor Fusion and other features.

Who it’s for

Machine learning engineers and researchers who need to train large models across clusters of GPUs or CPUs, and those who want a consistent way to handle distributed training across different deep learning frameworks.

Highlights

• Framework Agnostic: Supports TensorFlow, PyTorch, Keras, and MXNet.
• High Scaling Efficiency: Achieves high scaling efficiency (up to 90% for certain models) across hundreds of GPUs.
• Flexible Deployment: Can be run on local machines, Docker, Kubernetes, Spark, Ray, and HPC clusters.
• MPI/Gloo Support: Supports MPI for high-performance communication and Gloo for environments where MPI is not installed.