pytorch/gloo

What it solves

Gloo is a collective communications library designed to facilitate multi-machine training for machine learning applications. It provides the necessary primitives to synchronize and move data across a cluster of machines efficiently.

How it works

It implements several collective algorithms, such as barrier, broadcast, and allreduce. The library abstracts the underlying data transport, allowing it to use standard IP (TCP) or InfiniBand/RoCE for higher performance. For GPU-accelerated workloads, it supports GPUDirect to enable direct GPU-to-GPU memory transfers across machines, reducing the need to copy data between host and device memory.

Who it’s for

Developers and researchers working on distributed machine learning training who need a reliable way to coordinate data exchange between multiple compute nodes.

Highlights

• Collective Primitives: Includes essential operations like allreduce, broadcast, and barrier.
• Hardware Acceleration: Supports InfiniBand, RoCE, and GPUDirect for high-speed GPU transfers.
• Flexible Transport: Works with standard IP/TCP while providing optimized paths for both system memory and NVIDIA GPU memory buffers.
• Integration: Often used in combination with PyTorch Distributed for multi-node scaling.