NVIDIA/nccl

What it solves

It provides optimized communication primitives for GPUs, enabling efficient data exchange between multiple GPUs whether they are located on a single node or spread across multiple nodes.

How it works

NCCL implements standard communication routines such as all-reduce, all-gather, reduce, broadcast, and reduce-scatter, as well as general send/receive patterns. It is optimized for high bandwidth across various hardware platforms, including PCIe, NVLink, NVswitch, and networking protocols like InfiniBand Verbs and TCP/IP sockets.

Who it’s for

Developers building multi-GPU applications, often using single- or multi-process environments (such as MPI), who need high-performance inter-GPU communication.

Highlights

• Optimized for high bandwidth on PCIe, NVLink, and NVswitch.
• Supports InfiniBand Verbs and TCP/IP sockets for multi-node communication.
• Compatible with single- or multi-process applications.
• Implements a wide range of standard communication routines (e.g., all-reduce, all-gather).