ROCm/rccl

What it solves

RCCL (ROCm Communication Collectives Library) provides a standardized set of communication routines for GPUs. It solves the problem of efficiently moving and synchronizing data across multiple GPUs, whether they are located on a single node or spread across multiple nodes in a cluster.

How it works

RCCL implements standard collective communication patterns such as all-reduce, all-gather, reduce, broadcast, reduce-scatter, gather, scatter, and all-to-all, as well as direct GPU-to-GPU send and receive operations. It uses optimized ring and tree algorithms to maximize throughput and minimize latency. The library is optimized for high bandwidth across various hardware interfaces, including PCIe, xGMI, InfiniBand Verbs, and TCP/IP sockets.

Who it’s for

It is designed for developers building high-performance computing (HPC) and AI applications that require multi-GPU scaling and synchronization, typically using the ROCm software stack on AMD GPUs.

Highlights

• Comprehensive Collective Operations: Supports a wide array of standard communication routines (e.g., all-reduce, all-gather).
• High-Bandwidth Optimization: Optimized for PCIe, xGMI, InfiniBand, and TCP/IP.
• Flexible Deployment: Supports an arbitrary number of GPUs across single or multiple nodes.
• Multi-Process Support: Compatible with single-process or multi-process (e.g., MPI) applications.