NVIDIA/nccl-tests

What it solves

This project provides a suite of tests to verify the performance and correctness of the NVIDIA Collective Communications Library (NCCL). It allows developers and system administrators to ensure that multi-GPU and multi-node communication is operating at peak efficiency and that data is being transferred correctly across the network.

How it works

The toolkit consists of a series of binaries that execute various NCCL collective operations (such as Allreduce, Broadcast, and ReduceScatter). It can be configured to run on a single node with multiple GPUs or across multiple nodes using MPI (Message Passing Interface) to manage processes. Users can specify parameters like data size, the number of GPUs, and the number of iterations to measure latency and bandwidth (specifically "busbw").

Who it’s for

It is designed for engineers working with high-performance computing (HPC) and AI infrastructure, specifically those using NVIDIA GPUs for distributed training or inference of large-scale models.

Highlights

• Performance Benchmarking: Measures bandwidth and latency for a wide range of data sizes and collective operations.
• Correctness Checking: Includes a -c flag to verify that the results of thecollective operations are mathematically correct.
• Multi-Node Support: Integrates with MPI to scale tests across multiple physical servers.
• Flexible Configuration: Supports custom data types, reduction operations (sum, prod, min, max, avg), and CUDA graph capture for optimized execution.