NVIDIA/nvbandwidth

What it solves

It provides a standardized way to measure the actual memory bandwidth and latency between NVIDIA GPUs, as well as between the host CPU and GPUs. This is critical for optimizing high-performance computing and AI workloads where data transfer speeds across links (like NVLink) often become the bottleneck.

How it works

The tool performs various memory copy (memcpy) patterns using two different methods:

• Copy Engine (CE): Uses standard CUDA memcpy APIs.
• Streaming Multiprocessor (SM): Uses custom kernels to move data.

To ensure accuracy, it uses a "spin kernel" to eliminate the overhead of enqueuing operations from the measurement. For latency, it employs a pointer chasing technique (creating a linked list of memory addresses) to prevent hardware prefetching and data cache hits, ensuring a more realistic measurement of raw memory access time.

Who it’s for

• System Architects: To verify that GPU clusters are configured for peak performance.
• ML Engineers: To diagnose bottlenecks in multi-GPU training or inference pipelines.
• Cluster Administrators: To validate the health and performance of NVLink and internode memory exchange (IMEX) in multi-node deployments.

Highlights

• Multi-node Support: Can measure bandwidth across different physical nodes using MPI and NVIDIA IMEX.
• Flexible Testing: Supports unidirectional and bidirectional transfers, as well as specific GPU-to-GPU pair testing via UUIDs.
• Intelligent Sampling: For large clusters, it can sample a subset of GPU pairs to reduce test time while maintaining topology coverage.
• Detailed Metrics: Reports median or mean bandwidth and provides specific latency measurements.