microsoft/msccl

What it solves

MSCCL addresses the inefficiency of generic collective communication algorithms in distributed machine learning. Because different hardware topologies (how accelerators are connected) and buffer sizes have different latencies and bandwidths, a one-size-fits-all approach often fails to maximize performance. MSCCL allows users to implement and execute hyper-optimized communication patterns tailored to specific hardware setups.

How it works

Built on top of NVIDIA's NCCL, MSCCL acts as a runtime framework that executes custom collective communication algorithms. It works in tandem with the MSCCL toolkit, which provides a high-level Domain Specific Language (MSCCLang) and a compiler. The compiler transforms these custom algorithms into an Intermediate Representation (IR) in XML format, which the MSCCL runtime then executes on the backend. If no custom algorithm is provided, the system automatically falls back to standard NCCL generic algorithms.

Who it’s for

This tool is designed for researchers and engineers working with large-scale distributed AI training on multiple accelerators (such as GPUs on Azure NDv4), who need to squeeze maximum performance out of their hardware interconnects.

Highlights

• Custom Programmability: Enables the creation of optimized communication algorithms for specific topologies and buffer sizes.
• Performance Gains: Can achieve 2-3x speedups over vanilla NCCL on specific configurations like 8xA100 NVLink systems.
• Integrated Profiling: Includes NPKit to provide detailed timelines of send and receive operations to identify transmission bottlenecks.
• PyTorch Integration: Provides a path to replace default NCCL with MSCCL within PyTorch environments.