meta-pytorch/torchcomms

What it solves

It provides an experimental communications API for PyTorch, simplifying the process of managing collective communications across multiple GPUs or accelerators across different hardware backends.

How it works

torchcomms introduces a high-level collectives API that abstracts the underlying communication libraries. It allows users to initialize a communicator via new_comm() and perform operations like all_reduce (summing tensors across ranks) either synchronously or asynchronously to overlap communication with computation.

Who it’s for

Developers and researchers working with distributed training or large-scale model inference who need a unified interface for collective communications across various hardware accelerators.

Highlights

• Multi-Backend Support: Supports multiple backends including NCCL, NCCLX, RCCL, RCCLX, and XCCL (Intel XPU).
• Asynchronous Operations: Supports async_op=True to allow computation to happen while communication is in progress.
• Hardware Flexibility: Compatible with CUDA-capable GPUs, Intel XPUs, and AMD GPUs (via RCCL/RCCLX).
• PyTorch Integration: Designed to work seamlessly with torchrun and PyTorch tensors.