microsoft/mscclpp

What it solves

MSCCL++ is a GPU-driven communication stack designed to make inter-GPU communication more efficient and customizable. It addresses the complexity and performance bottlenecks of distributed GPU applications, particularly for large-scale AI models like LLMs, where standard collective communication libraries can be suboptimal for specific workloads (such as the difference between prompt processing and token sampling in LLM inference).

How it works

MSCCL++ provides a multi-layered abstraction system that allows developers to define communication logic directly within GPU kernels.

• Channels: It uses "Channels" as peer-to-peer connections between GPUs. These are defined on the host side but called as device functions within GPU kernels.
• Communication Primitives: It offers 1-sided, 0-copy synchronous and asynchronous primitives (like put(), get(), signal(), flush(), and wait()) that allow data to be sent to a remote GPU without requiring a corresponding receive instruction on the remote side.
• Channel Types: It provides PortChannel (which uses a host-side proxy to trigger DMA transfers) and MemoryChannel (which uses GPU threads to read/write remote memory directly for lower latency).
• Unified Interface: It abstracts away the underlying hardware, providing the same interface whether the GPUs are connected via NVLink, xGMI, or InfiniBand, and whether they are on the same node or different nodes.

Who it’s for

It is intended for developers of high-performance distributed AI applications, kernel developers, and researchers building scalable AI infrastructure, such as those working on LLM serving frameworks (e.g., SGLang, LMDeploy) or deep learning compilers (e.g., TVM).

Highlights

• GPU-Driven: Communication can be triggered and managed directly from within GPU kernels.
• 0-Copy Transfers: Transfers data directly between user buffers, saving bandwidth and memory.
• Hardware Agnostic: Unified abstractions for NVLink, xGMI, and InfiniBand across local and remote nodes.
• Customizable Proxies: Supports custom host-side proxies to optimize how GPU triggers are handled.
• Python Bindings: Includes interfaces for easier integration with Python-based AI applications.