uccl-project/uccl

UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

What it solves

UCCL is designed to address the performance bottlenecks and portability issues in GPU communication libraries. It provides a high-performance, flexible, and portable communication layer that supports heterogeneous GPU and networking hardware (Nvidia, AMD, Broadcom, AWS EFA), allowing ML workloads to achieve higher throughput and lower latency than traditional libraries like NCCL or RCCL.

How it works

UCCL operates as a communication library for GPUs, offering three primary components:

  • UCCL-collective (UCCL-Tran): A drop-in replacement for NCCL/RCCL that uses a software-defined transport layer. It employs packet spraying across up to 256 network paths to avoid congestion and implements advanced congestion control and selective repeat for loss recovery.
  • UCCL-P2P: Provides initiator-target transfer APIs for high-bandwidth message transfer, specifically optimized for next-gen 800Gbps NICs with multi-threaded transfer engines.
  • UCCL-EP: A portable implementation of expert-parallel communication (DeepEP-compatible) that achieves high performance across different GPU and NIC vendors.

Who it’s for

It is intended for developers and researchers working with large-scale ML training and inference, particularly those using heterogeneous hardware environments or those seeking to optimize GPU-to-GPU communication for tasks like KV cache transfer or expert-parallel (MoE) communication.

Highlights

  • Drop-in Replacement: UCCL-collective can replace NCCL/RCCL without requiring application code changes.
  • Heterogeneous Support: Works across Nvidia and AMD GPUs and various NICs (Broadcom, Nvidia, AWS EFA).
  • High Performance: Outperforms NCCL in AllReduce operations by up to 3.7x in certain settings.
  • Broad Adoption: Integrated into NVIDIA NeMo, AMD TheRock, and other distributed inference stacks.