ai-dynamo/nixl

NVIDIA Inference Xfer Library (NIXL)

What it solves

NIXL addresses the need for high-performance, point-to-point communication between nodes in AI inference frameworks. It simplifies the process of moving data across different types of memory (CPU and GPU) and storage systems (file, block, and object stores) by providing a unified abstraction layer.

How it works

NIXL uses a modular plug-in architecture to support various backends. It abstracts memory and storage types, allowing AI frameworks to transfer data efficiently without needing to manage the low-level details of the specific hardware or storage medium. It can integrate with UCX for transport and use ETCD for metadata distribution and coordination between nodes in distributed environments.

Who it’s for

This library is designed for developers building AI inference frameworks (such as NVIDIA Dynamo) and those working with distributed AI workloads that require optimized data transfer between GPU/CPU memory and various storage backends.

Highlights

  • Modular Plug-in Architecture: Supports a wide range of backends including UCX, POSIX, GDS, and others.
  • Cross-Platform Memory Support: Provides abstractions over CPU and GPU memory, as well as various storage types.
  • Multi-Language Bindings: Offers APIs for C++, Python, and Rust.
  • Distributed Coordination: Optional integration with ETCD for managing metadata exchange between distributed nodes.
  • Hardware Agnostic: Supports both NVIDIA GPUs (CUDA) and AMD GPUs (ROCm) through vendor-neutral build options.

Related

  • Project
    uccl-project/uccl

    An efficient, portable GPU communication library that provides high-performance collectives, P2P transfers, and expert-parallel communication across heterogeneous GPU and NIC hardware.

  • Project
    flashinfer-ai/flashinfer

    A high-performance GPU kernel library and generator for LLM inference that optimizes attention, GEMM, and MoE operations across diverse NVIDIA GPU architectures.

  • Project
    NVIDIA/TensorRT

    NVIDIA TensorRT is a high-performance deep learning inference library and SDK that accelerates AI model deployment on NVIDIA GPUs.

  • Project
    google/XNNPACK

    XNNPACK is a highly optimized library of low-level performance primitives used to accelerate neural network inference across ARM, x86, WebAssembly, and RISC-V platforms.

  • Project
    amd/RyzenAI-SW

    A suite of tools and runtime libraries for optimizing and deploying AI inference on AMD Ryzen AI-based PCs to maximize hardware acceleration.