mpi4jax/mpi4jax

Zero-copy MPI communication of JAX arrays, for turbo-charged HPC applications in Python :zap:

What it solves

It addresses the limited multi-host capabilities of the JAX framework, allowing users to scale JAX-based simulations to entire CPU and GPU clusters.

How it works

It enables zero-copy, multi-host communication of JAX arrays. This functionality is integrated directly into JAX's just-in-time (jit) compilation, meaning communication can occur from within jitted code and directly from GPU memory.

Who it’s for

Researchers and developers performing scientific computing workloads that require scaling across multiple hosts and GPUs using JAX.

Highlights

  • Zero-copy communication of JAX arrays.
  • Compatible with jax.jit for high-performance execution.
  • Supports multi-host scaling across CPU and GPU clusters.
  • Supports differentiating through certain MPI operations for differentiable programming.

Related

  • Project
    patrick-kidger/lineax

    A JAX library for linear solves and linear least squares, providing numerically stable routines for solving linear systems, including support for linear operators and GPU/TPU acceleration.

  • Project
    jax-ml/jax-triton

    An integration library that allows JAX users to call custom Triton kernels for high-performance GPU operations within JAX programs.

  • Project
    google-deepmind/gemma

    A JAX library for deploying and fine-tuning the Gemma family of open-weights large language models, supporting multi-modal conversations and multiple hardware backends.

  • Project
    blackjax-devs/blackjax

    A modular library of samplers for JAX that enables composable Bayesian inference on CPU, GPU, and TPU without requiring a full probabilistic programming language.

  • Project
    jax-ml/jax

    A Python library for accelerator-oriented array computation and program transformation, designed for high-performance numerical computing and large-scale machine learning.