NVIDIA/apex

What it solves

Apex provides a set of utilities to make mixed precision and distributed training in PyTorch more efficient and streamlined. It serves as a fast-track for introducing updated utilities to users before they are officially integrated into the upstream PyTorch codebase.

How it works

It implements a variety of custom C++/CUDA extensions that optimize common deep learning operations. These extensions include fused kernels for optimizers (like FusedAdam), normalization layers (like FusedLayerNorm and FusedRMSNorm), and communication primitives for distributed training (such as DistributedDataParallel and SyncBatchNorm). Users can install it as a Python-only package or build it with specific CUDA extensions to unlock maximum performance and numerical stability.

Who it’s for

Deep learning practitioners and researchers using PyTorch who need to optimize the performance, memory usage, and scaling of their model training on NVIDIA GPUs.

Highlights

• Mixed Precision Training: Tools to streamline the use of mixed precision to speed up training.
• Distributed Training: Utilities for scaling training across multiple GPUs.
• Fused Kernels: High-performance CUDA implementations of optimizers and normalization layers to reduce overhead.
• Flexible Installation: Supports various build configurations, from a simple Python-only install to a full suite of C++/CUDA extensions.