NVIDIA/nvidia-resiliency-ext

NVIDIA Resiliency Extension is a python package for framework developers and users to implement fault-tolerant features. It improves the effective training time by minimizing the downtime due to failures and interruptions.

What it solves

NVIDIA Resiliency Extension (NVRx) addresses the challenge of maintaining productivity during large-scale AI training. It prevents training runs from failing completely when hardware or software faults occur, and minimizes the loss of progress up to the same point of failure.

How it works

NVRx provides a modular set of tools for PyTorch-based workloads to handle failures automatically. It uses system-wide health checks and fault detection to identify hung ranks or slow "stragglers" (GPU/CPU performance issues) that impede efficiency. When a failure is detected, it enables in-process restarting of training without requiring the reallocation of compute nodes. It also provides frameworks for asynchronous and local checkpointing to save progress frequently and efficiently without slowing down the training process.

Who it’s for

Researchers and developers working on large-scale AI training, particularly those using PyTorch, PyTorch Lightning, or NVIDIA NeMo, who need to ensure their training runs are stable and resilient to hardware faults.

Highlights

  • Automatic Restarting: Restarts training in-job, avoiding the need to reallocate SLURM nodes.
  • Fault & Straggler Detection: Monitors GPU and CPU performance to identify hung ranks and slower nodes.
  • Efficient Checkpointing: Supports both asynchronous and local checkpointing to reduce work loss.
  • Framework Integration: Seamlessly integrates with PyTorch Lightning and NVIDIA NeMo.

Related

  • Project
    meta-pytorch/torchft

    A PyTorch library that implements per-step fault tolerance for large-scale training, allowing jobs to continue running even when individual workers or nodes fail.

  • Project
    official-stockfish/nnue-pytorch

    A PyTorch-based training framework for Stockfish's NNUE evaluation networks, enabling GPU-accelerated training and automated performance evaluation.

  • Project
    Lightning-AI/pytorch-lightning

    A deep learning framework that abstracts away the engineering boilerplate of PyTorch, enabling researchers to pretrain and finetune AI models at scale across various hardware.

  • Project
    galilai-group/stable-pretraining

    stable-pretraining is a PyTorch Lightning-based framework for foundation-model research that simplifies self-supervised, supervised, and multi-modal pretraining (SimCLR, DINO, MAE, CLIP, etc.) via a dict-based design, live evaluation callbacks, GPU-side augmentation, and SLURM-grade orchestration, with an experimental JAX backend.

  • Project
    pytorch/pytorch

    A Python-based machine learning framework that provides GPU-accelerated tensor computation and a dynamic autograd system for building deep neural networks.