meta-pytorch/torchft

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

What it solves

Training large-scale AI models often fails when a single GPU or node crashes, typically requiring the entire training job to be restarted from the last saved checkpoint. torchft provides per-step fault tolerance, allowing training to continue without interrupting the entire job when errors occur.

How it works

The library provides primitives for membership changes at the training step granularity. It uses a "lighthouse" server to coordinate across replica groups and determine worker health via per-step heartbeating. It also implements fault-tolerant versions of ProcessGroup and checkpoint transports that enable live recovery from healthy peers during scale-up operations.

Who it’s for

ML engineers and researchers training large models (such as Llama 3 70B) using PyTorch, specifically those utilizing replicated weight strategies like DDP or HSDP.

Highlights

  • Fault Tolerant Algorithms: Out-of-the-box support for Fault Tolerant DDP, Fault Tolerant HSDP, LocalSGD, and DiLoCo.
  • Lighthouse Coordination: A dedicated server that manages worker health and coordinates replica groups to avoid "stop-the-world" training pauses.
  • Live Recovery: Ability to recover state from healthy peers during scale-up operations using specialized checkpoint transports.
  • Integration: Designed to be integrated into existing PyTorch training scripts with minimal changes to the training loop.

Related

  • Project
    NVIDIA/nvidia-resiliency-ext

    A resiliency framework for PyTorch-based AI training that maximizes goodput by providing automatic fault detection, in-process restarting, and efficient checkpointing.

  • Project
    FareedKhan-dev/train-llm-from-scratch

    A complete from-scratch PyTorch implementation of an LLM pipeline, covering everything from raw text tokenization and Transformer architecture to SFT and advanced RLHF alignment techniques like PPO and GRPO.

  • Project
    sktime/pytorch-forecasting

    A PyTorch-based library for time series forecasting that provides a high-level API and state-of-the-art deep learning architectures like Temporal Fusion Transformers and N-BEATS.

  • Project
    Lightning-AI/pytorch-lightning

    A deep learning framework that abstracts away the engineering boilerplate of PyTorch, enabling researchers to pretrain and finetune AI models at scale across various hardware.

  • Project
    facebookresearch/multimodal

    A PyTorch library for training state-of-the-art multimodal models at scale, providing modular building blocks and pretrained weights for models like CLIP and BLIP-2.