yandex/YaFSDP

YaFSDP: Yet another Fully Sharded Data Parallel

What it solves

YaFSDP is designed to optimize the pre-training of Large Language Models (LLMs), specifically targeting the overhead associated with communications and memory operations in Sharded Data Parallelism. It aims to provide a faster alternative to PyTorch's FSDP (Fully Sharded Data Parallel) by improving GPU utilization and performance under high memory pressure.

How it works

It implements a Sharded Data Parallelism framework specifically tuned for transformer-like neural network architectures. By reducing communication and memory operation overheads, it allows for faster iteration times during training runs across multiple GPUs.

Who it’s for

ML engineers and researchers training large-scale transformer models (such as Llama 2 and Llama 3) across large GPU clusters (ranging from 64 to 256 devices) using the Hugging Face stack.

Highlights

  • Performance Gains: Up to 20% faster pre-training for LLMs compared to standard FSDP.
  • Scalability: Benchmarked on models from 7B to 70B parameters across up to 256 A100 GPUs.
  • Scalability: Supports causal pre-training and supervised fine-tuning (SFT).
  • Integration: Works with the Hugging Face ecosystem via provided examples and Docker images.

Related

  • Dispatch
    Accelerate Large Model Training using PyTorch Fully Sharded Data Parallel

    Hugging Face integrates PyTorch Fully Sharded Data Parallel (FSDP) into the Accelerate library, enabling the training of massive models by sharding optimizer states, gradients, and parameters across GPUs with optional CPU offloading.

  • Dispatch
    Accelerate ND-Parallel: Efficient Multi-GPU Training Guide

    Hugging Face has integrated ND-Parallelism into Accelerate and Axolotl, allowing users to combine Data, Fully Sharded Data, Tensor, and Context parallelism strategies to optimize multi-GPU training for massive models.

  • Dispatch
    Fine-tuning Llama 2 70B using PyTorch FSDP

    Hugging Face demonstrates how to fine-tune Llama 2 70B using PyTorch Fully Sharded Data Parallelism (FSDP) and Accelerate to overcome CPU RAM bottlenecks and optimize VRAM usage.

  • Project
    ServiceNow/Fast-LLM

    A high-performance library for training large language models that uses 3D parallelism and optimized kernels to reduce training time and cost.

  • Dispatch
    Ulysses Sequence Parallelism for Million-Token Context Training

    Hugging Face has integrated Ulysses Sequence Parallelism into Accelerate, Transformers, and TRL, enabling the training of LLMs with million-token contexts by distributing attention computation across multiple GPUs.