huggingface/accelerate

What it solves

PyTorch users often find writing and maintaining the boilerplate code required for distributed training (multi-GPU, TPU, or mixed precision) tedious and complex. 🤗 Accelerate removes this burden by abstracting the hardware-specific setup while allowing the user to maintain full control over their training loop.

How it works

Accelerate provides a thin wrapper around PyTorch. By adding a few lines of code to a standard training script—specifically using the Accelerator class—users can prepare their models, optimizers, and data loaders for any device configuration. It handles device placement and the backward pass automatically.

Additionally, it offers an optional CLI tool (accelerate config and accelerate launch) to configure the environment and launch scripts without needing to manually manage torch.distributed.run or TPU launchers. It also supports advanced integrations like DeepSpeed, FSDP, and Megatron-LM.

Who it’s for

Developers and researchers who prefer writing their own PyTorch training loops but want to run their code on any hardware configuration (single CPU, multi-GPU, TPU) without writing hardware-specific boilerplate.

Highlights

• Hardware Agnostic: Supports single/multi-CPU, single/multi-GPU, and TPUs across one or multiple nodes.
• Mixed Precision: Built-in support for FP16, BFloat16, and FP8 mixed precision.
• Minimal Code Changes: Integrates into existing PyTorch scripts with very few modifications.
• CLI Tooling: Simplifies environment configuration and launch process via a dedicated CLI.
• Notebook Support: Includes a notebook_launcher for distributed training in environments like Colab or Kaggle.