deepspeedai/DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

DeepSpeed – Scalable Deep Learning Training Framework

What it is DeepSpeed is an open‑source library (built on top of PyTorch) that makes it possible to train extremely large deep‑learning models – from hundreds of millions to hundreds of billions of parameters – efficiently and on a variety of hardware (NVIDIA, AMD, Intel Gaudi, Xeon CPUs, etc.). It provides system‑level optimizations, memory‑saving techniques, and parallelism primitives so that researchers and engineers can push model size and speed without writing low‑level CUDA/HIP code.

Key capabilities

  • ZeRO family – memory‑optimizing optimizers (ZeRO, ZeRO‑Infinity) that shard model states across GPUs, enabling trillion‑parameter training.
  • 3‑D Parallelism – combines data, tensor, and pipeline parallelism for maximal hardware utilization.
  • Offload engines – SuperOffload, ZeRO‑Offload, ZenFlow, Ulysses‑Offload move data to CPU, GPU memory, or other accelerators to reduce GPU memory pressure.
  • Specialized kernels – custom C++/CUDA/HIP ops (e.g., System DMA for AMD, DeepNVMe I/O scaling) that overlap communication and computation.
  • Integration layer – ready‑to‑use adapters for Hugging Face Transformers, Accelerate, PyTorch‑Lightning, MosaicML Composer, Determined AI, MMEngine, etc.
  • Support for many hardware back‑ends – NVIDIA, AMD, Intel Gaudi, Intel Xeon/XPU, Huawei Ascend NPU, and community‑contributed accelerators.

Typical workflow

  1. Install the deepspeed pip package (or build from source for custom ops).
  2. Write a standard PyTorch training script.
  3. Add a small JSON configuration that tells DeepSpeed which parallelism and memory‑offload strategies to use.
  4. Launch with deepspeed train.py --deepspeed_config cfg.json.
  5. Use ds_report to verify which ops were compiled and which hardware features are active.

Why it matters DeepSpeed has been used to train some of the world’s largest language models – e.g., Microsoft’s 530 B‑parameter MT‑NLG, BLOOM (176 B), Jurassic‑1 (178 B) – proving that the library can scale to the frontier of AI research. Its open‑source nature lets anyone reproduce these results or experiment with new parallelism ideas.

Getting started & resources

  • Installation – pip install deepspeed (requires PyTorch ≥ 2.0 and a CUDA/ROCm compiler). Windows is supported with a few feature caveats.
  • Documentation – https://www.deepspeed.ai/ (getting‑started guides, JSON config reference, API docs).
  • Community – regular public office‑hours on Zoom, Slack workspace, and active GitHub discussions.
  • Examples – the repo ships example scripts for ZeRO‑Offload, SDMA all‑gather, long‑sequence training (Arctic Long Sequence Training), and more.

Who should use it

  • Researchers building or fine‑tuning very large transformer models.
  • Engineers needing to squeeze the most performance out of multi‑GPU or heterogeneous clusters.
  • Teams that want a drop‑in acceleration layer for existing PyTorch codebases.

All information above is taken directly from the project's README.

Related

  • Dispatch
    Accelerate Large Model Training using DeepSpeed

    Hugging Face demonstrates how to use the Accelerate library to integrate DeepSpeed's ZeRO optimizations, enabling larger batch sizes and the training of models that would otherwise exceed GPU memory.

  • Project
    ServiceNow/Fast-LLM

    A high-performance library for training large language models that uses 3D parallelism and optimized kernels to reduce training time and cost.

  • Dispatch
    Hugging Face Transformers ZeRO Integration via DeepSpeed and FairScale

    Hugging Face Transformers v4.2.0 introduces experimental support for DeepSpeed and FairScale's ZeRO optimizations, enabling the training of larger models with higher batch sizes and reduced GPU memory requirements.

  • Project
    hpcaitech/ColossalAI

    Colossal-AI is an open-source system that makes training and running large AI models cheaper and faster through advanced parallel computing techniques, supporting models like LLaMA, GPT, and Stable Diffusion across multiple GPUs.

  • Project
    huggingface/nanotron

    A library for pretraining transformer models that provides a simple, scalable API and supports 3D parallelism for efficient large-scale training.