Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate

TL;DR

Hugging Face shows that the 176‑billion‑parameter BLOOM model can generate tokens in under 1 ms when run with DeepSpeed‑Inference tensor parallelism (or achieve comparable speeds with Accelerate pipeline parallelism) on a single node of eight 80 GB A100 GPUs, and also provides quantized 8‑bit alternatives that halve memory usage.

Hardware Requirements and Setup

  • Optimal configuration: 8 × 80 GB A100 GPUs (352 GB bf16 weights). Alternatives include 2 × 8 × 40 GB A100s, 2 × 8 × 48 GB A6000s, or 24 × 32 GB V100s.
  • Single‑node advantage: Intra‑node GPU interconnects are faster than inter‑node links, typically yielding higher throughput.
  • Lower‑end options: CPU or NVMe off‑load can run BLOOM on smaller GPUs, but generation latency increases substantially.
  • Quantized inference: 8‑bit models (via BitsAndBytes) require roughly half the GPU memory at a modest throughput penalty.

Benchmark Environment

  • Node: Jean Zay HPC, 8 × 80 GB A100, 512 GB CPU RAM, GPFS storage (~3 GB/s read speed).
  • Task: Greedy generation of 100 tokens (max_length=100, do_sample=False) with a short prompt and KV‑cache enabled.
  • Metrics: Model loading time (seconds) and per‑token throughput (ms per token = wall‑time ÷ (batch × tokens)).

Model Loading Times

Solution Load Time (s)
Accelerate 121
DeepSpeed‑Inference (shard‑int8) 61
DeepSpeed‑Inference (shard‑fp16) 60
DeepSpeed‑Inference (unsharded) 662
DeepSpeed‑ZeRO 462

Pre‑sharded DeepSpeed checkpoints load in ~1 minute, while unsharded checkpoints can take >10 minutes.

Token‑Generation Throughput (8 × 80 GB A100)

Solution (dtype) Batch 1 Batch 8 Batch 16 Batch 32 Batch 64 Batch 128 Batch 256 Batch 512
Accelerate bf16 230.38 ms 31.78 ms 17.84 ms 10.89 ms OOM – – –
Accelerate int8 286.56 ms 40.92 ms 22.65 ms 13.27 ms OOM – – –
DeepSpeed‑Inference fp16 44.02 ms 5.70 ms 3.01 ms 1.68 ms 1.00 ms 0.69 ms OOM –
DeepSpeed‑Inference int8 89.09 ms 11.44 ms 5.88 ms 3.09 ms 1.71 ms 1.02 ms 0.71 ms OOM
DeepSpeed‑ZeRO bf16 283 ms 34.88 ms OOM – – – – –

Key observations

  • DeepSpeed‑Inference achieves sub‑1 ms per token at batch 128 thanks to tensor parallelism (TP) and custom fused CUDA kernels.
  • Accelerate, using naive pipeline parallelism (PP), reaches ~10 ms per token at batch 32 but cannot exceed GPU memory limits beyond batch 64.
  • Quantized int8 runs halve memory consumption; DeepSpeed‑Inference int8 still reaches ~1 ms per token at batch 128, while Accelerate int8 runs out of memory earlier.

Quantized 8‑bit Throughput (4 × 80 GB A100)

Solution Batch 1 Batch 8 Batch 16 Batch 32 Batch 64 Batch 128
Accelerate int8 284.15 ms 40.14 ms 21.97 ms OOM – –
DeepSpeed‑Inference int8 156.51 ms 20.11 ms 10.38 ms 5.50 ms 2.96 ms OOM

Solution Details

HuggingFace Accelerate

  • Approach: Loads model weights lazily onto devices based on layer size and available memory; uses simple pipeline parallelism where only one GPU is active per layer.
  • Pros: Works out‑of‑the‑box on any hardware configuration; can offload to CPU or disk when GPU memory is insufficient.
  • Cons: GPUs idle during most of the forward pass, limiting peak throughput; larger batch sizes quickly hit OOM.
  • Usage:
    pip install transformers>=4.21.3 accelerate>=0.12.0
python bloom-inference-scripts/bloom-accelerate-inference.py \
  --name bigscience/bloom --batch_size 1 --benchmark
# 8‑bit quantized run
pip install bitsandbytes
python bloom-inference-scripts/bloom-accelerate-inference.py \
  --name bigscience/bloom --dtype int8 --batch_size 1 --benchmark

DeepSpeed‑Inference

  • Approach: Tensor‑parallelism splits each layer across GPUs; custom fused kernels reduce memory copies and kernel launches.
  • Performance drivers:
    1. TP vs PP – all GPUs compute simultaneously, increasing utilization.
    2. Fused kernels – lower memory overhead and fewer kernel launches.
  • Pre‑sharded checkpoints (microsoft/bloom-deepspeed-inference-fp16) load in ~1 minute; non‑sharded checkpoints load in 10‑20 minutes but run at the same speed after loading.
  • Quantized int8 (microsoft/bloom-deepspeed-inference-int8) halves memory needs and still attains sub‑1 ms per token at batch 128 on 8 × 80 GB A100s, or works on 4 × 80 GB A100s.
  • Usage:
    pip install deepspeed>=0.7.3
# Fast TP‑pre‑sharded fp16
deepspeed --num_gpus 8 bloom-inference-scripts/bloom-ds-inference.py \
  --name microsoft/bloom-deepspeed-inference-fp16
# Original checkpoint (slower load)
deepspeed --num_gpus 8 bloom-inference-scripts/bloom-ds-inference.py \
  --name bigscience/bloom
# 8‑bit version (half memory)
deepspeed --num_gpus 8 bloom-inference-scripts/bloom-ds-inference.py \
  --name microsoft/bloom-deepspeed-inference-int8 --dtype int8

DeepSpeed‑ZeRO Inference

  • Approach: Shards model states across GPUs (ZeRO‑3 style) and can run multiple independent generation streams in parallel, yielding an effective speedup proportional to the number of GPUs.
  • Limitations: The provided script runs the same input on all GPUs; achieving the theoretical 8‑× or 16‑× speedup requires custom per‑GPU streams.
  • Off‑loading options: CPU‑offload or NVMe‑offload enable inference on a single GPU at the cost of large latency.
  • Usage:
    pip install deepspeed
# Multi‑GPU ZeRO inference
deepspeed --num_gpus 8 bloom-inference-scripts/bloom-ds-zero-inference.py \
  --name bigscience/bloom --batch_size 1 --benchmark
# CPU off‑load (single GPU)
deepspeed --num_gpus 1 bloom-inference-scripts/bloom-ds-zero-inference.py \
  --name bigscience/bloom --batch_size 8 --cpu_offload --benchmark
# NVMe off‑load (single GPU)
deepspeed --num_gpus 1 bloom-inference-scripts/bloom-ds-zero-inference.py \
  --name bigscience/bloom --batch_size 8 \
  --nvme_offload_path=/path/to/nvme_offload --benchmark

Community Server and Client Extensions

  • Server implementations:
    • Mayank Mishra packaged the demo scripts into a ready‑to‑run web server.
    • Nicolas Patry created a high‑performance Rust‑based server.
  • Client‑side projects:
    • Thomas Wang is developing a custom CUDA‑kernel‑accelerated BLOOM model.
    • The HuggingFace JAX team released a JAX inference backend for BLOOM.
  • Keeping up‑to‑date: The transformers-bloom-inference repository aggregates the latest scripts and server implementations.

Practical Takeaways

  • For maximum raw throughput on a single node, use DeepSpeed‑Inference with a pre‑sharded fp16 checkpoint; expect sub‑1 ms per token at batch 128.
  • When GPU memory is limited, switch to 8‑bit quantization via BitsAndBytes (DeepSpeed or Accelerate) to halve memory requirements with a modest speed loss.
  • Accelerate remains the most flexible solution for heterogeneous hardware, but its pipeline parallelism yields higher latency than DeepSpeed’s tensor parallelism.
  • ZeRO inference offers a path to scale across many GPUs or to run on a single GPU with off‑loading, useful when large GPU clusters are unavailable.

“The highest batch size we were able to run without OOM was 128 in this case.” – Hugging Face blog, describing DeepSpeed‑Inference int8 on 8 × 80 GB A100s.

This post reflects the state of BLOOM inference performance as of September 2022. For newer optimizations, consult the transformers-bloom-inference GitHub repository.

Sources