Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate

TL;DR

Hugging Face shows that the 176‑billion‑parameter BLOOM model can generate tokens in under 1 ms when run with DeepSpeed‑Inference tensor parallelism (or achieve comparable speeds with Accelerate pipeline parallelism) on a single node of eight 80 GB A100 GPUs, and also provides quantized 8‑bit alternatives that halve memory usage.

---

Hardware Requirements and Setup

• Optimal configuration: 8 × 80 GB A100 GPUs (352 GB bf16 weights). Alternatives include 2 × 8 × 40 GB A100s, 2 × 8 × 48 GB A6000s, or 24 × 32 GB V100s.
• Single‑node advantage: Intra‑node GPU interconnects are faster than inter‑node links, typically yielding higher throughput.
• Lower‑end options: CPU or NVMe off‑load can run BLOOM on smaller GPUs, but generation latency increases substantially.
• Quantized inference: 8‑bit models (via BitsAndBytes) require roughly half the GPU memory at a modest throughput penalty.

---

Benchmark Environment

• Node: Jean Zay HPC, 8 × 80 GB A100, 512 GB CPU RAM, GPFS storage (~3 GB/s read speed).
• Task: Greedy generation of 100 tokens (max_length=100, do_sample=False) with a short prompt and KV‑cache enabled.
• Metrics: Model loading time (seconds) and per‑token throughput (ms per token = wall‑time ÷ (batch × tokens)).

---

Model Loading Times

Solution	Load Time (s)
Accelerate	121
DeepSpeed‑Inference (shard‑int8)	61
DeepSpeed‑Inference (shard‑fp16)	60
DeepSpeed‑Inference (unsharded)	662
DeepSpeed‑ZeRO	462

Pre‑sharded DeepSpeed checkpoints load in ~1 minute, while unsharded checkpoints can take >10 minutes.

---

Token‑Generation Throughput (8 × 80 GB A100)

Solution (dtype)	Batch 1	Batch 8	Batch 16	Batch 32	Batch 64	Batch 128	Batch 256	Batch 512
Accelerate bf16	230.38 ms	31.78 ms	17.84 ms	10.89 ms	OOM	–	–	–
Accelerate int8	286.56 ms	40.92 ms	22.65 ms	13.27 ms	OOM	–	–	–
DeepSpeed‑Inference fp16	44.02 ms	5.70 ms	3.01 ms	1.68 ms	1.00 ms	0.69 ms	OOM	–
DeepSpeed‑Inference int8	89.09 ms	11.44 ms	5.88 ms	3.09 ms	1.71 ms	1.02 ms	0.71 ms	OOM
DeepSpeed‑ZeRO bf16	283 ms	34.88 ms	OOM	–	–	–	–	–

Key observations

• DeepSpeed‑Inference achieves sub‑1 ms per token at batch 128 thanks to tensor parallelism (TP) and custom fused CUDA kernels.
• Accelerate, using naive pipeline parallelism (PP), reaches ~10 ms per token at batch 32 but cannot exceed GPU memory limits beyond batch 64.
• Quantized int8 runs halve memory consumption; DeepSpeed‑Inference int8 still reaches ~1 ms per token at batch 128, while Accelerate int8 runs out of memory earlier.

---

Quantized 8‑bit Throughput (4 × 80 GB A100)

Solution	Batch 1	Batch 8	Batch 16	Batch 32	Batch 64	Batch 128
Accelerate int8	284.15 ms	40.14 ms	21.97 ms	OOM	–	–
DeepSpeed‑Inference int8	156.51 ms	20.11 ms	10.38 ms	5.50 ms	2.96 ms	OOM

---

Solution Details

HuggingFace Accelerate

• Approach: Loads model weights lazily onto devices based on layer size and available memory; uses simple pipeline parallelism where only one GPU is active per layer.
• Pros: Works out‑of‑the‑box on any hardware configuration; can offload to CPU or disk when GPU memory is insufficient.
• Cons: GPUs idle during most of the forward pass, limiting peak throughput; larger batch sizes quickly hit OOM.
• Usage:

  pip install transformers>=4.21.3 accelerate>=0.12.0
  python bloom-inference-scripts/bloom-accelerate-inference.py \
    --name bigscience/bloom --batch_size 1 --benchmark
  # 8‑bit quantized run
  pip install bitsandbytes
  python bloom-inference-scripts/bloom-accelerate-inference.py \
    --name bigscience/bloom --dtype int8 --batch_size 1 --benchmark
  

DeepSpeed‑Inference

• Approach: Tensor‑parallelism splits each layer across GPUs; custom fused kernels reduce memory copies and kernel launches.
• Performance drivers:
  1. TP vs PP – all GPUs compute simultaneously, increasing utilization.
  2. Fused kernels – lower memory overhead and fewer kernel launches.
• Pre‑sharded checkpoints (microsoft/bloom-deepspeed-inference-fp16) load in ~1 minute; non‑sharded checkpoints load in 10‑20 minutes but run at the same speed after loading.
• Quantized int8 (microsoft/bloom-deepspeed-inference-int8) halves memory needs and still attains sub‑1 ms per token at batch 128 on 8 × 80 GB A100s, or works on 4 × 80 GB A100s.
• Usage:

  pip install deepspeed>=0.7.3
  # Fast TP‑pre‑sharded fp16
  deepspeed --num_gpus 8 bloom-inference-scripts/bloom-ds-inference.py \
    --name microsoft/bloom-deepspeed-inference-fp16
  # Original checkpoint (slower load)
  deepspeed --num_gpus 8 bloom-inference-scripts/bloom-ds-inference.py \
    --name bigscience/bloom
  # 8‑bit version (half memory)
  deepspeed --num_gpus 8 bloom-inference-scripts/bloom-ds-inference.py \
    --name microsoft/bloom-deepspeed-inference-int8 --dtype int8
  

DeepSpeed‑ZeRO Inference

• Approach: Shards model states across GPUs (ZeRO‑3 style) and can run multiple independent generation streams in parallel, yielding an effective speedup proportional to the number of GPUs.
• Limitations: The provided script runs the same input on all GPUs; achieving the theoretical 8‑× or 16‑× speedup requires custom per‑GPU streams.
• Off‑loading options: CPU‑offload or NVMe‑offload enable inference on a single GPU at the cost of large latency.
• Usage:

  pip install deepspeed
  # Multi‑GPU ZeRO inference
  deepspeed --num_gpus 8 bloom-inference-scripts/bloom-ds-zero-inference.py \
    --name bigscience/bloom --batch_size 1 --benchmark
  # CPU off‑load (single GPU)
  deepspeed --num_gpus 1 bloom-inference-scripts/bloom-ds-zero-inference.py \
    --name bigscience/bloom --batch_size 8 --cpu_offload --benchmark
  # NVMe off‑load (single GPU)
  deepspeed --num_gpus 1 bloom-inference-scripts/bloom-ds-zero-inference.py \
    --name bigscience/bloom --batch_size 8 \
    --nvme_offload_path=/path/to/nvme_offload --benchmark
  

---

Community Server and Client Extensions

• Server implementations:
  • Mayank Mishra packaged the demo scripts into a ready‑to‑run web server.
  • Nicolas Patry created a high‑performance Rust‑based server.
• Client‑side projects:
  • Thomas Wang is developing a custom CUDA‑kernel‑accelerated BLOOM model.
  • The HuggingFace JAX team released a JAX inference backend for BLOOM.
• Keeping up‑to‑date: The transformers-bloom-inference repository aggregates the latest scripts and server implementations.

---

Practical Takeaways

• For maximum raw throughput on a single node, use DeepSpeed‑Inference with a pre‑sharded fp16 checkpoint; expect sub‑1 ms per token at batch 128.
• When GPU memory is limited, switch to 8‑bit quantization via BitsAndBytes (DeepSpeed or Accelerate) to halve memory requirements with a modest speed loss.
• Accelerate remains the most flexible solution for heterogeneous hardware, but its pipeline parallelism yields higher latency than DeepSpeed’s tensor parallelism.
• ZeRO inference offers a path to scale across many GPUs or to run on a single GPU with off‑loading, useful when large GPU clusters are unavailable.

---

“The highest batch size we were able to run without OOM was 128 in this case.” – Hugging Face blog, describing DeepSpeed‑Inference int8 on 8 × 80 GB A100s.

---

This post reflects the state of BLOOM inference performance as of September 2022. For newer optimizations, consult the transformers-bloom-inference GitHub repository.