Accelerating Stable Diffusion Inference on Intel Sapphire Rapids CPUs

TL;DR

Hugging Face demonstrates that Stable Diffusion inference can be accelerated from 32.3 s to ~5 s on Intel Sapphire Rapids (Xeon Sapphire Rapids) CPUs by combining Optimum Intel/OpenVINO, system‑level memory and threading optimizations, Intel Extension for PyTorch (IPEX) with BF16/AMX, and the DPMSolverMultistepScheduler.

The Diffusers library – baseline performance

The diffusers library provides a simple Python API for Stable Diffusion. Using the default float32 pipeline on an Amazon EC2 r7iz.metal-16xl instance (64 vCPU, 512 GB RAM, Ubuntu 20.04) yields an average latency of 32.3 seconds per image (20 inference steps). The same hardware generation (Sapphire Rapids) is already faster than the previous‑generation Ice Lake Xeons, which take about 45 seconds for the identical workload.

Optimum Intel and OpenVINO – 2× speedup

Optimum Intel offers a drop‑in replacement for StableDiffusionPipeline called OVStableDiffusionPipeline. Installing with pip install optimum[openvino] enables automatic conversion of the PyTorch model to OpenVINO format and inference in bfloat16.

from optimum.intel.openvino import OVStableDiffusionPipeline
ov_pipe = OVStableDiffusionPipeline.from_pretrained(model_id, export=True)
latency = elapsed_time(ov_pipe, prompt)
print(latency)  # → ~16.7 s

The OpenVINO pipeline reduces latency to 16.7 s, a 2× improvement over the vanilla Diffusers baseline.

Fixed‑shape optimization – additional 3.5×

OpenVINO also supports static input shapes. By reshaping the pipeline to a fixed resolution (e.g., 512 × 512) the latency drops further to 4.7 seconds.

ov_pipe.reshape(batch_size=1, height=512, width=512, num_images_per_prompt=1)
latency = elapsed_time(ov_pipe, prompt)

Combined with Sapphire Rapids, this yields almost a 10× speedup compared to Ice Lake.

System‑level optimizations – 3× faster without code changes

Memory allocation and threading affect the large multi‑gigabyte models used by Stable Diffusion. The following system tweaks were applied on the same EC2 instance:

  • Install and preload jemalloc for high‑performance memory management.
  • Install intel‑mkl and preload libiomp5.so to enable Intel OpenMP runtime.
  • Set OMP_NUM_THREADS=32 to match the core count.
  • Pin the Python process to a subset of cores with numactl -C 0-31.
sudo apt-get install -y libjemalloc-dev intel-mkl
export LD_PRELOAD=$LD_PRELOAD:/usr/lib/x86_64-linux-gnu/libjemalloc.so
export LD_PRELOAD=$LD_PRELOAD:/usr/lib/x86_64-linux-gnu/libiomp5.so
export OMP_NUM_THREADS=32
numactl -C 0-31 python sd_blog_1.py

These changes reduce the vanilla Diffusers latency from 32.3 s to 11.8 s, roughly a 3× gain.

IPEX and BF16 – another 2× boost

The Intel Extension for PyTorch (IPEX) leverages AVX‑512 VNNI and AMX (Advanced Matrix Extensions) present on Sapphire Rapids. After installing intel_extension_for_pytorch==1.13.100, each pipeline component (UNet, VAE, text encoder, safety checker) is converted to channels‑last layout and optimized with IPEX in bfloat16 mode.

import torch, intel_extension_for_pytorch as ipex
# Convert modules to channels‑last
pipe.unet = pipe.unet.to(memory_format=torch.channels_last)
# ... repeat for vae, text_encoder, safety_checker
# Optimize with IPEX
pipe.unet = ipex.optimize(pipe.unet.eval(), dtype=torch.bfloat16, inplace=True, sample_input=input_example)
# Run under autocast
with torch.cpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
    latency = elapsed_time(pipe, prompt)
    print(latency)  # → ~5.4 s

Latency improves from 11.8 s to 5.4 seconds, more than a 2× acceleration thanks to BF16 and AMX.

Faster scheduler – final 6.5× overall speedup

Diffusers supports interchangeable schedulers that control the denoising step schedule. The DPMSolverMultistepScheduler offers the best speed/quality trade‑off at as few as 20 steps.

from diffusers import DPMSolverMultistepScheduler
scheduler = DPMSolverMultistepScheduler.from_pretrained(model_id, subfolder="scheduler")
pipe = StableDiffusionPipeline.from_pretrained(model_id, scheduler=scheduler)
latency = elapsed_time(pipe, prompt)
print(latency)  # → ~5.05 s

With this scheduler, the final latency is 5.05 seconds, representing a 6.5× improvement over the original Sapphire Rapids baseline (32.3 s) and roughly 10× faster than Ice Lake.

Full environment snapshot

The benchmark was performed on:

  • Instance: Amazon EC2 r7iz.metal-16xl (64 vCPU, 512 GB RAM)
  • OS: Ubuntu 20.04, Linux kernel 5.15.0-1031-aws
  • Libraries: libjemalloc‑dev 5.2.1‑1, intel‑mkl 2020.0.166‑1, PyTorch 1.13.1, Intel Extension for PyTorch 1.13.1, Transformers 4.27.2, Diffusers 0.14, Accelerate 0.17.1, OpenVINO 2023.0.0.dev20230217, Optimum 1.7.1, Optimum‑Intel 1.7

Implications and next steps

Reducing Stable Diffusion inference to a few seconds on commodity CPUs opens up new use cases:

  • Real‑time image generation in customer‑facing applications without GPU infrastructure.
  • Cost‑effective synthetic data generation for training other models.
  • On‑premise deployment where data privacy or latency constraints preclude cloud GPUs.

Developers can start by following the provided GitLab demo repository, which contains the full scripts used in the blog post. For further scaling, a forthcoming Hugging Face article will cover distributed fine‑tuning of diffusion models on Sapphire Rapids clusters.

Resources

Sources