Fine-tuning Stable Diffusion on Intel Sapphire Rapids CPUs using Hugging Face Diffusers

Hugging Face shows how to fine-tune a Stable Diffusion model on Intel Sapphire Rapids CPUs using textual inversion with just five example images, demonstrating CPU‑based fine‑tuning via Intel Extension for PyTorch and oneCCL.

Cluster Setup

Four Intel Xeon Platinum 8480+ servers, each with two Sapphire Rapids CPUs, provide 224 logical cores (56 physical cores and 112 threads per socket) for distributed fine‑tuning. The lscpu output confirms the architecture, core counts, thread counts, and the presence of the AMX instruction set. The servers are accessed via IP addresses 192.168.20.2, 192.168.21.2, 192.168.22.2, and 192.168.23.2, listed in a nodefile for MPI.

Software Environment and Dependencies

The environment uses Python 3.9, PyTorch CPU build, Transformers, Accelerate version 0.19.0, oneCCL bindings for PyTorch, Intel Extension for PyTorch (IPEX), and gperftools for tcmalloc. After creating a conda environment named diffuser, the dependencies are installed with pip and conda. The Diffusers repository is cloned from GitHub and installed from source on each node.

Preparing the Data and Model Code

Five example images are downloaded from the sd-concepts-library/dicoo dataset and placed in /home/devcloud/dicoo on every node. The textual inversion script (examples/textual_inversion/textual_inversion.py) is edited to import intel_extension_for_pytorch and apply ipex.optimize to both the U‑Net and VAE models, using the training dtype (bf16). This modification must be applied on all nodes before launching the job.

Configuring the Distributed Job

Environment variables are set on the primary node and propagated to workers: I_MPI_HYDRA_IFACE selects the network interface, oneccl_bindings_for_pytorch_path locates the oneCCL bindings, LD_PRELOAD includes OpenMP and tcmalloc libraries, CCL_ATL_TRANSPORT is set to ofi, and CCL_WORKER_COUNT is set to 1. The model name is defined as runwayml/stable-diffusion-v1-5 and the data directory as /home/devcloud/dicoo.

Running the Fine‑tuning

The fine‑tuning job is launched with mpirun -f nodefile -n 16 -ppn 4, which starts 16 MPI ranks (four per node). The accelerate launch command runs the modified textual inversion script with the following arguments: --pretrained_model_name_or_path=$MODEL_NAME --train_data_dir=$DATA_DIR --learnable_property="object" --placeholder_token="<dicoo>" --initializer_token="toy" --resolution=512 --train_batch_size=1 --seed=7 --gradient_accumulation_steps=1 --max_train_steps=200 --learning_rate=2.0e-03 --scale_lr --lr_scheduler="constant" --lr_warmup_steps=0 --output_dir=./textual_inversion_output --mixed_precision bf16 --save_as_full_pipeline. Training for 200 steps completes in about five minutes, as indicated by the busy cluster screenshot.

Troubleshooting Distributed Training

If the distributed job fails, the recommended approach is to log into each node, replicate the primary node’s environment, and run the training script locally with the same arguments. Successful local runs indicate that the node’s dependencies and data paths are correct; otherwise, missing dependencies or incorrect image locations are revealed. After confirming all nodes work locally, return to the primary node to double‑check the nodefile, environment variables, and the mpirun command.

Generating Images with the Fine‑tuned Model

After training, the model is exported and optimized with Optimum Intel and OpenVINO: OVStableDiffusionPipeline.from_pretrained(model_id, export=True) followed by ov_pipe.reshape(batch_size=5, height=512, width=512, num_images_per_prompt=1) and ov_pipe.save_pretrained. Loading the optimized pipeline (OVStableDiffusionPipeline.from_pretrained(model_id, num_inference_steps=20)) and prompting with ["a yellow <dicoo> robot at the beach, high quality" * 5] produces five images in under five seconds on a single CPU, as noted in a previous post. The generated images show that the model learned the new concept—specifically that dicoos have glasses—from only five training images. A longer run of 3,000 steps (about one hour) yields higher‑quality results.

Conclusion

Xeon Sapphire Rapids CPUs provide an affordable, widely available, and reusable alternative to GPUs for both fine‑tuning and inference of Stable Diffusion models. Because the same CPUs can serve other workloads such as web servers or databases, they offer a flexible option for enterprises looking to adopt generative AI without investing in specialized hardware.