Stable Diffusion Core ML on Apple Silicon – How to Run and Optimize

TL;DR

Hugging Face published Core ML versions of Stable Diffusion (v1.4, v1.5, v2‑base, v2.1‑base) that run locally on Apple Silicon, and provided Python and Swift inference scripts plus a ready‑to‑use Mac App Store app.

Available Core ML Checkpoints

  • Stable Diffusion v1.4 – converted model at apple/coreml-stable-diffusion-v1-4.
  • Stable Diffusion v1.5 – converted model at apple/coreml-stable-diffusion-v1-5.
  • Stable Diffusion v2 base – converted model at apple/coreml-stable-diffusion-2-base.
  • Stable Diffusion v2.1 base – converted model at apple/coreml-stable-diffusion-2-1-base.

All checkpoints are hosted on the Hugging Face Hub and can run on any Apple Silicon device using the CPU, GPU, or Apple Neural Engine (NE). Different variants (attention implementation and packaging) are provided to suit hardware and language‑specific needs.

Performance Variants and Recommendations

  • Attention implementations – original (CPU/GPU only, sometimes faster) vs. split_einsum (compatible with CPU, GPU, and NE). Choose based on device capabilities.
  • Packaging – packages for Python inference; compiled for Swift (splits the large UNet into multiple .mlmodelc files for iOS/iPadOS compatibility).
  • Best‑case benchmark – on a MacBook Pro (M1 Max, 32 GPU cores, 64 GB RAM) with macOS Ventura 13.1 Beta 4, using original attention and all compute units, Stable Diffusion v1.4 generates an image in 18 seconds.

⚠️ Note macOS Ventura 13.1 introduced Core ML improvements required for these models; earlier macOS versions may produce black images or slower runtimes.

Python Inference Workflow

Prerequisites

pip install huggingface_hub
pip install git+https://github.com/apple/ml-stable-diffusion

Download a Checkpoint

from huggingface_hub import snapshot_download
from pathlib import Path

repo_id = "apple/coreml-stable-diffusion-v1-4"
variant = "original/packages"
model_path = Path("./models") / (repo_id.split('/')[-1] + "_" + variant.replace('/', '_'))
snapshot_download(repo_id, allow_patterns=f"{variant}/*", local_dir=model_path, local_dir_use_symlinks=False)
print(f"Model downloaded at {model_path}")

Run Inference

python -m python_coreml_stable_diffusion.pipeline \
  --prompt "a photo of an astronaut riding a horse on mars" \
  -i models/coreml-stable-diffusion-v1-4_original_packages \
  -o ./output.png \
  --compute-unit ALL \
  --seed 93
  • --compute-unit options: ALL, CPU_AND_GPU, CPU_ONLY, CPU_AND_NE.
  • To use a different checkpoint, add --model-version <hub-id> (e.g., runwayml/stable-diffusion-v1-5).

Swift Inference Workflow

Download a Compiled Checkpoint

from huggingface_hub import snapshot_download
from pathlib import Path

repo_id = "apple/coreml-stable-diffusion-v1-4"
variant = "original/compiled"
model_path = Path("./models") / (repo_id.split('/')[-1] + "_" + variant.replace('/', '_'))
snapshot_download(repo_id, allow_patterns=f"{variant}/*", local_dir=model_path, local_dir_use_symlinks=False)
print(f"Model downloaded at {model_path}")

Run Inference

git clone https://github.com/apple/ml-stable-diffusion
cd ml-stable-diffusion
swift run StableDiffusionSample \
  --resource-path models/coreml-stable-diffusion-v1-4_original_compiled \
  --compute-units all \
  "a photo of an astronaut riding a horse on mars"
  • --compute-units values: all, cpuOnly, cpuAndGPU, cpuAndNeuralEngine.
  • Compiled models load faster on app startup, yielding slightly lower latency for subsequent generations.

Converting Custom Models (Bring‑Your‑Own‑Model)

If you have fine‑tuned or otherwise customized a Stable Diffusion model (e.g., DreamBooth, Textual Inversion), you must run Apple’s conversion script yourself. Follow the official instructions in the Apple repo: https://github.com/apple/ml-stable-diffusion#converting-models-to-coreml.

Next Steps and Community Opportunities

  • Build native macOS, iPhone, and iPad applications that generate images entirely on‑device.
  • Integrate additional schedulers in Swift for faster sampling.
  • Expand the pipeline to other generative tasks (e.g., inpainting, upscaling).
  • Explore quantization and other optimizations to further reduce latency and memory usage.

The released Core ML checkpoints and tooling lower the barrier for developers to harness Stable Diffusion on Apple Silicon, opening the door to privacy‑preserving, offline generative AI experiences.

Sources