Faster Stable Diffusion with Core ML on iPhone, iPad, and Mac

TL;DR

Apple and Hugging Face have introduced new Core ML optimizations, specifically 6-bit palettization and updated attention layer implementations, to make Stable Diffusion run faster and with lower memory overhead on iPhone, iPad, and Mac. These improvements allow for significant compression of model weights with minimal loss in precision, enabling more efficient on-device generative AI.

Core ML Optimizations and 6-Bit Palettization

Core ML leverages the CPU, GPU, and Neural Engine of Apple devices to run machine learning models on-device. To improve efficiency for large models like Stable Diffusion, Apple has introduced a new compression technique called 6-bit palettization via the coremltools.optimize submodule.

How Palettization Works

Palettization is a form of quantization that compresses model weights from 16-bit floating-point representations to 6 bits per parameter. It works similarly to color palettes in computer graphics: a fixed number of colors (the palette) is defined, and image colors are replaced with indices of the closest available colors in that palette.

Memory and Execution Efficiency

In previous Core ML versions, compressed weights were uncompressed upon loading from disk, meaning memory usage remained equal to the uncompressed model size. The updated framework now converts palletized weights on the fly as inference progresses from layer to layer. This approach is faster because it reduces the volume of memory transfers, which are the primary bottleneck in execution, rather than increasing the computational load of uncompression.

Updated Stable Diffusion Implementation

Apple's ml-stable-diffusion repository, based on the diffusers library, has been updated to incorporate these new optimizations:

  • Quantization Support: Users can now quantize models to 8, 6, 4, or 2 bits using the --quantize-nbits flag. 6-bit quantization is recommended as the optimal balance between precision and performance.
  • Attention Layer Optimizations: A new method called SPLIT_EINSUM_V2 splits query sequences into chunks of 512 to avoid creating large intermediate tensors. This optimization can improve performance by 10% to 30% on the Neural Engine.

Available Optimized Models

Six-bit palettized versions of the following official Stable Diffusion models are now available on the Hugging Face Hub:

Model Uncompressed 6-bit Palettized
Stable Diffusion 1.4 Core ML float16 Core ML 6-bit palettized
Stable Diffusion 1.5 Core ML float16 Core ML 6-bit palettized
Stable Diffusion 2 base Core ML float16 Core ML 6-bit palettized
Stable Diffusion 2.1 base Core ML float16 Core ML 6-bit palettized

Deployment and Custom Model Conversion

System Requirements

To utilize 6-bit models, devices must run development versions of iOS/iPadOS 17 or macOS 14 (Sonoma), as these contain the necessary updated Core ML framework.

Converting Custom Models

Developers can convert fine-tuned or Dreambooth models using the ml-stable-diffusion repo. The process involves:

  1. Using coremltools 7.0 beta and Xcode 15.0 beta.
  2. Running the torch2coreml conversion script with the --quantize-nbits 6 flag.
  3. Choosing between ORIGINAL attention implementation (typically better for macOS) and SPLIT_EINSUM_V2 (typically faster for iOS).

Advanced Compression: Training-Time Quantization

While 6-bit palettization is an example of post-training compression, coremltools also introduces training-time compression. This allows developers to fine-tune a model while simultaneously performing weight compression. By using a differentiable algorithm for weight clustering, the quantization table can be optimized during training to minimize loss, potentially enabling 4-bit or 2-bit compression with lower quality degradation than post-training methods.

Sources