Stable Diffusion XL on Mac with Advanced Core ML Quantization

Hugging Face and Apple have released a Core ML port of Stable Diffusion XL, enabling high-quality 1024x1024 image generation on Apple Silicon Macs. The release introduces mixed-bit palettization, a compression technique that significantly reduces model size and memory requirements without the severe quality loss associated with uniform quantization.

Core ML Implementation and Performance

Stable Diffusion XL is now available in Core ML format, allowing it to run natively in Swift applications on Apple Silicon Macs running the public beta of macOS 14. The implementation currently utilizes the ORIGINAL attention implementation designed for CPU and GPU compute units. Note that the refiner stage has not yet been ported.

Hardware Performance Benchmarks

End-to-end latency and diffusion speed vary by device when using CPU_AND_GPU compute units and the ORIGINAL attention implementation:

Device End-to-End Latency (s) Diffusion Speed (iter/s)
MacBook Pro (M1 Max) 46 0.46
MacBook Pro (M2 Max) 37 0.57
Mac Studio (M1 Ultra) 25 0.89
Mac Studio (M2 Ultra) 20 1.11

Mixed-Bit Palettization Technique

Mixed-bit palettization is a post-training quantization method that optimizes model size by assigning different bit depths to different layers based on their impact on output quality. Unlike standard 6-bit palettization, which applies a uniform bit depth, mixed-bit palettization determines the optimal number of bits per layer to minimize quality degradation.

How Mixed-Bit Palettization Works

The process consists of two primary phases:

  1. Analysis Phase: The system examines each layer and tests various bit depths (1, 2, 4, and 8 bits). Quality degradation is measured by comparing the Peak Signal-to-Noise Ratio (PSNR) between the quantized model and the original float16 model using a set of inputs. The goal is to identify the lowest bit depth per layer that keeps the PSNR above a specific threshold.
  2. Application Phase: A "recipe" (a JSON dictionary specifying the bit depth for each layer) is generated during analysis and then applied to the model weights.

Compression Results

Applying a 4.5-bit average recipe to the Stable Diffusion XL UNet resulted in a 71% reduction in size, decreasing from 4.8 GB to 1.4 GB.

Technical analysis indicates that mixed-bit palettization outperforms linear 8-bit quantization at the same model size (except for 1-bit quantization). While high compression (e.g., 3.41-bit) may lead to a loss of specific prompt details—such as the disappearance of a surfboard in a "surfing dog" prompt—the 4.50-bit and 6.55-bit versions remain close to the original 16-bit quality in terms of realism.

Deployment and Conversion Resources

Users and developers can deploy Stable Diffusion XL on Mac through several provided resources:

  • Pre-converted Models: Hugging Face provides both a full-precision pipeline (apple/coreml-stable-diffusion-xl-base) and a mixed-bit palettized pipeline (apple/coreml-stable-diffusion-mixed-bit-palettization).
  • Conversion Tools: Apple's ml-stable-diffusion repository allows users to convert their own fine-tuned models. For XL models, users must use the --attention-implementation ORIGINAL flag.
  • Custom Quantization: Scripts for mixed_bit_compression_pre_analysis.py and mixed_bit_compression_apply.py are available for those wishing to generate and apply their own quantization recipes.
  • Pre-computed Recipes: To avoid the lengthy analysis phase, pre-computed recipes for Stable Diffusion 1.5, 2.1, and XL 1.0 base are available for community use.

Sources