apple-aiml-research/ml-stable-diffusion

Stable Diffusion with Core ML on Apple Silicon

What it solves

This project enables the execution of Stable Diffusion image generation models on Apple Silicon devices (Mac, iPad, and iPhone). It addresses the challenge of running large, computationally expensive diffusion models on mobile and desktop hardware by providing tools for model conversion and optimization specifically for Apple's Core ML framework.

How it works

The project provides two primary components:

  1. Python Package (python_coreml_stable_diffusion): Used to convert PyTorch models (via Hugging Face diffusers) into the Core ML format. It includes tools for weight compression and optimization, such as post-training palettization and Mixed-Bit Palettization (MBP) to reduce model size and memory usage.
  2. Swift Package (StableDiffusion): A developer-facing library that can be integrated into Xcode projects. It allows apps to load the converted Core ML models and perform image generation directly on the device.

Who it’s for

  • App Developers: Those wanting to integrate high-performance text-to-image generation into iOS, iPadOS, or macOS applications.
  • ML Engineers: Researchers and developers looking to optimize Stable Diffusion models for edge deployment on Apple hardware.

Highlights

  • Hardware Acceleration: Optimized for the Apple Neural Engine (ANE), GPU, and CPU.
  • Advanced Compression: Supports data-free post-training palettization (up to 8-bit) and Mixed-Bit Palettization (MBP) for extreme size reduction without significant quality loss.
  • Memory Management: Includes a reduceMemory option to load and unload models just-in-time, enabling execution on devices with limited RAM (e.g., iPhone 12 Mini).
  • Broad Compatibility: Supports multiple Stable Diffusion versions, including SD 2.1 and SDXL.

Related

  • Project
    Comfy-Org/ComfyUI-Manager

    ComfyUI‑Manager is an extension for the visual AI‑pipeline UI ComfyUI. It adds a “Manager” panel that lets users browse, install, update, enable/disable, and remove custom node packages and model files from a curated registry or local cache. Features include one‑click installs, snapshot‑based environment restore, workflow sharing to external sites, component copy‑&‑paste, a CLI tool for head‑less use, and extensive configuration (security level, network mode, pip/uv handling). It streamlines handling the large ecosystem of community‑made nodes that power Stable‑Diffusion image generation in ComfyUI.

  • Project
    wkentaro/labelme

    labelme is a cross‑platform Python/Qt desktop app for drawing bounding boxes, polygons, masks, etc., on images or video. Annotations are saved as JSON and can be exported to VOC or COCO formats, making it a practical tool for creating training data for computer‑vision models. Install via pip, a paid standalone executable, or Linux packages; recent versions add AI‑assisted mask generation (SAM, YOLO‑World).

  • Project
    danielgatis/rembg

    A tool for removing image backgrounds that can be used as a CLI, Python library, or HTTP server, supporting various AI models and hardware acceleration.

  • Project
    invoke-ai/InvokeAI

    Invoke AI is an open‑source, locally‑run visual‑media generation platform. It provides a web‑based UI with a unified canvas, node‑based workflow editor, gallery management, and support for dozens of modern text‑to‑image models (Stable Diffusion, Flux, Qwen‑Image, etc.). Users install via the provided launcher, run a local server, and create or refine images through an intuitive browser interface. The project targets artists, designers, and developers who need a flexible, extensible AI image creation tool.