krita-ai-diffusion: a generative AI plugin for Krita that integrates diffusion models for precise image editing and painting

What it solves

This project provides a seamless integration of generative AI into the Krita painting and editing workflow. It addresses the unpredictability of text-to-image generation by offering precise control over image regions, existing content, and structural guidance, allowing artists to use AI as a tool for painting and editing rather than just a prompt-based generator.

How it works

The plugin acts as a frontend for diffusion models (such as Stable Diffusion and Flux) using ComfyUI as the backend. It allows users to perform inpainting, live painting, and upscaling. It leverages ControlNet for structural guidance (like depth maps and line art) and IP-Adapter for style and composition transfer, integrating these complex AI parameters into Krita's layer-based workflow.

Who it’s for

Digital artists and illustrators using Krita who want to enhance their creative process with generative AI without leaving their painting software.

Highlights

• Live Painting: Provides real-time AI interpretation of the canvas for immediate feedback.
• Inpainting and Generative Fill: Use selections to add, remove, or expand objects.
• Structural Control: Support for ControlNet (Scribble, Pose, Depth, etc.) and IP-Adapter for reference images.
• Regional Prompting: Assign specific text descriptions to different image areas defined by layers.
• Local Execution: Supports running models locally on NVIDIA, AMD, Intel, and Apple Silicon hardware.
• Flexible Model Support: Compatible with Flux, Stable Diffusion 1.5, XL, and others.