nihui/zimage-ncnn-vulkan

What it solves

This project provides a high-performance, portable implementation of the Z-Image image generation model. It eliminates the need for complex environments like CUDA or PyTorch, allowing users to run advanced image generation, inpainting, and outpainting on a wide variety of hardware across Windows, Linux, and macOS.

How it works

It uses the ncnn universal neural network inference framework and Vulkan API to execute the Z-Image foundation model (a single-stream diffusion transformer). By leveraging Vulkan, the software can run on Intel, AMD, NVIDIA, and Apple Silicon GPUs. It supports several advanced workflows:

• Text-to-Image: Generating images from text prompts.
• LanPaint Inpaint/Outpaint: Modifying specific areas of an image using a mask or expanding the canvas boundaries.
• Hugging Face Models: It loads pre-trained weights from the Z-Image-Turbo and ControlNet model folders.
• ControlNet: Using a control image (such as a pose or Canny edge map) to guide the structure of the generated image.
• Tile ControlNet: Upscaling low-resolution images to higher resolutions.

Who it’s for

Users who want to generate AI images locally without installing heavy machine learning frameworks, and those who have diverse GPU hardware (non-NVIDIA) that still want to utilize hardware acceleration.

Highlights

• Cross-Platform Portable: Works on Windows, Linux, and macOS without requiring CUDA or PyTorch.
• Broad Hardware Support: Compatible with any Vulkan-capable GPU (Intel, AMD, NVIDIA, Apple Silicon).
• Advanced Image Editing: Integrated support for LanPaint inpainting and outpainting.
• Structural Guidance: Includes ControlNet support for pose, Canny, and grayscale guidance.
• Upscaling: Features a Tile ControlNet mode for increasing image resolution.