nunchux-ai/nunchaku

What it solves

Nunchaku is a high-performance inference engine designed to make large diffusion models (like FLUX.1 and SANA) run faster and use significantly less memory without sacrificing visual quality. It addresses the challenge of 4-bit quantization, where "outliers" in neural network weights and activations typically cause a loss in image fidelity.

How it works

The engine implements SVDQuant, a post-training quantization technique. It works in three stages:

1. Outlier Migration: It moves outliers from activations to weights to make activations easier to quantize.
2. Low-Rank Decomposition: It uses Singular Value Decomposition (SVD) to split the updated weights into a low-rank component (which stays at 16-bit precision) and a residual component (which is quantized to 4-bit).
3. Kernel Fusion: To prevent the low-rank branch from slowing down the system, Nunchaku uses specialized fused kernels that combine projection and quantization steps, reducing data movement overhead.

Who it’s for

It is primarily for developers and researchers working with diffusion models who need to deploy them on consumer-grade hardware (such as NVIDIA RTX 4090 or 5090 GPUs) with limited VRAM.

Highlights

• Massive Memory Reduction: Reduces the 12B FLUX.1 model size by 3.6x and memory usage by 3.5x.
• Significant Speedups: Achieves up to 10.1x speedup over 16-bit models on laptop GPUs by eliminating the need for CPU offloading.
• Broad Model Support: Supports FLUX.1 (including Krea and Kontext variants), SANA, and Qwen-Image.
• Hardware Optimization: Optimized for NVIDIA GPUs, including native support for NVFP4 precision on the RTX 5090.
• Ecosystem Integration: Provides a ComfyUI plugin and a Python backend for modular integration.