Hugging Face Integrates Nunchaku 4-bit Diffusion Inference into Diffusers

Hugging Face Integrates Nunchaku 4-bit Diffusion Inference into Diffusers

Hugging Face has integrated Nunchaku Lite into the Diffusers library, allowing users to run large diffusion transformers with 4-bit weights and activations (W4A4). This integration reduces VRAM requirements by up to 50% and improves denoising latency by approximately 30% compared to BF16 baselines, making high-performance image generation accessible on consumer GPUs.

SVDQuant and the Nunchaku Engine

Nunchaku Lite is based on SVDQuant, a quantization method designed to handle the large outliers found in both weights and activations of diffusion transformers. Unlike standard 4-bit quantization, SVDQuant isolates activation outliers by moving them into the weights and representing the most difficult parts of each weight matrix with a small 16-bit low-rank branch. The remaining residual is then quantized to 4 bits.

The reference Nunchaku CUDA inference engine optimizes this process using fused kernels that combine the low-rank down projection with the quantization kernel and the low-rank up projection with the 4-bit compute kernel, effectively eliminating memory access overhead for the 16-bit branch.

Nunchaku Lite Integration in Diffusers

Nunchaku Lite provides a streamlined integration path within Diffusers, enabling the loading of Nunchaku-style checkpoints via from_pretrained() without requiring a separate inference engine or local CUDA compilation. It achieves this by patching nn.Linear modules with runtime SVDQ/AWQ linear layers.

Nunchaku Lite employs two primary kernel families:

  • svdq_w4a4: Used for transformer attention and MLP projections where most compute occurs. It is available in INT4 and NVFP4 variants.
  • awq_w4a16: Used for memory-bound, precision-sensitive layers like adaptive normalization and modulation projections (e.g., FLUX adanorm_single / adanorm_zero), utilizing 4-bit weights and 16-bit activations.

While Nunchaku Lite does not include the architecture-specific fused execution paths of the original Nunchaku engine, it still delivers a roughly 30% speedup and the same VRAM reduction.

Hardware Support and Performance

Support for Nunchaku Lite kernels varies by GPU generation and precision:

Scheme Precision Supported GPUs
svdq_w4a4 nvfp4 Blackwell (RTX 50 series, RTX PRO 6000, B200)
svdq_w4a4 int4 Turing / Ampere / Ada (RTX 30 & 40 series, A100, L40S)
awq_w4a16 int4 Turing / Ampere / Ada (RTX 30 & 40 series, A100, L40S)

Volta and Hopper GPUs are currently unsupported.

Benchmarks

Measured on an NVIDIA RTX PRO 6000 (Blackwell) at 1024x1024 resolution using an ERNIE-Image-Turbo checkpoint:

| Configuration | Full pipeline | Denoise loop | Peak VRAM | Speedup | | :--- | :--- | :--- | :--- | :--- | :--- | | BF16 baseline | 3.00 s | 2.86 s | 31.1 GB | 1.0x | | Nunchaku Lite NVFP4 | 2.27 s | 2.13 s | 20.6 GB | 1.35x | | Nunchaku Lite NVFP4 + torch.compile | 1.68 s | 1.53 s | 20.6 GB | 1.8x | | Nunchaku Lite NVFP4 + NF4 text encoder | 2.29 s | 2.13 s | 16.0 GB | 1.35x |

Combining Nunchaku Lite with torch.compile increases the end-to-end speedup to 1.8x, while adding a bitsandbytes NF4 quantized text encoder further reduces peak VRAM to 16.0 GB.

Quantizing Custom Models with diffuse-compressor

Nunchaku Lite is architecture-agnostic. The diffuse-compressor toolkit allows users to calibrate, quantize, package, and publish their own Diffusers models. The workflow involves:

  1. Inspection: A generic scanner identifies SVDQ W4A4 targets (repeated transformer blocks) and AWQ W4A16 targets (modulation linears).
  2. Quantization: Running SVDQuant to create a quantized checkpoint (supporting int4 or nvfp4).
  3. Packaging: Combining the quantized transformer with the base pipeline and creating a nunchaku_lite configuration in transformer/config.json.
  4. Verification: Loading the model via DiffusionPipeline.from_pretrained() and pushing the final result to the Hugging Face Hub.

Structural Rewrites

For maximum performance, some models require structural rewrites—such as combining separate Q, K, and V projections into a single to_qkv module. While the generic diffuse-compressor path cannot infer these changes, they can be implemented via model-specific target configs and runtime adapters provided by rootonchair/nunchaku-lite to enable fused operations.

Ready-to-Use Checkpoints

Several pre-quantized checkpoints are available on the Hub, including:

  • ERNIE-Image-Turbo: Available in INT4 and NVFP4 variants with NF4 text encoders.
  • Krea 2 Turbo: Available in NVFP4.
  • lite-infer: A collection of additional Nunchaku Lite checkpoints.

Sources