ControlNet in Diffusers

Hugging Face has integrated ControlNet into the diffusers library via the StableDiffusionControlNetPipeline, allowing users to condition image generation with specific spatial contexts such as depth maps, segmentation maps, scribbles, and keypoints. This integration provides a minimal interface to customize the generation process, enabling tasks like converting sketches to artistic drawings or maintaining exact poses across different characters.

ControlNet Architecture and Training

ControlNet is a framework introduced by Lvmin Zhang and Maneesh Agrawala that adds conditional control to text-to-image diffusion models. The architecture operates by creating a "trainable copy" of the pre-trained parameters of a diffusion model (such as Stable Diffusion's latent UNet) while keeping a "locked copy" of the original parameters.

Key technical details of the training process include:

  • Parameter Preservation: The locked copy preserves the general knowledge learned from large datasets, while the trainable copy learns task-specific spatial conditions.
  • Zero Convolutions: The trainable and locked copies are connected via "zero convolution" layers. These layers are optimized during training to ensure that the semantics already learned by the frozen model are preserved as new conditions are integrated.
  • Conditioning Specificity: Every new type of spatial conditioning requires a separate set of trained ControlNet weights. For example, separate weights are used for Canny edge maps and semantic segmentation maps.

Inference and Memory Management

Running ControlNet requires both the pre-trained diffusion model weights and the specific ControlNet weights. For instance, using Stable Diffusion v1-5 with a ControlNet checkpoint adds approximately 700 million additional parameters, increasing memory requirements compared to standard Stable Diffusion.

To optimize performance, Hugging Face recommends several techniques within the diffusers implementation:

  • Smart CPU Offloading: Using enable_model_cpu_offload() ensures that model components (CLIP text encoder, UNet, ControlNet, VAE decoder) are only loaded into GPU memory when needed, significantly reducing VRAM consumption.
  • Fast Schedulers: Replacing the default PNDMScheduler with the UniPCMultistepScheduler can reduce the number of inference steps from 50 to 20 without significant loss in quality.
  • Attention Acceleration: Enabling xformers memory-efficient attention further optimizes speed and memory.

With these optimizations, image generation can take approximately 3 seconds on a V100 GPU using roughly 4 GB of VRAM.

Implementation and Use Cases

The StableDiffusionControlNetPipeline allows for flexible image generation by combining text prompts with spatial conditionings.

Single Conditioning Examples

  • Canny Edge Detection: Users can use OpenCV to generate a Canny edge map of an image, which ControlNet then uses to maintain the exact composition of the original image while changing the subject (e.g., rendering different celebrities in the pose of a classic painting).
  • OpenPose: Using the OpenposeDetector from controlnet_aux, users can extract human poses from an image and apply those exact poses to new generated characters, such as superheroes doing yoga.
  • DreamBooth Integration: ControlNet can be combined with fine-tuned models. A model fine-tuned via DreamBooth (e.g., a Mr. Potato Head model) can be used within the StableDiffusionControlNetPipeline to place a specific subject into a controlled spatial composition.

Combining Multiple Conditionings

Multiple ControlNet conditionings can be used simultaneously for a single image. By passing a list of ControlNetModel instances and a corresponding list of conditionings to the pipeline, users can combine different spatial controls.

To optimize multi-conditioning results, the following strategies are recommended:

  • Masking: Masking areas of one conditioning map so they do not overlap with another (e.g., masking the center of a Canny map to make room for an OpenPose map).
  • Conditioning Scale: Adjusting the controlnet_conditioning_scale to prioritize one spatial condition over another.

Supported Conditioning Models

The following conditioning models are supported in Diffusers:

  • Depth (sd-controlnet-depth)
  • HED (sd-controlnet-hed)
  • Normal (sd-controlnet-normal)
  • Scribble (sd-controlnet-scribble)
  • Segmentation (sd-controlnet-seg)
  • OpenPose (sd-controlnet-openpose)
  • MLSD (sd-controlnet-mlsd)
  • Canny (sd-controlnet-canny)

Sources