Efficient Controllable Generation for SDXL with T2I-Adapters

T2I-Adapter-SDXL is a lightweight, plug-and-play guidance model that allows users to control Stable Diffusion XL (SDXL) generation using external signals while keeping the base model frozen. By aligning internal model knowledge with external control signals, T2I-Adapters provide a computationally efficient alternative to ControlNet for achieving rich editing and control effects.

Efficiency Gains Over ControlNet

T2I-Adapters offer a significant reduction in parameter count and computational cost compared to ControlNet. While ControlNet requires running both the ControlNet and UNet during every denoising step and involves copying the UNet encoder, T2I-Adapters are run only once for the entire denoising process.

Comparison of model parameters and storage (fp16) for SDXL-based control models:

Model Type Model Parameters Storage (fp16)
ControlNet-SDXL 1251 M 2.5 GB
ControlLoRA (rank 128) 197.78 M 396 MB
T2I-Adapter-SDXL 79 M 158 MB

As shown, T2I-Adapter-SDXL achieves a 93.69% reduction in parameters and a 94% reduction in storage compared to ControlNet-SDXL.

Technical Implementation and Training

T2I-Adapter-SDXL drives the 2.6B parameter SDXL model using a small 79M parameter adapter. The models were trained using the diffusers library on 3 million high-resolution image-text pairs from the LAION-Aesthetics V2 dataset.

Training Configuration

  • Training Steps: 20,000 to 35,000
  • Batch Size: Total batch size of 128 (16 per GPU via data parallel)
  • Learning Rate: Constant 1e-5
  • Mixed Precision: fp16

Controlling Generation in Diffusers

Integration with the diffusers library is handled via the StableDiffusionXLAdapterPipeline. The generation process requires preparing a condition image (e.g., a lineart map) and passing it along with a prompt to the pipeline.

Key Control Parameters

Two primary arguments allow users to fine-tune the influence of the adapter:

  1. adapter_conditioning_scale: Controls the strength of the conditioning effect. Higher values increase the influence of the external signal on the output.
  2. adapter_conditioning_factor: Determines the duration of the conditioning application during the denoising process. A value of 1.0 applies the adapter to all timesteps, while 0.5 applies it only to the first 50% of steps.

Supported Conditioning Modalities

T2I-Adapter-SDXL provides pre-trained checkpoints for several different control signals, enabling diverse image generation workflows:

  • Lineart Guided: Uses the TencentARC/t2i-adapter-lineart-sdxl-1.0 model.
  • Sketch Guided: Uses the TencentARC/t2i-adapter-sketch-sdxl-1.0 model.
  • Canny Guided: Uses the TencentARC/t2i-adapter-canny-sdxl-1.0 model.
  • Depth Guided: Available via TencentARC/t2i-adapter-depth-midas-sdxl-1.0 and TencentARC/t2i-adapter-depth-zoe-sdxl-1.0.
  • OpenPose Guided: Uses the TencentARC/t2i-adapter-openpose-sdxl-1.0 model.

Sources