Dreambooth Stable Diffusion fine‑tuning guide with Diffusers

TL;DR

Hugging Face’s Dreambooth guide shows that Stable Diffusion fine‑tuning works best with a low learning rate, enough training steps (800‑1200 for faces, 400‑600 for objects), prior‑preservation for human subjects, and optional text‑encoder fine‑tuning; these settings prevent over‑fitting and improve image fidelity.

Recommended Hyper‑parameters

  • Learning rate: Use a low LR (e.g., 1e‑6 to 2e‑6). Higher LRs cause rapid over‑fitting.
  • Training steps: Increase steps until quality stabilizes. Objects converge around 400‑600 steps; faces need 800‑1200 steps.
  • Batch size: 2 per GPU (batch‑size 4 on two 40 GB A100s) works well for objects; use batch‑size 2 for faces.
  • Prior preservation: Essential for faces; generate or supply class‑level images to keep the model from collapsing to the few training photos.
  • Scheduler choice: During inference, the DDIM scheduler outperforms PNDM and LMSDiscrete, especially when the model shows signs of over‑fitting. Running ~100 inference steps further cleans noisy patches.
  • Text encoder: Fine‑tuning the CLIP text encoder alongside the UNet dramatically improves realism and prompt interpretability, but requires ≥24 GB GPU memory. 8‑bit Adam, fp16, or gradient accumulation can reduce memory to 16 GB.
  • EMA: Exponential moving average (EMA) did not noticeably affect results.
  • Token selection: The special token sks is unnecessary; any natural token describing the target works.

Learning‑Rate Impact

Across four datasets (cat toy, pighead, Mr. Potato Head, and a human face), low learning rates consistently yielded higher‑quality generations. High learning rates (5e‑6) produced noisy artifacts and rapid over‑fitting, while low rates (2e‑6) preserved detail and allowed the model to generalize beyond the training images.

Experiment Details

All experiments used the train_dreambooth.py script from the Diffusers repository, the AdamW optimizer, and two 40 GB A100 GPUs. The seed and all hyper‑parameters were held constant except for learning rate, step count, and prior preservation.

Object Fine‑tuning (Cat Toy, Pighead, Mr. Potato Head)

  • Batch size: 4 (2 per GPU)
  • Steps: 400
  • Learning rates: 5e‑6 (high) vs. 2e‑6 (low)
  • Prior preservation: Not used

Human Face Fine‑tuning (Kramer character)

  • Batch size: 2 (1 per GPU)
  • Steps: 800 – 1200
  • Learning rates: 5e‑6 (high) vs. 2e‑6 (low)
  • Prior preservation: Tested both with and without

Memory‑saving tricks such as 8‑bit Adam, fp16 training, or gradient accumulation enable these runs on 16 GB GPUs (e.g., Google Colab, Kaggle).

Prior Preservation for Faces

Prior preservation mixes images of the target subject with class‑level images (e.g., other people) during training. The script can auto‑generate the class images using Stable Diffusion.

  • With prior preservation (1200 steps, LR = 2e‑6): Images retain the target’s identity while preserving background diversity.
  • Without prior preservation (same steps & LR): Results contain more noisy blotches and exhibit stronger over‑fitting.

Scheduler Effects During Inference

Scheduler Observation
PNDM Generates acceptable images when training is well‑balanced but struggles with over‑fitted models.
LMSDiscrete Produces severe artifacts and low quality in over‑fitted cases.
DDIM Consistently yields cleaner outputs; additional inference steps (~100) resolve residual noise.

The same pattern appears across subjects (faces, objects), though the gap is most pronounced for faces.

Fine‑tuning the Text Encoder

The original Dreambooth paper freezes the CLIP text encoder, but Hugging Face’s experiments show that unfreezing it dramatically improves results, especially for facial subjects.

  • Frozen encoder: Generates plausible images but often retains over‑fitting artifacts.
  • Fine‑tuned encoder: Produces more realistic faces, better prompt adherence, and reduced over‑fitting.

Fine‑tuning the encoder adds memory overhead; a GPU with at least 24 GB VRAM is recommended, though 16 GB can suffice with mixed‑precision and optimizer tricks.

Combining Textual Inversion with Dreambooth

A hybrid experiment ran Textual Inversion for 2000 steps followed by Dreambooth for 500 steps (LR = 1e‑6). The resulting images were better than plain Dreambooth but did not match the quality of full text‑encoder fine‑tuning. The approach appears to over‑fit style details while remaining viable on 16 GB GPUs.

Practical Takeaways

  1. Start with a low learning rate (1e‑6 – 2e‑6) and increase steps gradually. This balances under‑ and over‑fitting.
  2. Use prior preservation for any human subject. It mitigates over‑fitting and improves realism.
  3. Prefer the DDIM scheduler and ~100 inference steps for final sampling. This cleans up noise without extra training.
  4. Fine‑tune the text encoder when possible. The quality boost justifies the extra memory cost.
  5. Leverage memory‑saving techniques (8‑bit Adam, fp16, gradient accumulation) to run on consumer‑grade GPUs.

Ethical Reminder

Dreambooth should never be used for malicious purposes, to generate harmful content, or to impersonate individuals without consent. All fine‑tuned models remain subject to the CreativeML Open RAIL‑M license governing Stable Diffusion distribution.

Sources