adobe-research/custom-diffusion

What it solves

Custom Diffusion enables the fine-tuning of text-to-image diffusion models (like Stable Diffusion) to learn new concepts—such as specific objects, people, or artistic styles—using only a small number of images (typically 4 to 20). It specifically addresses the need for fast training and low storage overhead when adding new concepts to a pre-trained model.

How it works

Instead of updating the entire model, Custom Diffusion fine-tunes only a small subset of parameters: the key and value projection matrices within the cross-attention layers. To prevent the model from forgetting existing knowledge (overfitting), it uses a small set of regularization images. For personal categories, it introduces a unique modifier token (e.g., V* dog). The system also supports joint training for multiple concepts and an optimization-based method to merge weights from two separately fine-tuned models.

Who it’s for

This project is designed for researchers and developers working with generative AI who want to customize image generation models with specific personal or stylistic concepts without requiring massive datasets or extensive computing resources.

Highlights

• Efficient Training: Can be trained in approximately 6 minutes using 2 A100 GPUs.
• Low Storage: Each additional concept requires only about 75MB of extra storage.
• Multi-Concept Support: Capable of combining multiple new objects, styles, or categories in a single generation.
• Model Compression: Includes tools to compress fine-tuned weights using low-rank approximation, potentially reducing model size to ~15MB.