Shilin-LU/MACE

What it solves

MACE addresses the challenge of "mass concept erasure" in text-to-image diffusion models. It prevents models from generating images of unwanted concepts (such as specific objects, celebrities, explicit content, or artistic styles) when prompted. Unlike previous methods that could only handle a few concepts at once, MACE can scale up to 100 concepts while balancing the ability to erase synonyms (generality) without damaging unrelated concepts (specificity).

How it works

MACE uses a finetuning framework that combines two primary techniques:

1. Cross-Attention Refinement: It uses a closed-form solution to refine the projection matrices in the cross-attention blocks of the U-Net, which stops the model from embedding residual information of target phrases into other words.
2. LoRA Finetuning: For every concept targeted for removal, MACE learns a distinct Low-Rank Adaptation (LoRA) module to eliminate that concept's intrinsic information.

These multiple LoRA modules are then integrated using another closed-form solution to ensure they don't interfere with each other or cause the model to forget other useful knowledge (catastrophic forgetting).

Who it’s for

Researchers and developers working with diffusion models who need to implement safety filters or remove copyrighted/harmful content from a generative AI model at scale.

Highlights

• Scalability: Capable of erasing up to 100 concepts simultaneously.
• Balance: Maintains a high degree of specificity (preserving unrelated concepts) and generality (erasing synonyms).
• Versatility: Proven effective across object, celebrity, explicit content, and artistic style erasure.
• Modular Integration: Uses a system of multiple LoRAs that can be fused without mutual interference.