aMUSEd: Efficient Text-to-Image Generation

Hugging Face has introduced aMUSEd, an efficient non-diffusion text-to-image model. As an open reproduction of Google's MUSE, aMUSEd leverages Masked Image Modeling (MIM) to provide a faster, more interpretable alternative to traditional latent diffusion models.

Masked Image Modeling Architecture

aMUSEd utilizes a Masked Image Modeling (MIM) approach, which allows for fewer inference steps and higher interpretability compared to latent diffusion. The model architecture consists of three primary components: a VQGAN for tokenization, a CLIP-L/14 text encoder for prompt embeddings, and a U-ViT model for predicting masked patches.

Training Process

During training, the model follows these steps:

  1. Tokenization: Input images are converted into image tokens using a VQGAN.
  2. Masking: Image tokens are are masked according to a cosine masking schedule.
  3. Prediction: The masked tokens, conditioned on prompt embeddings from the CLIP-L/14 text encoder, are passed to the U-ViT model to predict the masked patches.

Inference Process

During inference, the model generates images through an iterative process:

  1. Embedding: The input prompt is embedded using the CLIP-L/14 text encoder.
  2. Iterative Prediction: The process starts with randomly masked tokens. The U-ViT model predicts the masked tokens, and only a percentage of the most confident predictions (determined by the mask schedule and the number of steps N) are kept. The remaining tokens are re-masked and passed back to the U-ViT model.
  3. Decoding: The final output is passed through the VQGAN decoder to produce the final image.

Key Differences from MUSE

aMUSEd differs from the original MUSE model in three key ways:

  • Single-Stage Prediction: It does not use the two-stage approach for predicting final masked patches.
  • Text Conditioning: It uses CLIP L/14 instead of T5 for computing text embeddings.
  • Micro-conditioning: Following SDXL, it incorporates additional conditioning such as image size and cropping.

Capabilities and Performance

aMUSEd is designed for efficiency and speed. With approximately 800M parameters (including the text encoder and VQ-GAN), the model is highly performant.

Inference Speed and Latency

As a benchmark conducted on an A100 GPU, aMUSEd demonstrates significantly lower inference latencies compared to other models. This efficiency makes it a suitable candidate for on-device applications.

Zero-Shot Inpainting

Because of its pre-training objective, aMUSEd is capable of zero-shot image inpainting, a feature not present in models like SDXL.

Style Transfer

Following the capabilities of MUSE, aMUSEd demonstrates an ability for style transfer using a single image, which provides potential for personalized and style-specific image generation.

Fine-Tuning and Accessibility

aMUSEd is released under an OpenRAIL license, making it commercially friendly. Hugging Face provides a training script for fine-tuning on custom datasets.

Hardware Requirements for Fine-Tuning

  • Standard Fine-Tuning: Using the 8-bit Adam optimizer and float16 precision, fine-tuning requires just under 11GB of GPU VRAM.
  • LoRA: Using Low-Rank Adaptation (LoRA), memory requirements are further reduced to 7GB of VRAM.

Limitations

While aMUSEd is not state-of-the-art in terms of raw image quality, it was released to encourage the community to explore the potential of MIM for image generation. The authors highlight several benefits of the MIM framework, including inference efficiency, smaller model size for on-device use, and the ability to perform task transfer without expensive fine-tuning.

Sources