apple-aiml-research/ml-mdm

What it solves

It addresses the computational and optimization challenges of training high-resolution diffusion models. Specifically, it allows for the efficient training of high-quality text-to-image models at resolutions up to 1024x1024 pixels, even with relatively small datasets like CC12M.

How it works

The project implements "Matryoshka Diffusion Models," a technique that enables a single pixel-space model to handle multiple resolutions. It provides an end-to-end framework including implementations of U-Nets and Nested U-Nets, and supports pipelines like DDPM. The system is designed for scalability, supporting parallel training via torchrun and integration with S3 for large-scale dataset storage and retrieval.

Who it’s for

Researchers and developers working on image and video synthesis who want to train high-resolution diffusion models efficiently or use pretrained checkpoints for text-to-image generation.

Highlights

• Multi-resolution support: Capable of generating images at 64, 256, and 1024 pixel resolutions.
• Scalable Training: Includes tools for parallel training and data scraping for datasets like CC12M.
• S3 Integration: Supports direct loading and filtering of datasets from S3 buckets using regular expressions.
• Web Demo: Provides a ready-to-use web UI for generating samples from pretrained models.