apple/ml-mdm

What it solves

Learning high-dimensional diffusion models for high-resolution image and video synthesis is often computationally difficult and optimization-heavy. ml_mdm implements Matryoshka Diffusion Models (MDM), which allow for the efficient training of high-quality text-to-image models at resolutions up to 1024x1024 pixels, even with relatively small datasets like CC12M.

How it works

The framework provides an end-to-end pipeline for training and sampling from pixel-space diffusion models. It utilizes U-Nets and Nested U-Nets to handle multi-scale synthesis. The project includes tools for data preparation (such as scraping CC12M), training via parallel execution (train_parallel.py), and sampling via a dedicated CLI or a web-based UI for image generation.

Who it’s for

This project is designed for researchers and developers working on generative AI, specifically those interested in high-resolution text-to-image synthesis and the Matryoshka Diffusion technique.

Highlights

• High Resolution Support: Capable of training models at resolutions up to 1024x1024.
• Pretrained Models: Provides checkpoints for 64, 256, and 1024 resolution models trained on Flickr data.
• End-to-End Tooling: Includes scripts for dataset scraping, parallel training, and batch sampling.
• Flexible Configuration: Uses simple_parsing to manage model and pipeline configurations via YAML files.