yisol/IDM-VTON

IDM-VTON — What It Does

IDM-VTON is an AI system for virtual try-on: you give it a photo of a person and a photo of a garment, and it generates a realistic image of that person wearing that garment. The name stands for "Improving Diffusion Models for Virtual Try-On" — it's built on diffusion models, the same type of AI used in image generators like Stable Diffusion.

The project is the official code release for a peer-reviewed paper (published at ECCV 2024). The authors provide:

• Training code — to train the model on your own data
• Inference code — to run the trained model on new images
• A local demo — a web interface you can launch on your own computer
• A hosted demo — a free online version on Hugging Face Spaces

How It Works (High Level)

The system combines several AI components:

• Diffusion model — the core image generator that creates the final try-on image
• IP-Adapter — a tool that helps the model understand the garment's appearance
• DensePose — maps the person's body shape so the garment fits correctly
• Human parsing — identifies body parts (arms, torso, etc.) to know where clothing goes
• Masking — separates the person from the background and existing clothes

These pieces work together so the generated image looks authentic — the garment follows the person's pose, folds naturally, and matches lighting.

What You Need to Use It

Datasets — The code expects one of two standard fashion datasets:

• VITON-HD — pairs of person photos and garment photos
• DressCode — organized by clothing type (dresses, upper body, lower body)

Both require pre-processing (DensePose images, captions, masks) that the README explains how to set up.

Hardware — Training uses GPU acceleration (via the accelerate library). The README doesn't specify minimum specs, but diffusion model training typically needs a powerful GPU.

Commands You'd Run

• Training: accelerate launch train_xl.py with arguments for batch size, output directory, and data location
• Inference (VITON-HD): accelerate launch inference.py with image dimensions, step count, and seed
• Inference (DressCode): accelerate launch inference_dc.py with an extra --category argument (e.g., upper_body)
• Local demo: python gradio_demo/app.py — but first you must download three checkpoint files (DensePose, human parsing models, and OpenPose body model) and place them in the ckpt folder

Important Details

• License: CC BY-NC-SA 4.0 — non-commercial use only, and any derivatives must be shared under the same license
• Credits: The code builds on several open-source projects (IP-Adapter, OOTDiffusion, DCI-VTON, SCHP, DensePose), all acknowledged in the README
• Paper: Available on arXiv (2403.05139) for those who want the technical details
• Online demo: The Hugging Face Space lets you try it without installing anything

What This Project Is Not

• It's not a finished consumer app — it's a research codebase requiring setup and technical knowledge
• It's not for commercial use (due to the license)
• It doesn't include the datasets — you must download them separately

In short: if you're a researcher or developer interested in virtual try-on technology, this is a well-documented, peer-reviewed implementation you can train, test, and extend — as long as you stay within non-commercial use.