huawei-bayerlab/marigold-v2

Marigold V2 – Diffusion‑Transformer Depth & Dense‑Vision Models

What it is – Marigold V2 is a research‑grade codebase that turns a pretrained diffusion transformer (the Qwen‑Image‑Edit‑2509 model) into fast, single‑step predictors for several dense image‑to‑image tasks: monocular depth (both log‑depth and linear depth), see‑through depth, surface normals, and albedo. The authors provide a lightweight fine‑tuning recipe that runs on a single consumer GPU (≈5 days for the full depth model, <1 day for the other variants) and still reaches state‑of‑the‑art accuracy on standard benchmarks.

Key ideas

• Diffusion transformer repurposing – The large diffusion model is frozen; only small LoRA adapters and a VAE decoder are trained, making fine‑tuning cheap.
• Single‑step inference – After training, the model predicts the target dense map in one forward pass, without iterative diffusion sampling.
• Unified architecture – The same backbone can be swapped to output depth, normals, or albedo simply by loading a different checkpoint and prompt embedding.
• Affine‑invariant depth – Depth predictions are up to an unknown scale/shift per image, which matches the evaluation protocol used in the paper.

What you get

• Ready‑to‑use inference script (scripts/infer.py) for any image folder.
• Evaluation launchers for the major depth and normal benchmarks (NYUv2, KITTI, ETH3D, ScanNet, DIODE, iBims‑1, Sintel, etc.).
• Training scripts and a modular YAML‑based configuration system that lets you reproduce the published models or add new tasks.
• Pre‑trained checkpoints (LoRA adapters + VAE decoder) hosted on Hugging Face, plus pre‑computed Qwen text‑prompt embeddings so the 7 B text encoder never needs to be loaded.

Quick start (Linux, Python 3.10, CUDA GPU)

# clone and set up the conda env (CUDA 12.8 wheels by default)
git clone https://github.com/huawei-bayerlab/marigold-v2.git
cd marigold-v2
bash setup/setup_env.sh   # creates env "marigold-v2"
conda activate marigold-v2

# download model weights and demo assets (skip large datasets)
python scripts/download_assets.py --skip-datasets

# run depth inference on the example images
python scripts/infer.py \
  --modality depth \
  --image_dir assets/examples \
  --output_dir output/examples

The results appear as *.npy depth maps and visual PNGs under output/examples.

How to train / fine‑tune

1. Download the training data (Hypersim, Virtual KITTI 2) and the DINOv3 features required for the iREPA loss:

python scripts/download_assets.py --include-dinov3

2. Run the two‑stage depth training (stage 1 ≈ 5 days, stage 2 ≈ 1 day on a 32 GB GPU):

# Stage 1 – iREPA + pixel losses
python marigoldv2/script/train/train.py \
  --config marigoldv2/experiments/20260316_qwen_depth/training_relative_log_depth_config.yaml \
  --output_dir output/train_runs --no_wandb

# Stage 2 – SinkLoss + VAE fine‑tuning (initialize from stage 1 checkpoint)
python marigoldv2/script/train/train.py \
  --config marigoldv2/experiments/20260316_qwen_depth/training_relative_log_depth_config_stage2.yaml \
  --output_dir output/train_runs --no_wandb

Replace the config path to train normals, albedo, or any of the other depth variants listed in the Checkpoints table.

Extending to a new dense task The framework is driven by YAML configs that declare:

• dataset definition (marigoldv2/config/datasets/)
• a list of Python modules to register custom components (register_modules)
• a network_graph describing the forward pass (e.g., encode → DiT → decode → post‑process)
• a loss_graph that pulls predictions and targets together
• an optimization block for schedules, quantisation, and LoRA settings. To add a new task, copy an existing experiment folder, implement a small dataset loader, an output adapter (e.g., convert decoded RGB to XYZ normals), and a loss function, then point a new YAML config at them. Inference automatically supports the new modality after adding a MODALITIES entry in scripts/infer.py.

Benchmarks & performance (reported in the paper)

Metric	NYUv2	KITTI	ETH3D	ScanNet	DIODE
AbsRel ↓ / δ1 ↑ (depth)	3.6 / 98.0	5.4 / 97.4	2.8 / 99.2	3.7 / 97.9	5.2 / 97.1
Mean angular error ↓ / 11.25° ↑ (normals)	16.6 / 61.2	14.1 / 67.4	15.9 / 70.9	28.7 / 27.6
Albedo (Hypersim test) – PSNR 20.78, SSIM 0.811, LPIPS 0.195

Resources

• Demo: Hugging Face Space – https://huggingface.co/spaces/toshas/Marigold-V2
• Model weights: https://huggingface.co/huawei-bayerlab/marigold-v2-0
• Paper: https://arxiv.org/abs/2609.08084 (to appear in ACM TOG, SIGGRAPH Asia 2026)
• Website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web

When to use it

• You need high‑quality monocular depth or other dense predictions without the overhead of multi‑step diffusion sampling.
• You have a single GPU and want to fine‑tune a model on a custom dataset in a few days.
• You want a single codebase that can be extended to new dense‑vision outputs (normals, albedo, see‑through depth, etc.).

---

Citation

@article{pavlovic2026marigoldv2,
  author = {Pavlovic, Igor and Wandel, Thiemo and Obukhov, Anton and Bartolomei, Luca and Davydov, Andrey and Tosi, Fabio and Poggi, Matteo and S{"u}sstrunk, Sabine and Dai, Dengxin},
  title = {Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation},
  year = {2026},
  journal = {ACM Trans. Graph.},
  volume = {45},
  number = {6},
  pages = {204},
  doi = {10.1145/3842528},
}