huawei-bayerlab/marigold-v2

Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

Marigold V2 – Diffusion‑Transformer Depth & Dense‑Vision Models

What it is – Marigold V2 is a research‑grade codebase that turns a pretrained diffusion transformer (the Qwen‑Image‑Edit‑2509 model) into fast, single‑step predictors for a variety of dense image modalities: monocular depth, see‑through depth, surface normals, and albedo. The authors provide a family of checkpoints, a lightweight fine‑tuning recipe (LoRA adapters + a tiny VAE decoder), and scripts for inference, evaluation, and training.

Key ideas

  • Reuse a frozen diffusion backbone – the large Qwen text‑to‑image model is kept frozen; only small trainable adapters (LoRA) and a VAE decoder are learned, making fine‑tuning cheap (≈ 1 week on a single 32 GB GPU).
  • Single‑step dense prediction – unlike classic diffusion pipelines that require many sampling steps, Marigold V2 produces the final dense map in one forward pass.
  • Multiple output modalities – the same backbone can be repurposed for depth (log, linear, disparity, see‑through), normals, and albedo by swapping the LoRA checkpoint and a prompt embedding.
  • Affine‑invariant depth – depth values are output up to an unknown scale/shift per image, which matches the evaluation protocol used in the paper.

Provided components

Component What you get
Checkpoints LoRA adapters (stored as trainables.safetensors) for each modality and depth parameterisation. All share the frozen Qwen backbone and pre‑computed text‑prompt embeddings, so the 7 B text encoder never loads.
Inference script (scripts/infer.py) Simple CLI to run depth, normals, or albedo on any folder of images. Outputs NumPy .npy maps and visual PNGs.
Evaluation launchers Bash scripts that download standard depth/normal benchmarks (NYUv2, KITTI, ETH3D, ScanNet, DIODE, iBims‑1, etc.) and compute the metrics reported in the paper.
Training framework YAML‑driven config system (marigoldv2/experiments/...) plus a small registry that wires datasets, transforms, network blocks, loss graphs, and optimisers. Supports single‑GPU and multi‑GPU (accelerate launch).
Dataset helpers Scripts to download and preprocess the large Hypersim, Virtual KITTI‑2, LayeredDepth‑Syn, and DINOv3 perceptual‑loss checkpoints.
Demo & web UI Hugging Face Spaces links (Marigold‑V2‑web) for quick visual testing.

Quick start (Linux, Python 3.10, CUDA GPU)

# Clone & set up environment (conda, CUDA 12.8 wheels)
git clone https://github.com/huawei-bayerlab/marigold-v2.git
cd marigold-v2
bash setup/setup_env.sh   # creates conda env "marigold-v2"
conda activate marigold-v2

# Download model weights and example images
python scripts/download_assets.py --skip-datasets

# Run depth on the bundled examples
python scripts/infer.py \
  --modality depth \
  --image_dir assets/examples \
  --output_dir output/examples

The default depth checkpoint (depth/Log-stage2) yields the state‑of‑the‑art numbers reported in the README (e.g., AbsRel 3.6 % on NYUv2, δ1 98 %).

Training a new model (rough outline)

  1. Prepare data – the repo includes scripts to fetch Hypersim, Virtual KITTI‑2, and optional layered‑depth data.
  2. Choose a config from marigoldv2/experiments/... (e.g., training_relative_log_depth_config_stage2.yaml).
  3. Launch training:
    python marigoldv2/script/train/train.py \
  --config marigoldv2/experiments/20260316_qwen_depth/training_relative_log_depth_config_stage2.yaml \
  --output_dir output/train_runs --no_wandb
    – Stage 1 (160 k steps) takes ~5 days on a 32 GB GPU; Stage 2 (30 k steps) ~1 day.
  4. The resulting trainables.safetensors can be used directly with scripts/infer.py.

Extending to a new dense task – The framework is modular:

  • Add a dataset YAML and a manifest loader in marigoldv2/config/datasets/.
  • Write a small network_graph step that converts the decoded RGB output into the desired representation (e.g., a semantic map).
  • Add a loss module (e.g., cross‑entropy) and a validation metric.
  • Register the new modules in a fresh experiment folder and point a new training config at them. The README includes a step‑by‑step guide using the surface‑normal experiment as a template.

Performance & hardware notes

  • Inference at 1024×1024 needs ~17 GB GPU memory; 2048×2048 needs ~29 GB.
  • The diffusion transformer is quantized to 4 bit on load, which reduces memory and speeds up inference.
  • For depth‑completion the repo provides tiled‑local‑adaptation scripts that improve MAE when run on higher‑resolution inputs.

Where to find more

TL;DR – Marigold V2 is a research‑focused, open‑source toolkit that repurposes a frozen diffusion transformer into fast, high‑quality monocular depth, see‑through depth, normal, and albedo predictors, with scripts for end‑to‑end training, evaluation, and inference. It is squarely in the AI/ML domain and not an off‑topic collection.

Related