prs-eth/Marigold

What it solves

Marigold provides a way to extract high-resolution image characteristics from a single image (monocular analysis). It solves the problem of accurately estimating depth, surface normals, and intrinsic image properties (like albedo and lighting) without requiring expensive specialized sensors or massive real-world datasets for every new task.

How it works

Marigold repurposes pretrained latent diffusion models (specifically Stable Diffusion) by adapting them for dense image analysis. Instead of generating new images, it uses a fine-tuning protocol and small synthetic datasets to leverage the rich visual knowledge already stored in the generative model. This allows the model to perform zero-shot generalization, meaning it can accurately analyze images it has never seen before.

Who it’s for

This tool is designed for computer vision researchers and developers working on 3D scene reconstruction, image analysis, and graphics, as well as those needing high-quality depth and normal maps from standard 2D images.

Highlights

• Multi-modal Analysis: Supports monocular depth estimation, surface normal prediction, and intrinsic image decomposition (appearance and lighting).
• Hugging Face Integration: Pipelines are integrated into the diffusers library for easy access.
• Efficient Training: Can be trained on a single GPU over a few days using small synthetic datasets.
• Zero-Shot Generalization: Demonstrates state-of-the-art performance on unseen data without further training.