VisualComputingInstitute/diffusion-e2e-ft

What it solves

This project addresses the inefficiency and slow inference speeds of probabilistic diffusion estimators used for depth and surface normal estimation. While standard diffusion models require multiple iterative steps to produce a result, this project provides a method to fine-tune these models into single-step deterministic estimators that are faster and more accurate.

How it works

The researchers developed an end-to-end (E2E) fine-tuning approach for image-conditional diffusion models. By applying task-specific losses, they can transform models—either starting from pretrained diffusion estimators like Marigold and GeoWizard or directly from Stable Diffusion—into deterministic models. These fine-tuned models can generate depth maps or surface normals in a single inference step using zero noise, significantly reducing the time required for a prediction without sacrificing quality.

Who it’s for

This is primarily for computer vision researchers and developers working on 3D scene understanding, depth estimation, and surface normal prediction from 2D images.

Highlights

• Single-Step Inference: Converts slow, multi-step diffusion processes into fast, deterministic single-step predictions.
• High Performance: Outperforms original diffusion estimators in both speed and accuracy across multiple benchmarks (NYUv2, KITTI, ScanNet, etc.).
• Broad Compatibility: Supports fine-tuning from various starting points, including Stable Diffusion, Marigold, and GeoWizard.
• Multi-Modality: Capable of estimating both depth and surface normals.