RWTHVision/diffusion-e2e-ft

[WACV'25 Oral] Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think

What it solves

This project addresses the inefficiency and slow inference speeds of probabilistic diffusion estimators used for depth and surface normal estimation. While standard diffusion models require multiple denoising steps to produce a result, this approach enables the creation of single-step, deterministic models that maintain or improve accuracy while significantly reducing computation time.

How it works

The project implements end-to-end (E2E) fine-tuning on image-conditional diffusion models. By starting with either a pretrained diffusion estimator (like Marigold or GeoWizard) or a general-purpose model like Stable Diffusion and applying task-specific losses, the models are converted into deterministic estimators. These fine-tuned models can generate depth maps or surface normals in a single inference step using zero noise, bypassing the iterative sampling process typically required by diffusion models.

Who it’s for

This is designed for computer vision researchers and developers working on 3D scene understanding, specifically those needing fast, high-quality depth and surface normal estimation from single RGB images.

Highlights

  • Single-Step Inference: Reduces inference time to as low as 121ms per image on an NVIDIA RTX 4090.
  • Deterministic Output: Eliminates the probabilistic nature of diffusion sampling for more consistent results.
  • Versatile Base Models: Supports fine-tuning from Stable Diffusion, Marigold, and GeoWizard.
  • Multi-Modality: Provides specialized checkpoints for both depth estimation and surface normal estimation.

Related

  • Project
    emdgroup/baybe

    A Bayesian Design of Experiments toolbox for efficiently finding optimal parameter configurations in complex search spaces, widely used for chemical and materials optimization.

  • Project
    skilltraining-project/skill-training

    Skill Training lets an agent learn professional craft from finished human work without fine-tuning or labels. By corrupting a screenplay into an outline, reconstructing it, and turning the gap into editable Markdown skill cards, it improves writing behaviors that transfer to other models.

  • Project
    wilsonrljr/sysidentpy

    A Python library for system identification and time series forecasting using NARMAX models, offering tools for model structure selection and parameter estimation.

  • Project
    datawhalechina/agentic-rl-lab

    A collection of tutorials and reproducible code for state-of-the-art Agentic Reinforcement Learning algorithms, designed to lower the barrier to entry for researchers and developers.