NVlabs/nvdiffrecmc

Official code for the NeurIPS 2022 paper "Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising".

What it solves

It addresses the challenge of decomposing a scene into its fundamental components—topology (shape), materials, and lighting—using only multi-view image observations.

How it works

The project implements a method that jointly optimizes the 3D geometry, material properties, and environment lighting. It utilizes Monte Carlo rendering and denoising to refine these elements based on the provided images, allowing for the extraction of a 3D model that can be exported to software like Blender.

Who it’s for

It is designed for researchers and developers in computer vision and graphics who have access to high-end NVIDIA GPUs (supporting OptiX 7.3+) and want to reconstruct detailed 3D assets from images.

Highlights

  • Jointly optimizes shape, materials, and lighting from multi-view images.
  • Supports extraction of diffuse albedo textures.
  • Includes a script for importing reconstructed geometry and lighting into Blender.
  • Provides configurations for reproducing results from the NeRF, NeRD, and NeRFactor datasets.

Related

  • Project
    facebookresearch/pytorch3d

    A PyTorch-based library for 3D Computer Vision research providing differentiable rendering and efficient tools for manipulating 3D meshes and point clouds.

  • Project
    hanna-xu/U2Fusion

    A unified unsupervised image fusion network that merges multi-modal, multi-exposure, and multi-focus images into a single high-quality image without requiring labeled training data.

  • Project
    NVlabs/nvdiffrec

    A differentiable rendering framework that extracts triangular 3D models, materials, and lighting from multi-view images through joint optimization.

  • Project
    TorchIO-project/torchio

    A Python library for PyTorch that provides tools for the efficient loading, preprocessing, and augmentation of 3D medical images.

  • Project
    pytorch/vision

    A PyTorch library providing popular datasets, model architectures, and image transformations for computer vision.