NVlabs/nvdiffrec

Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".

What it solves

It enables the extraction of high-quality triangular 3D models, including their materials and lighting, from a set of multi-view image observations. This allows for the joint optimization of a scene's topology, surface appearance, and lighting environment simultaneously.

How it works

The system uses a differentiable rendering pipeline to optimize 3D assets. It employs differentiable marching tetrahedrons (adapted from NVIDIA's Kaolin library) and supports advanced isosurfacing techniques like FlexiCubes. To simplify the autodiff process, a version of the library is available that uses slangpy to leverage the autodiff capabilities of Slang instead of manually crafted CUDA extensions.

Who it’s for

It is designed for researchers and developers in computer vision and 3D graphics who have access to high-end NVIDIA GPUs with significant memory.

Highlights

  • Jointly optimizes geometry (topology), materials, and lighting.
  • Produces standard triangular 3D meshes.
  • Supports FlexiCubes for improved isosurfacing.
  • Integrates with nvdiffrast and tiny-cuda-nn for performance.

Related

  • Project
    NVlabs/nvdiffrecmc

    A PyTorch implementation for jointly reconstructing 3D topology, materials, and lighting from multi-view images using Monte Carlo rendering and denoising.

  • Project
    facebookresearch/pytorch3d

    A PyTorch-based library for 3D Computer Vision research providing differentiable rendering and efficient tools for manipulating 3D meshes and point clouds.

  • Project
    mitsuba-renderer/mitsuba3

    A research-oriented, differentiable rendering system for forward and inverse light transport simulation with deep Python integration.

  • Project
    nachifur/RDDM

    A residual denoising diffusion model framework for high-quality image restoration and generation, supporting tasks like deraining and deblurring.

  • Project
    NVIDIA/NeMo-Retriever

    A scalable framework for extracting text, tables, charts, and infographics from documents to create embeddings for RAG and generative AI applications.