mitsuba-renderer/mitsuba3

Mitsuba 3: A Retargetable Forward and Inverse Renderer

What it solves

Mitsuba 3 provides a research-oriented system for simulating both forward and inverse light transport. It allows researchers to create highly realistic images and, crucially, perform "inverse rendering" by calculating how changes in input parameters (like camera position or material properties) affect the final image.

How it works

It uses a core library and a plugin system for materials and light sources. The system is built on Dr.Jit, a specialized just-in-time (JIT) compiler that allows the same code to be retargeted for different tasks, such as switching between simple RGB rendering and complex differential spectral transport on a GPU. It integrates deeply with Python, allowing users to develop rendering algorithms that are JIT-compiled and differentiated on the fly.

Who it’s for

It is designed for researchers in computer graphics and other disciplines who need a high-performance, differentiable rendering system for experimentation.

Highlights

  • Differentiable Rendering: Can compute derivatives of the entire simulation relative to inputs like geometry, textures, and camera pose.
  • Python-First: Materials and algorithms can be written in Python and compiled on the fly.
  • High Performance: Uses LLVM for CPUs and CUDA/OptiX for NVIDIA GPUs with hardware acceleration.
  • Flexible Modalities: Supports monochromatic, RGB, and spectral rendering, with optional polarization support.

Related

  • Project
    NVlabs/nvdiffrec

    A differentiable rendering framework that extracts triangular 3D models, materials, and lighting from multi-view images through joint optimization.

  • Project
    nv-tlabs/3dgrut

    A hybrid 3D rendering framework that combines Gaussian rasterization and ray tracing to support distorted cameras and complex light effects like reflections and shadows.

  • Project
    vccimaging/DeepLens

    A differentiable optical lens simulator for computational imaging and optical design that allows lenses to be optimized as learnable parameters within a PyTorch training loop.

  • Project
    NVIDIA-AI-IOT/Lidar_AI_Solution

    A collection of highly optimized CUDA and TensorRT implementations for 3D Lidar perception, speeding up sparse convolutions and sensor fusion for self-driving systems.

  • Project
    fastvideo/gpu-camera-sample

    A high-performance GPU-accelerated camera application for real-time raw image processing and ISP pipelines, supporting a wide range of industrial cameras.