nv-tlabs/3dgrut

Ray tracing and hybrid rasterization of Gaussian particles

What it solves

3DGRUT addresses the limitations of traditional Gaussian Splatting (3DGS) and ray tracing. While 3DGS is fast but struggles with distorted cameras (like rolling shutters) and complex light effects, traditional ray tracing is more flexible but often too slow for real-time use. 3DGRUT provides a hybrid approach to render high-quality 3D scenes with support for complex camera distortions and secondary light effects (reflections, refractions, and shadows).

How it works

The project implements three distinct rendering techniques:

  • 3D Gaussian Ray Tracing (3DGRT): Performs ray tracing of volumetric Gaussian particles, enabling support for distorted cameras and secondary rays for complex light phenomena.
  • 3D Gaussian Unscented Transform (3DGUT): A rasterization-based framework that supports distorted cameras while maintaining the high efficiency of rasterization.
  • 3DGRUT: A hybrid method that renders primary rays via rasterization (for speed) and secondary rays via ray tracing (for visual fidelity).

Who it’s for

Researchers and developers working in 3D reconstruction, neural rendering, and computer graphics who need to handle distorted camera inputs or simulate realistic light transport in Gaussian-based scenes.

Highlights

  • Hybrid Rendering: Combines rasterization and ray tracing to balance performance and visual quality.
  • Distorted Camera Support: Handles complex, time-dependent effects like rolling shutters.
  • Advanced Light Effects: Efficiently simulates reflections, refractions, and shadows.
  • Extensible Framework: Supports Neural Harmonic Textures (NHT), MCMC densification, and physically plausible ISP post-processing.
  • Production Ready: Includes export options for USD (ParticleField), PLY, and NuRec formats for use in tools like NVIDIA Omniverse.

Related

  • Dispatch
    Introduction to 3D Gaussian Splatting

    3D Gaussian Splatting is a rasterization technique that enables real-time rendering of photorealistic 3D scenes learned from a small set of images.

  • Project
    zju3dv/InfiniSplat

    InfiniSplat is a system for large-baseline monocular view synthesis that reconstructs 3D Gaussian Splatting scenes from a single RGB image or an RGB-depth pair.

  • Project
    Linketic/CityGaussian

    A framework for high-quality, large-scale 3D scene reconstruction using Gaussian Splatting, featuring multi-GPU support and geometrically accurate rendering.

  • Project
    nerfstudio-project/gsplat

    A CUDA-accelerated library for the rasterization of 3D Gaussians, providing a faster and more memory-efficient way to render radiance fields compared to the official implementation.

  • Project
    inspatio/querysplat

    QuerySplat is a 3D Gaussian Splatting prediction framework that decouples geometry and appearance representations to improve 3D scene reconstruction from images.