nv-tlabs/3dgrut
Ray tracing and hybrid rasterization of Gaussian particles
What it solves
This project addresses the limitations of traditional 3D Gaussian Splatting (3DGS), specifically the inability to handle distorted cameras (like those with rolling shutters) and the difficulty of rendering complex optical phenomena such as reflections, refractions, and shadows.
How it works
The project implements three complementary techniques:
- 3D Gaussian Ray Tracing (3DGRT): Instead of splatting, it performs ray tracing of volumetric Gaussian particles. This allows for the simulation of secondary rays (reflections/refractions) and support for complex camera distortions, though it requires dedicated ray-tracing hardware.
- 3D Gaussian Unscented Transform (3DGUT): A rasterization-based framework that enables support for distorted cameras while maintaining the high efficiency of traditional rasterization.
- 3DGRUT: A hybrid approach that combines both methods, using rasterization for primary rays (for speed) and ray tracing for secondary rays (for visual complexity).
Who it’s for
It is designed for researchers and developers working on neural reconstruction and 3D scene rendering who need to support non-ideal camera effects or high-fidelity optical simulations.
Highlights
- Hybrid Rendering: Combines rasterization and ray tracing to balance performance and visual quality.
- Hardware Acceleration: Optimized for NVIDIA GPUs with RT cores.
- Broad Export Support: Trained scenes can be exported to USD (ParticleField), NuRec USDZ for Omniverse, or PLY formats.
- Advanced Features: Supports Neural Harmonic Textures (NHT), MCMC densification, and physically plausible image signal processing (PPISP).