3dgrut: a hybrid Gaussian Splatting framework supporting ray-traced reflections and distorted camera effects

What it solves

3DGRUT addresses the limitations of traditional Gaussian Splatting (3DGS) by enabling the rendering of complex optical phenomena—such as reflections, refractions, and shadows—and supporting distorted cameras with time-dependent effects like rolling shutters. It provides a way to balance the high visual fidelity and flexibility of ray tracing with the speed of rasterization.

How it works

The project implements three distinct but related rendering approaches:

• 3D Gaussian Ray Tracing (3DGRT): Instead of splatting, this method performs ray tracing of volumetric Gaussian particles. This allows for secondary rays (reflections/refractions) and complex camera distortions, though it requires dedicated ray-tracing hardware and is slower than rasterization.
• 3D Gaussian Unscented Transform (3DGUT): This method brings support for distorted cameras and time-dependent effects into a rasterization framework, maintaining high efficiency.
• 3DGRUT (Hybrid): A hybrid technique that renders primary rays via rasterization (for speed) and secondary rays via ray tracing (for visual effects like reflections), combining the strengths of both.

Who it’s for

Researchers and developers working in 3D scene reconstruction, computer graphics, and neural rendering who need to simulate complex lighting effects or handle non-ideal camera hardware.

Highlights

• Hybrid Rendering: Combines rasterization and ray tracing to optimize performance and flexibility.
• Hardware Acceleration: Optimized for NVIDIA GPUs with RT cores.
• Broad Export Support: Trained scenes can be exported to USD (ParticleField), NuRec USDZ for Omniverse, or PLY formats.
• Advanced Training Features: Supports MCMC densification, selective Adam optimization, and physically plausible image signal processing (PPISP).