Introduction to 3D Gaussian Splatting

3D Gaussian Splatting is a rasterization technique that allows for the real-time rendering of photorealistic scenes derived from small samples of images. It represents a shift from traditional triangle-based rasterization to a Gaussian-based approach, enabling high-fidelity visual reconstruction of 3D environments.

Core Mechanism of 3D Gaussian Splatting

3D Gaussian Splatting functions as a rasterization technique, meaning it takes data describing a scene and draws that data onto a screen. Unlike traditional computer graphics that rely on triangle rasterization, this method uses Gaussians to represent the scene.

Each individual Gaussian is defined by four primary parameters:

  • Position: The XYZ coordinates of the Gaussian.
  • Covariance: A 3x3 matrix defining how the Gaussian is stretched or scaled.
  • Color: The RGB value of the Gaussian.
  • Alpha: The transparency level (\u03b1) of the Gaussian.

By rasterizing millions of these Gaussians—sometimes up to 7 million for a single scene—the system can create a dense, photorealistic representation of a 3D object or environment.

The Technical Pipeline

The process of creating a 3D Gaussian Splatting scene involves four distinct stages:

1. Structure from Motion (SfM)

The pipeline begins with Structure from Motion to estimate a 3D point cloud from a set of 2D images. This is typically achieved using the COLMAP library.

2. Conversion to Gaussians

Each point in the estimated point cloud is converted into a Gaussian. While this allows for immediate rasterization, the initial conversion only provides position and color data. To achieve high-quality results, the representation must undergo a training process.

3. Training via Stochastic Gradient Descent

Training is performed using Stochastic Gradient Descent (SGD) to optimize the Gaussian parameters. The process follows these steps:

  1. Rasterization: Gaussians are rendered into an image using a differentiable rasterizer.
  2. Loss Calculation: The difference between the rasterized image and the ground truth image is calculated.
  3. Parameter Adjustment: Gaussian parameters are adjusted based on the loss.
  4. Densification and Pruning: The system automatically manages the number of Gaussians. If a gradient is large, small Gaussians are cloned and large Gaussians are split. Conversely, if a Gaussian's alpha value becomes too low, it is removed (pruned).

4. Differentiable Gaussian Rasterization

To enable training via SGD, the rasterizer must be differentiable. The rendering process involves projecting each Gaussian into 2D from the camera's perspective, sorting them by depth, and blending them front-to-back for each pixel.

Performance Trade-offs and Limitations

While 3D Gaussian Splatting offers high-quality, real-time performance, it introduces specific technical constraints:

Advantages

  • High-quality, photorealistic scene reconstruction.
  • Fast, real-time rasterization speeds.
  • Relatively fast training times.

Disadvantages

  • High VRAM Usage: Requires 4GB of VRAM for viewing and 12GB for training.
  • Large Storage Requirements: A single scene can exceed 1GB on disk.
  • Pipeline Incompatibility: The original CUDA implementation is not natively compatible with existing production rendering pipelines like Vulkan, DirectX, or WebGPU.
  • Static Nature: The original method is designed for static scenes.

Industry Implications and Future Outlook

3D Gaussian Splatting provides a dense representation of 3D space, which may have significant implications for Embodied AI research, where representing 3D space accurately is a primary challenge. Furthermore, research into Dynamic 3D Gaussians suggests that the technique could eventually support animation.

Regarding production integration, the primary bottleneck is the efficient sorting of millions of Gaussians. The original implementation uses CUB device radix sort, which is a CUDA-specific optimization. However, several adaptations have already emerged, including viewers for WebGPU, WebGL, and Unity. While some of these use quad-based rasterization—which can potentially lower quality or performance—optimization tricks are already demonstrating that high-quality results are possible outside of the CUDA environment.

Sources