Jumpat/SegAnyGAussians

What it solves

SAGA (Segment Any 3D GAussians) enables the segmentation of individual objects within a 3D scene reconstructed via 3D Gaussian Splatting (3DGS). It allows users to isolate and extract specific 3D objects from a complex scene using interactive prompts or open-vocabulary queries, solving the problem of 3D scenes being represented as a single, unsegmented cloud of Gaussians.

How it works

The system leverages the pre-trained Segment Anything Model (SAM) and CLIP features to associate 2D image masks and semantic features with 3D Gaussian attributes. It follows a multi-step pipeline:

1. 3DGS Training: A standard 3D Gaussian Splatting model is first trained on the scene.
2. Feature Extraction: 2D masks from SAM and CLIP features are extracted from the images.
3. Affinity Training: The project trains 3D Gaussian affinity features using a contrastive learning approach to ensure that Gaussians belonging to the same object are similar in feature space.
4. Segmentation: Users provide point prompts via a GUI or notebook, and the system calculates similarity to segment the 3D Gaussians.

Who it’s for

This tool is designed for researchers and developers working in 3D computer vision, scene reconstruction, and augmented reality who need to be able to identify and isolate specific objects within a 3D-GS reconstructed environment.

Highlights

• Interactive GUI: Includes a dearpygui-based interface for rotating, panning, and inputting point prompts to segment objects in real-time.
• Open-Vocabulary Segmentation: Supports querying objects using text-based descriptions via CLIP features (implemented in Jupyter Notebook).
• 3D Clustering: Integrates HDBSCAN to perform 3D clustering for object decomposition.
• Flexible Rendering: Ability to render the segmented objects separately from the background or generate 2D rendered masks.