MIT-SPARK/VGGT-SLAM

VGGT-SLAM: Dense RGB SLAM Optimized on the SL(4) Manifold

What it solves

VGGT-SLAM provides a way to perform real-time, dense scene reconstruction and mapping. It allows a system to build a detailed 3D map of an environment as it moves through it, while simultaneously estimating its own position (SLAM - Simultaneous Localization and Mapping).

How it works

The system uses a feed-forward approach for dense reconstruction. It integrates several components to build the map incrementally:

  • Perception Encoder and SAM 3 are used for optional open-set 3D object detection, allowing users to query specific objects (e.g., "coffee machine") and see them highlighted with 3D bounding boxes on the map.
  • It supports both offline processing of image folders and online real-time operation using a RealSense camera.
  • It utilizes SL(4) manifold optimization for mapping accuracy.

Who it’s for

This project is designed for researchers and developers working in robotics, autonomous navigation, and 3D scene reconstruction.

Highlights

  • Real-time Performance: Capable of dense feed-forward scene reconstruction in real-time.
  • Open-set Object Detection: Supports querying for arbitrary objects in 3D space using text queries.
  • Hardware Integration: Direct support for RealSense cameras for live mapping.
  • Flexible Visualization: Uses Viser for incremental map construction and point cloud visualization.