MIT-SPARK/DAAAM
Describe Anything, Anywhere, at Any Moment (DAAAM), a novel approach to real-time, large-scale, spatio-temporal memory
What it solves
DAAAM solves the challenge of creating real-time, semantically rich 3D maps of dynamic environments. It allows robots to understand "where" and "what" things are in a space, including moving objects, by building a 4D scene graph on the fly.
How it works
The system uses a "foundation-model-first" approach. It combines Segment Anything Model (SAM) for segmentation, BotSort for tracking, and Vision-Language Models (VLMs) for grounding. These inputs are fed into the Hydra framework to construct a hierarchical 4D scene graph that represents the objects and their relationships in a 3D space over time.
Who it’s for
This project is designed for robotics researchers and developers building autonomous systems that need high-level semantic understanding of their surroundings in real-time.
Highlights
- Real-time performance for 4D scene graph construction.
- Integration of SAM, BotSort, and VLMs for semantic grounding.
- State-of-the-art results on NaVQA and SG3D benchmarks.
- Optimization-based frontend for generating semantic descriptions from localized captioning models.
Related
- Project
- Project
- Project
- Project