GREAT-WHU/MASt3R-Fusion

Integrating Feed-Forward Visual Model with IMU, GNSS for High-Functionality SLAM.

What it solves

MASt3R-Fusion addresses the challenge of creating high-accuracy, robust 3D perception and mapping (SLAM) for large-scale environments. It specifically solves the problem of maintaining metric scale and global consistency in monocular visual-inertial systems by integrating deep learning-based pointmap regression with multi-sensor data.

How it works

The system tightly integrates a feed-forward visual model (based on MASt3R) for pointmap regression with data from an Inertial Measurement Unit (IMU) and Global Navigation Satellite System (GNSS). It employs a real-time sliding-window tracking mechanism combined with a factor graph (implemented via a modified GTSAM) for globally consistent optimization. The pipeline includes loop closure detection to handle cross-temporal data association and supports automatic calibration of intrinsics and poses.

Who it’s for

This project is designed for researchers and engineers working on robotics, autonomous vehicles, and 3D mapping applications that require precise, large-scale spatial awareness and multi-sensor fusion.

Highlights

  • Multi-Sensor Fusion: Tightly integrates visual data with IMU and GNSS for improved robustness and metric scale.
  • Global Consistency: Combines real-time tracking with global optimization and loop closure detection.
  • Efficient Memory Usage: Capable of processing arbitrarily long sequences using only 8 GB of GPU memory.
  • Full-View Depth: Provides comprehensive depth estimation for the entire field of view.
  • Auto Calibration: Supports joint optimization of camera intrinsics and poses.

Related

  • Project
    NVlabs/SimFoundry

    SimFoundry converts real-world videos into physics-ready simulation scenes for robotics, automating object reconstruction and the generation of diverse object variations.

  • Project
    DevYangJC/Argus

    Argus is an open‑source, enterprise‑grade RAG platform that ingests private documents, builds both vector and keyword indexes, and answers questions with a LLM that must cite real sources. It includes a Spring AI‑based ReactAgent for chat, hybrid retrieval with RRF fusion, multi‑level evidence scoring, role‑based security, and a Vue 3 front‑end.

  • Project
    aws-deepracer-community/deepracer-for-cloud

    A toolkit for running AWS DeepRacer training environments on cloud VMs or local computers, enabling independent, scalable reinforcement learning for autonomous racing.

  • Project
    BAAI-DCAI/SpatialBot

    SpatialBot is a Vision-Language-Action model that enhances spatial understanding and depth perception in VLMs, enabling precise spatial reasoning and robot manipulation.