naver/must3r

MUSt3R – Multi‑view Network for Stereo 3D Reconstruction

What it is – MUSt3R is a research‑grade Python library that implements the MUSt3R model, a multi‑view neural network for reconstructing dense 3‑D geometry from a collection of images. It builds on the earlier DUSt3R architecture, adding symmetric processing, a multi‑layer memory, and online camera‑pose estimation so that it can handle unordered image sets, video streams, or live webcam input.

Key capabilities

• Stereo‑style 3‑D reconstruction from any number of views (ordered or unordered).
• Online pose and structure estimation – the model predicts camera poses while it builds the 3‑D map, enabling SLAM‑like operation.
• Memory mechanism – a learnable token memory stores intermediate geometry, allowing the network to refine past frames and to retrieve relevant keyframes for unordered collections.
• Two model sizes – a 224‑pixel version (ViT‑L encoder, ViT‑B decoder) for fast prototyping and a 512‑pixel version for higher‑quality results.
• Retrieval mode – a lightweight “training‑free” encoder + codebook can be used to find matching keyframes when the input order is unknown.

How to get it

1. Create an environment (Python 3.11, CMake ≥ 3.14). The README recommends micromamba but any conda/virtualenv works.
2. Install PyTorch (2.7.0) with the CUDA build that matches your GPU.
3. Optional but recommended: install xformers for memory‑efficient attention.
4. Install the package directly from the repo:

   pip install must3r@git+https://github.com/naver/must3r.git
   # add [optional] extras for pillow‑heif, curope, etc.
   

5. For development you can clone the repo, install the submodules, and build the auxiliary asmk and optional RoPE CUDA kernels (curope).

Pre‑trained checkpoints

Model	Resolution	Encoder	Decoder	Download
MUSt3R_224_cvpr.pth	224×224	ViT‑L	ViT‑B	link
MUSt3R_512_cvpr.pth	512×384 … 512×160	ViT‑L	ViT‑B	link
MUSt3R_512.pth (finetuned)	same as above	ViT‑L	ViT‑B	link

Retrieval‑only weights (*_retrieval_trainingfree.pth) and their codebooks (*_retrieval_codebook.pkl) are also provided for the unordered‑image mode.

Demo interfaces

• Gradio + viser (offline) – Run python demo.py … (or the installed must3r_demo entry point). Images are uploaded via a Gradio UI; the 3‑D reconstruction is streamed to a viser web viewer that updates in real time. Options let you choose resolution, mixed‑precision (--amp bf16|fp16), and whether to expose the demo on the local network.
• Live visual‑odometry (online) demo – python slam.py … (installed as must3r_slam). Works with a webcam, a video file, or a folder of frames. The demo shows keyframe selection, pose tracking, and a 3‑D point cloud visualised with Open3D. Advanced flags control memory refresh, keyframe spacing, focal‑length estimation, and GPU‑accelerated RoPE kernels for speed.

Training The repository includes a full training pipeline (train.py, eval.py). Training uses a causal variant of the model (CausalMUSt3R) that processes blocks of 5 images with a causal attention mask, which speeds up gradient computation. The README lists important hyper‑parameters (memory size, dropout mode, mixed‑precision, number of views, etc.) and shows a sample torchrun command for multi‑GPU training.

License MUSt3R is released under a Non‑Commercial License. In addition to the repository’s LICENSE, the NOTICE file details the restrictive licenses of the training datasets (e.g., the “mapfree” dataset). Users must agree to both the MUSt3R license and the dataset licenses before using the provided checkpoints.

Who might use this

• Researchers exploring neural‑SLAM, multi‑view geometry, or dense reconstruction.
• Developers building AR/VR pipelines that need on‑the‑fly 3‑D mapping from a handheld camera.
• Anyone needing a ready‑to‑run demo for stereo reconstruction that can handle both ordered video streams and unordered photo collections.

---

All information above is taken directly from the repository’s README; no additional features have been inferred.