apple-aiml-research/ml-matrix3d

What it solves

Matrix3D provides a unified model for photogrammetry, replacing the need for separate specialized models to handle different 3D vision tasks. It allows users to perform pose estimation, depth prediction, and novel view synthesis within a single framework, even when starting with unposed images or a single view.

How it works

The system uses a large photogrammetry model that employs masked inference to dynamically switch between or combine different tasks. By manipulating modality flags and view identifiers, the model can process various input states (such as a single image or multiple posed images) to predict 3D properties. These outputs can then be fed into a modified 3D Gaussian Splatting (3DGS) pipeline to generate full 3D reconstructions.

Who it’s for

This tool is designed for researchers and developers working in computer vision and 3D reconstruction who need a flexible way to estimate camera poses, predict depth, or synthesize new views from limited image data.

Highlights

• All-in-One Architecture: A single model that handles pose estimation, depth prediction, and novel view synthesis.
• Flexible Input: Supports tasks ranging from single-view to 3D reconstruction to unposed few-shot 3D reconstruction.
• 3DGS Integration: Includes a modified 3D Gaussian Splatting pipeline to turn model predictions into 3D scenes.
• Masked Inference: Allows for the dynamic composition of photogrammetry subtasks based on the input configuration.