nianticlabs/map-free-reloc

[ECCV 2022] Map-free Visual Relocalization: Metric Pose Relative to a Single Image

What it solves

This project addresses the challenge of visual relocalization—determining a camera's position and orientation (pose) relative to a scene. Traditional methods require extensive 3D maps built from hundreds of images and scale calibration. This project introduces "Map-free Relocalization," which allows for instant, metric-scaled relocalization using only a single reference photo of a scene.

How it works

The system estimates the metric relative pose between a reference image and a query image. It provides two primary approaches for this:

  1. Relative Pose Regression (RPR): Uses neural networks to directly regress the 6D rotation and 3D translation between two images. The project provides various encoders, aggregators, and loss functions to customize these models.
  2. Feature Matching + Depth Estimation: Combines feature matching (using tools like SuperPoint, SuperGlue, or LoFTR) with scale estimation from depth models (such as DPT KITTI or DPT NYU) and solvers (Essential Matrix or PnP) to calculate the pose.

Who it’s for

Researchers and developers working in computer vision, robotics, and augmented reality who need to localize a device within a scene without the overhead of creating a full 3D map.

Highlights

  • Map-free Approach: Eliminates the need for large scene-specific 3D maps and scale calibration.
  • New Dataset: Includes a crowd-sourced dataset of 655 outdoor scenes with metric poses.
  • Flexible Baselines: Provides implementations for both direct regression and feature-matching pipelines.
  • Benchmarking Tools: Includes a reference PyTorch dataloader, submission scripts, and a public leaderboard for evaluation.