facebookresearch/boxer

What it solves

Boxer addresses the challenge of "lifting" 2D object detections (bounding boxes) from images into accurate 3D oriented bounding boxes (OBBs) in an open-world setting. It specifically focuses on indoor object detection, allowing users to identify objects in 3D space using only posed images and semi-dense point clouds.

How it works

Boxer uses a pipeline that combines 2D detection with 3D estimation. It leverages OWLv2 for open-vocabulary 2D detection (allowing users to prompt for specific objects via text) and a model called BoxerNet to lift these 2D boxes into 3D. The system can process single images or video sequences, utilizing camera intrinsics, 6 DoF poses, and gravity direction to orient the boxes correctly. It also supports online 3D tracking for temporal consistency across frames and offline 3D fusion to merge multiple detections into a single global 3D representation.

Who it’s for

This tool is designed for researchers and developers working in computer vision, robotics, and indoor scene understanding who need to convert 2D image detections into spatially accurate 3D object locations and orientations.

Highlights

• Open-Vocabulary Detection: Integrates OWLv2 to detect objects based on arbitrary text prompts.
• Robust 3D Lifting: Converts 2D bounding boxes into 3D oriented bounding boxes (OBBs).
• Temporal Consistency: Includes an online tracker to maintain object identity across video frames.
• Global Fusion: Supports post-hoc fusion to merge detections from multiple viewpoints into a static global map.
• Broad Dataset Support: Compatible with Project Aria, CA-1M, SUN-RGBD, and ScanNet data.