TencentARC/Track4World

What it solves

Track4World addresses the challenge of dense 3D tracking of every pixel in a monocular video. It enables the estimation of 3D scene flow between arbitrary frame pairs, allowing users to track the movement of all pixels in a global, world-centric coordinate system rather than just a few sparse points or camera-relative positions.

How it works

The project uses a feedforward approach to estimate dense 3D scene flow. It supports multiple depth backbones (such as Depth Anything v3, MoGe, and Pi3) to reconstruct 3D motion. For world-centric tracking, it can integrate with DINO and SAM2 to segment dynamic objects, which helps in separating foreground objects from the static background for better visualization and tracking accuracy.

Who it’s for

This tool is designed for researchers and developers working in computer vision, 3D scene reconstruction, and motion analysis who need high-density 3D tracking from single-camera video feeds.

Highlights

• Dense 3D Tracking: Tracks every pixel across frames in a world-centric coordinate system.
• Feedforward Estimation: Efficiently estimates 3D scene flow between arbitrary frame pairs.
• Metric Scale Support: Provides meter-level coordinate output when using the Depth Anything v3 backbone.
• Multi-Backbone Compatibility: Compatible with various depth estimation models including MoGe and Pi3.