google-deepmind/tapnet

Tracking Any Point (TAP)

What it solves

This project provides a suite of tools and models for "Tracking Any Point" (TAP), which allows for the precise tracking of any trackable point on a solid physical surface across video frames. Unlike traditional box or segment tracking, TAP focuses on point-level precision, handling deformable surfaces and long-term tracking across diverse objects without requiring class-specific training.

How it works

The repository implements several evolving architectures:

  • TAPIR: A two-stage algorithm that first independently locates candidate point matches for a query point on every frame, then refines the trajectory and query features using local correlations.
  • TAPNext / TAPNext++: Formulates point tracking as a next-token prediction problem, propagating information through a network to track points. TAPNext++ specifically improves stability for long-term tracking, occlusions, and re-detection.
  • BootsTAP: A training method that uses large unlabeled real-world video datasets to improve accuracy by enforcing consistency across spatial transformations and corruptions.
  • TRAJAN: A point trajectory autoencoder that reconstructs missing point trajectories based on a set of support trajectories, allowing for the comparison of motion patterns across different videos.

Who it’s for

  • Computer Vision Researchers: Those working on point tracking, optical flow, and motion analysis.
  • Robotics Engineers: Developers using RoboTAP to execute manipulation tasks through visual imitation.
  • AI Developers: Anyone needing high-precision point tracking for video analysis or generative video evaluation.

Highlights

  • Multiple Benchmarks: Includes TAP-Vid, RoboTAP, and TAPVid-3D for 2D and 3D point tracking evaluation.
  • Diverse Model Options: Offers various checkpoints including TAPIR, BootsTAPIR, and TAPNext, with implementations in both Jax and PyTorch.
  • Real-time Capability: Includes a causal TAPIR model capable of real-time tracking on GPU platforms.
  • Robotics Integration: Specifically extends point tracking for real-world robotics manipulation via RoboTAP.