google-deepmind/tapnet

What is TAP‑Net?

TAP‑Net is the codebase for Tracking Any Point (TAP), a research project from Google DeepMind that tackles the problem of following an arbitrary pixel‑level point through a video. Unlike classic object‑tracking (which follows whole boxes) or optical flow (which works only on short windows), TAP asks a model to receive a single query point on a single frame and then predict where that exact point moves to (or if it becomes invisible) in every other frame, even on deformable surfaces, through occlusions, and over long time spans.

Main components shipped in the repo

Component	What it does	How you can use it
TAP‑Vid & TAPVid‑3D datasets	Large benchmark collections (2‑D and 3‑D) with ground‑truth point tracks for real and synthetic videos.	Download the data and run the provided evaluation scripts to measure a model’s accuracy on the TAP task.
TAPIR	A two‑stage point‑tracking model (matching + refinement) that runs fast and sets the state‑of‑the‑art on TAP‑Vid.	Pre‑trained checkpoints (JAX & PyTorch) are provided; you can run inference via the Colab notebooks or the real‑time demo script.
BootsTAPIR	Same architecture as TAPIR but trained with a self‑supervised “bootstrapped” loss on massive unlabeled video, giving a noticeable boost in accuracy.	Same usage as TAPIR; just load the BootsTAPIR checkpoint.
TAPNext / TAPNext++	A newer, simpler tracker that treats point tracking as a next‑token prediction problem. TAPNext++ adds long‑term stability, occlusion handling and re‑detection.	Run via the provided Colab notebooks (JAX or PyTorch) or the offline demo.
RoboTAP	Demonstrates how TAPIR point tracks can be turned into few‑shot visual‑imitation policies for robot manipulation. Includes a robotics‑specific dataset and clustering code.	Use the clustering notebook to turn point tracks into action primitives for imitation learning.
TRAJAN	A trajectory auto‑encoder that learns a latent space for sets of point tracks, useful for comparing motion distributions or evaluating generative video models.	Run the TRAJAN Colab demo to encode/decode trajectories.
Training scripts	JAX‑based training pipelines for the original TAP‑Net baseline and for TAPIR on the synthetic Kubric data.	Follow the tapnet/training README to reproduce the models or fine‑tune on your own data.

Getting started quickly

1. Colab demos – The repository ships with a dozen ready‑to‑run notebooks (e.g., torch_tapnextpp_demo.ipynb, tapir_demo.ipynb). Open any of them on Google Colab, press Run all, and you can upload your own video to see a point tracked in seconds.
2. Live demo – Clone the repo, install the tapnet package, download a checkpoint (e.g., causal_tapir_checkpoint.npy), and run python -m tapnet.live_demo. On a modest RTX 4000 you’ll get ~17 fps on 480×480 frames.
3. Evaluation – Use the tapnet/tapvid and tapnet/tapvid3d folders to download the benchmark data and call the provided metric scripts to compute the standard “AJ” (average Jaccard) scores.

Why it matters

• Fine‑grained motion understanding – Point‑level tracking is a more precise probe of a model’s grasp of geometry and dynamics than bounding‑box tracking.
• Robotics bridge – RoboTAP shows that accurate point tracks can be directly leveraged for imitation learning, turning visual data into robot actions.
• Research platform – The repo bundles datasets, pretrained weights (JAX + PyTorch), training code, and a suite of demos, making it a one‑stop shop for anyone wanting to experiment with point‑tracking or build on top of it.

Resources linked from the README

• Project webpages: TAP‑Vid, TAPIR, RoboTAP, BootsTAP, TAPVid‑3D, TAPNext, TRAJAN, TAPNext++
• Blog post and paper links for deeper background
• HuggingFace hub for downloading checkpoints
• Kubric visualisation notebook for generating synthetic ground‑truth tracks

---

TL;DR: TAP‑Net is a genuine, research‑grade open‑source project that provides datasets, state‑of‑the‑art models, training pipelines, and interactive demos for the Tracking Any Point problem—a fine‑grained video‑tracking task with applications in computer vision and robotics.