google-deepmind/representations4d

What it solves

This repository provides a collection of advanced video and vision representations designed to improve how AI models understand spatial (3D) and temporal (1D) dimensions—collectively referred to as 4D representations. It addresses the challenge of scaling self-supervised learning for non-semantic tasks like camera pose estimation, object tracking, and depth estimation.

How it works

The project implements several distinct architectural approaches to video representation learning:

• Scaling 4D Representations: Uses masked auto-encoding (MAE) with transformer video models to scale performance on spatial-temporal tasks.
• Moving Off-the-Grid (MooG): Employs cross-attention and positional embeddings to allow latent tokens to move freely across space and time, rather than being tied to a fixed pixel grid, enabling better object-centric tracking.
• Recurrent Video Masked Autoencoders (RVM): A recurrent transformer-based approach that uses an asymmetric masking objective and pixel reconstruction loss to achieve high parameter efficiency in video representation learning.
• Omnivorous Vision Encoder: A post-training recipe for models like DINOv2 to align features across different sensory views (e.g., RGB and depth maps) of the same scene.
• GenCeption: Converts a pre-trained video generative diffusion backbone (WAN 2.1) into a feed-forward perception model that performs various vision tasks via text instructions.

Who it’s for

This is for researchers and developers working on computer vision, video understanding, and 3D scene reconstruction who need high-performance, self-supervised encoders and backbones for downstream spatial-temporal tasks.

Highlights

• Massive Scaling: Demonstrates scaling of self-supervised video models from 20M to 22B parameters.
• Object-Centric Tracking: MooG allows tokens to bind to meaningful objects rather than pixels.
• Parameter Efficiency: RVM provides up to 30x greater parameter efficiency compared to other models without distillation.
• Generalist Perception: GenCeption enables a single model to handle depth, surface normals, and camera pose estimation using text prompts.