facebookresearch/Ego4d

What it solves

It provides the infrastructure and tools to access, process, and train models on two of the world's largest egocentric video datasets: Ego4D and Ego-Exo4D. These datasets are designed to help AI researchers develop better first-person perception and understanding of human activities.

How it works

The project provides a Python module (ego4d) that acts as a toolkit for researchers. It includes command-line interfaces (CLIs) for downloading the massive amounts of video data, a simplified API for reading videos using libraries like TorchAudio and PyAV, and a feature extraction API with wrappers for models like Omnivore and SlowFast. It also includes research code for training models, such as the Contrastive Language Ego-centric video Pre-training (CLEP) implementation.

Who it’s for

AI and computer vision researchers focusing on egocentric (first-person) video analysis, human activity recognition, and multi-modal learning.

Highlights

• Massive Scale: Access to over 3,700 hours of annotated first-person video in Ego4D and over 1,286 hours of multi-view video in Ego-Exo4D.
• Multi-View Perspective: Ego-Exo4D provides synchronized first-person (Aria glasses) and third-person (GoPro) perspectives.
• Integrated Tooling: Includes dedicated CLI downloaders, feature extraction wrappers, and a visualization engine.
• Research Ready: Provides example notebooks and baseline research code for pre-training models on egocentric data.