facebookresearch/mae_st

Official Open Source code for "Masked Autoencoders As Spatiotemporal Learners"

What it solves

This project provides a PyTorch implementation of Masked Autoencoders (MAE) adapted for spatiotemporal learning, allowing models to learn powerful video representations by reconstructing missing parts of a video sequence.

How it works

The system uses a masking strategy where a large portion of the video (up to 98% in demos) is masked out. The model is then tasked with reconstructing the original video frames, forcing it to learn the spatial and temporal relationships between frames to fill in the gaps.

Who it’s for

This is intended for researchers and developers working on video understanding, self-supervised learning, and computer vision tasks that require pre-trained spatiotemporal models.

Highlights

  • Pre-trained checkpoints for ViT-Large and ViT-Huge architectures on Kinetics-400, 600, and 700 datasets.
  • Support for both pre-training and fine-tuning workflows.
  • Interactive visualization demo to see the model's reconstruction capabilities.
  • Built on top of the PyTorch and timm library.

Related

  • Project
    facebookresearch/multimodal

    A PyTorch library for training state-of-the-art multimodal models at scale, providing modular building blocks and pretrained weights for models like CLIP and BLIP-2.

  • Project
    pyg-team/pytorch-frame

    A modular deep learning extension for PyTorch designed to build neural network models on heterogeneous tabular data containing mixed column types like text, images, and numerical values.

  • Project
    InterDigitalInc/CompressAI

    A PyTorch library and evaluation platform for end-to-end deep learning-based image and video compression research.

  • Project
    sstary/SSRS

    A PyTorch collection of deep learning models for remote sensing semantic segmentation, featuring single-modal, multimodal fusion, and unsupervised domain adaptation techniques.

  • Project
    pytorch/pytorch

    A Python-based machine learning framework that provides GPU-accelerated tensor computation and a dynamic autograd system for building deep neural networks.