facebookresearch/mmf

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

What it solves

MMF provides a modular framework to simplify the development and research of multimodal models that combine vision and language. It serves as a scalable starter codebase for researchers to bootstrap new projects or participate in vision-language challenges like TextVQA and VQA.

How it works

Built on PyTorch, MMF is an un-opinionated and fast framework that supports distributed training. It provides reference implementations of state-of-the-art vision and language models, allowing users to integrate datasets and models to build multimodal research projects.

Who it’s for

AI researchers and developers working on multimodal AI, specifically those focusing on the intersection of vision and language.

Highlights

  • Modular design for flexible multimodal research.
  • Reference implementations of state-of-the-art models.
  • Support for distributed training via PyTorch.
  • Starter codebase for major vision-language challenges (e.g., The Hateful Memes, TextVQA, TextCaps, and VQA).

Related

  • Project
    open-mmlab/mmengine

    A foundational PyTorch library that serves as a training engine for deep learning models, providing standardized components for training loops, configuration, and monitoring.

  • Project
    kengz/SLM-Lab

    A modular PyTorch framework for Deep Reinforcement Learning that allows users to configure and run RL experiments via JSON specs across various Gymnasium environments.

  • Project
    facebookresearch/multimodal

    A PyTorch library for training state-of-the-art multimodal models at scale, providing modular building blocks and pretrained weights for models like CLIP and BLIP-2.

  • Project
    pyg-team/pytorch-frame

    A modular deep learning extension for PyTorch designed to build neural network models on heterogeneous tabular data containing mixed column types like text, images, and numerical values.

  • Project
    Osilly/Vision-DeepResearch

    A framework and benchmark for developing Multimodal Large Language Models capable of deep research through iterative visual and textual search across images and videos.