apple-aiml-research/ml-4m

What it solves

4M provides a framework for training "any-to-any" multimodal foundation models. It addresses the challenge of scaling AI models to handle dozens of different modalities and tasks within a single architecture, allowing for flexible generation and transfer between diverse data types.

How it works

The framework uses a combination of tokenization and masked modeling to process diverse modalities. By converting different data types into tokens, the model can be trained to predict masked tokens, enabling it to learn representations that work across various modalities. This allows the model to perform tasks such as RGB-to-all generation (generating multiple modalities from a single RGB image) or caption-to-all generation (generating multiple modalities from text).

Who it’s for

This project is designed for AI researchers and developers working on multimodal foundation models, specifically those interested in any-to-any generation, vision tasks, and scalable multimodal training.

Highlights

• Any-to-Any Capability: Supports generation across tens of modalities and tasks.
• Scalable Architecture: Provides models ranging from 198M to 2.8B parameters (4M-B, 4M-L, 4M-XL).
• Diverse Modality Support: Includes tokenizers for RGB, depth, normals, edges, semantic segmentation, CLIP, DINOv2, ImageBind, SAM instances, and 3D human poses.
• Specialized Variants: Offers text-to-image specialist models and super-resolution models.
• Open-sourced: Provides both the training framework and pre-trained checkpoints via Hugging Face Hub.