OpenEnvision/Awesome-Multimodal-Modeling
Awesome Multimodal Modeling [Covers MLLM, UMM, and NMM]
📚 What is Awesome‑Multimodal‑Modeling?
Awesome‑Multimodal‑Modeling is a curated, community‑maintained list that surveys the whole landscape of multimodal machine‑learning models – systems that process more than one type of data (e.g., images + text). The repository does not contain code for a single model; instead it organises papers, model collections, and resources into a clear taxonomy that reflects how modern research thinks about multimodal architectures.
🎯 Goal & Audience
- Goal: Give researchers, students, and engineers a single place to see what models exist, how they differ architecturally, and where to find their implementations (often on Hugging Face). It also tracks a short‑term “survey paper” that will be released later.
- Audience: Anyone who wants to understand the evolution from early‑fusion models (e.g., CLIP) to the newest native multimodal models that are trained from scratch, as well as developers looking for ready‑to‑use model collections.
🗂️ How the List Is Structured
| Section | What you’ll find |
|---|---|
| 1️⃣ Introduction & Definitions | Precise, architecture‑first definitions of four model families: • Traditional (pre‑2023 fusion‑focused) • Multimodal Large Language Models (MLLMs) – pretrained vision encoder + pretrained LLM linked together • Unified Multimodal Models (UMMs) – a single framework that can both understand and generate across modalities • Native Multimodal Models (NMMs) – trained from scratch on all modalities (early‑fusion or late‑fusion) |
| 2️⃣ Traditional Multimodal Models | Papers on multimodal representation, alignment, and pre‑training (e.g., CLIP, ALIGN, ViLBERT, BLIP). |
| 3️⃣ MLLMs | Taxonomy based on the visual‑adapter used (MLP projector, Q‑Former, cross‑attention, hybrid). Lists notable models such as LLaVA, Qwen‑VL, InternVL, MiniCPM‑V, CogVLM. |
| 4️⃣ Unified Multimodal Models (UMMs) | Break‑down by generation paradigm (diffusion‑based, autoregressive, hybrid) and by “any‑to‑any” capability. |
| 5️⃣ Native Multimodal Models (NMMs) | Design analyses, scaling‑law discussions, and concrete early‑fusion (e.g., Emu3) and late‑fusion examples. |
| 6️⃣ Closed‑Source Models | Chronological list of proprietary multimodal systems (2023‑2026). |
| 7️⃣ Resources | Links to related awesome‑lists, survey slides, code repos, and tools. |
🔧 How to Contribute
- PRs welcome – the repo uses a simple validation rule set (entries must have a reliable source, fit the taxonomy, and include a consistent format). See the How to Contribute section for entry templates.
- Model zoos – the maintainers also host two Hugging Face collections (UMM Zoo and NMM Zoo) that link directly to open‑source implementations.
📢 Recent News (as of Q3 2026)
- Two new model collections on Hugging Face: a Unified Multimodal Model Zoo and a Native Multimodal Model Zoo.
- The repo hit 100 ⭐ in one day after its launch in April 2026.
- A comprehensive survey paper is promised soon, building on the taxonomy defined here.
📚 Why It Matters
Multimodal AI is moving fast, and papers often use overlapping terminology (e.g., “multimodal LLM”, “unified model”). This list standardises the vocabulary and highlights architectural differences that matter for research and product development:
- Understanding vs. generation capabilities
- Whether a model re‑uses pretrained backbones or is trained from scratch
- The fusion strategy (early vs. late) and its impact on scalability
Having a single, well‑organized reference saves time, reduces confusion, and helps newcomers pick the right model family for their use‑case.
📄 License
The list is released under CC0 1.0 (public domain), so anyone can copy, remix, or redistribute it without restriction.
Related
- Dispatch
- Dispatch
- Dispatch
- Dispatch
- Dispatch