zli12321/Vision-Language-Models-Overview

What this project is.

It's a continuously updated collection of vision-language model (VLM) papers, models, benchmark links, and GitHub repositories. To use the README's own description: it compiles awesome papers, models, and model-repo links covering state-of-the-art VLMs, their benchmarks, post-training/alignment methods, and applications. It also includes its own published survey paper (CVPR 2025 workshop) and a live companion dashboard called VLM Trends.

The central narrative: four architecture eras in six years.

The README frames the field's history as a clear story, shown in a diagram and explained in the text:

• Era 1 – Two-tower models (e.g., CLIP, BLIP-2, Flamingo): vision and language are separate 'towers'; one is fused into the other with a learned bridge, output is text-only.
• Era 2 – LLM-as-trunk (e.g., LLaVA, Qwen2.5-VL, GPT-4V): a pretrained large language model becomes the core, and vision is bolted on as an adapter — still text-only output.
• Era 3a – Native multimodal input, text out: image/video/audio enter one fused token stream, but the model still only outputs autoregressive text (examples: Qwen3.5, Gemini 3, GPT-5.4, Claude Opus 4.6).
• Era 3b – Omni-modal unified I/O: the same fused trunk gains decoder heads (image/video VAE or DiT, audio codec) so it can generate images, video, and speech — not just text (examples: Emu3, Qwen3.5-Omni, LLaDA2.0-Uni). Generation-only video models like Sora 2 and Veo 3 reuse this decoder stack.
• Era 4 – World-action models (2026→): the trunk adds action as a first-class modality, plus persistent state and a policy head, closing the loop: perceiver, generator, and policy in one network (examples: Cosmos 3, Kairos, DreamX-World 1.0, SolarWM).

The README notes these eras coexist — the real design question is how much the model should generate and act.

What's actually inside the repo.

1. SoTA VLM table — a large table of frontier models (Claude, GPT, Gemini, Qwen, Kimi K3, SenseNova, GLM, etc.) with columns: release date, architecture, training data, parameter count, vision encoder, and backbone. Many cells are marked 'Undisclosed' — the README is explicit about not guessing at proprietary details.
2. Benchmarks & datasets — evaluation suites and datasets for VLMs, including pre-training/post-training corpora.
3. Post-training / alignment — sections on RL alignment, SFT, prompt engineering, and related GitHub repos.
4. Applications — embodied VLM agents, generative visual media, robotics/embodied AI (manipulation, navigation, HRIs, autonomous driving), and human-centered AI (web agents, accessibility, medical, social good).
5. Challenges — a dedicated section on current problems: hallucination, safety, fairness, alignment, efficient training, and scarcity of high-quality data.
6. Progressive research reports — 11 dated mini-surveys (latest: 2026-10-09) tracking major model releases, benchmarks, and post-training methods before they're folded into the main tables.

Companion tools.

• arXiv paper (2501.02189) — the formal survey that the repo accompanies.
• VLM Trends — a live web dashboard that tracks what changed today: new models, papers, benchmarks, and datasets pulled daily from arXiv, Hugging Face, GitHub, and Semantic Scholar, scored on a published rubric.

Who it's for / how to engage.

The README invites contributions and discussion. Papers marked with a star come from the maintainers; the README asks that if you find them useful you star the repo or cite the paper. For a non-expert: this is a good 'map of the territory' — it answers what models exist, what families they fall into, how they connect, and where the open problems are.