Idefics2 8B Vision-Language Model Release – Architecture, Data, and Performance

TL;DR

Idefics2, an 8‑billion‑parameter open‑source vision‑language model released by Hugging Face, delivers state‑of‑the‑class performance on visual question answering and OCR tasks while being fully compatible with 🤗 Transformers for easy fine‑tuning.

Overview of Idefics2

Idefics2 is a general‑purpose multimodal model that accepts arbitrary sequences of text and images and generates textual responses. It can answer questions about images, describe visual content, create multi‑image stories, extract information from documents, and perform basic arithmetic. The model is released under the Apache 2.0 license, making it freely usable and redistributable.

Technical Highlights

Model Size and Architecture

  • Parameters: 8 B, a ten‑fold reduction compared with Idefics1 (80 B) while delivering superior performance.
  • Backbones: Built on Mistral‑7B‑v0.1 (language) and SigLIP‑SO400M‑Patch14‑384 (vision), both Apache‑2.0 licensed.
  • Vision Integration: Images are processed at native resolution (up to 980 × 980) using the NaViT strategy, avoiding fixed‑size resizing. An optional sub‑image splitting (4‑way) follows the SPHINX/LLaVA‑NeXT approach for very high‑resolution inputs.
  • Feature Fusion: Visual features are encoded, pooled via a learned Perceiver, projected with an MLP, and concatenated with text embeddings to form an interleaved sequence. This replaces the gated cross‑attention used in Idefics1 and simplifies multimodal interaction.

Training Data

Idefics2 was pretrained on a mixture of openly available datasets:

  • Web documents: Wikipedia and the OBELICS dataset (image‑text pairs from web documents).
  • Image‑caption pairs: Public Multimodal Dataset and LAION‑COCO.
  • OCR data: PDF‑A (English), IDL, and Rendered‑Text datasets.
  • Image‑to‑code data: WebSight.

For instruction tuning, the team compiled The Cauldron, an open collection of 50 manually curated multimodal instruction datasets formatted for multi‑turn conversations. Idefics2 was fine‑tuned on The Cauldron together with standard text‑only instruction data.

Performance Benchmarks

Idefics2 achieves top‑of‑class results among 8‑B‑scale models and competes with much larger proprietary systems. Key scores (higher is better) on popular multimodal benchmarks are:

Benchmark Idefics2 (no split) Idefics2 (with split)
MMMU (val/test) 43.5 / 37.9 43.0 / 37.7
MathVista (test‑mini) 51.6 51.4
TextVQA (val) 70.4 73.0
MMBench (test) 76.8 76.7
VQAv2 (test‑dev) 80.8 81.2
DocVQA (test) 67.3 74.0

These results place Idefics2 ahead of other open 7‑B/13‑B models such as DeepSeek‑VL, LLaVA‑NeXT‑Mistral‑7B, and LLaVA‑NeXT‑13B, and within striking distance of closed‑source giants like Gemini 1.5 Pro and Claude 3 Haiku.

New Capabilities Over Idefics1

  • Native‑resolution vision: NaViT‑style handling of images up to 980 × 980 preserves detail and aspect ratio.
  • Enhanced OCR: Integrated OCR‑focused training data improves transcription of text in images, charts, and documents.
  • Simplified fusion: Replaced gated cross‑attention with Perceiver pooling and MLP projection, yielding a cleaner architecture and faster inference.
  • Sub‑image splitting: Optional 4‑way split enables processing of very large images without sacrificing performance.

Collectively, these upgrades deliver a significant performance jump despite the model being 10× smaller than its predecessor.

Getting Started

Idefics2 is hosted on the Hugging Face Hub and supported out‑of‑the‑box in the latest transformers release. The following Python snippet demonstrates basic usage:

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForVision2Seq, load_image

DEVICE = "cuda:0"

# Load example images
image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")
image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg")

processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b")
model = AutoModelForVision2Seq.from_pretrained("HuggingFaceM4/idefics2-8b").to(DEVICE)

messages = [
    {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What do we see in this image?"}]},
    {"role": "assistant", "content": [{"type": "text", "text": "In this image, we can see the Statue of Liberty in New York."}]},
    {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "And how about this image?"}]},
]

prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image1, image2], return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}

generated_ids = model.generate(**inputs, max_new_tokens=200)
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])

A Colab notebook for fine‑tuning Idefics2 on custom data is also provided by the authors.

Resources

  • Model cards: idefics2-8b, idefics2-8b-base, idefics2-8b-chatty
  • Datasets: The Cauldron, OBELICS, WebSight
  • Demo spaces: Interactive playgrounds on Hugging Face Spaces
  • Paper: arXiv pre‑print 2405.02246

License

Idefics2 weights are released under the Apache 2.0 license, consistent with the underlying Mistral‑7B and SigLIP backbones.

Implications for the Community

The release of a high‑performing, fully open‑source 8‑B vision‑language model lowers the barrier for research and product development in multimodal AI. By providing both the model and a curated instruction dataset, Hugging Face enables rapid prototyping, fine‑tuning, and benchmarking without the licensing constraints of larger proprietary systems. The architectural simplifications and native‑resolution handling also set a new baseline for efficient multimodal fusion in future open models.

Sources