IDEFICS: An Open Reproduction of State-of-the-art Visual Language Model

Hugging Face has released IDEFICS (Image-aware Decoder Enhanced à la Flamingo with Interleaved Cross-attentionS), an open-access visual language model. IDEFICS is an open reproduction of DeepMind's Flamingo, designed to provide the AI community with a transparent, open-access alternative to proprietary multimodal models.

Model Capabilities and Variants

IDEFICS is a multimodal model capable of accepting arbitrary sequences of interleaved images and text as input and generating coherent text as output. This allows the model to perform tasks such as answering questions about images, describing visual content, and creating stories grounded in multiple images.

The model is available in two primary sizes and two functional variants:

  • Parameter Sizes: 9 billion and 80 billion parameters.
  • Model Variants: A base version and an instructed version (e.g., idefics-80B-instruct and idefics-9B-instruct) specifically adapted for conversational use cases.

Technical Architecture and Foundation

IDEFICS is built using publicly available models and data. It leverages the following components:

  • Language Model: LLaMA v1.
  • Vision Encoder: OpenCLIP.

These two pre-trained models are connected via newly initialized parameters that were trained by Hugging Face. While the base models are frozen, these connecting parameters are released under an MIT license.

Training Data and the OBELICS Dataset

IDEFICS was trained on a mixture of open datasets, including Wikipedia, the Public Multimodal Dataset, and LAION. To enhance its capabilities, Hugging Face created and released OBELICS, a new dataset consisting of 141 million interleaved image-text documents scraped from the web, containing a total of 353 million images and 115 billion tokens.

Ethical Evaluation and Transparency

To ensure transparency and safety, Hugging Face followed an ethical charter focused on being self-critical, transparent, and fair. The release process included:

  • Red Teaming: Internal evaluations using adversarial prompting with images and text to identify and mitigate potential biases.
  • Documentation: The release of technical lessons and mistakes encountered during the building process to provide a learning resource for the community.
  • Tooling: The provision of interactive visualizations for the OBELICS dataset via Nomic AI.

Licensing and Access

Access to IDEFICS is subject to the licenses of its base components:

  • CLIP-ViT-H-14-laion2B-s32B-b79K: Released under an MIT license.
  • llama-65b: Released under a non-commercial research license (requiring a Meta application form).
  • Additional Weights: The newly trained connecting parameters are released under an MIT license.

IDEFICS is integrated into the transformers library, allowing users to load the model via IdeficsForVisionText2Text and AutoProcessor.

Sources