Visual Salamandra 7B Release

Visual Salamandra is a 7-billion parameter multimodal model that extends the Salamandra large language model (LLM) to process images and video. It is designed to provide a compact and efficient solution for vision-language alignment, specifically prioritizing the representation of European languages within a multimodal framework.

Technical Architecture and Vision-Language Alignment

Visual Salamandra utilizes a late-fusion architecture to bridge the gap between visual and textual modalities. The system is built upon the Salamandra Instructed 7B model and integrates the following components:

  • Vision Encoder: Google's SigLIP-So400m encoder, which processes images at a 384x384 resolution with 14 patches.
  • Projector: A custom-trained 2-layer multilayer perceptron (MLP) that maps image embeddings from the encoder into the latent space of the LLM.
  • Backbone: The Salamandra Instructed 7B foundational model.

Four-Phase Training Process

The model was developed through a structured four-phase training pipeline to ensure robust alignment and generalization:

  1. Projector Pre-training: The initial phase focuses exclusively on training the MLP projector to align image features with the LLM's latent space.
  2. High-Quality Vision Pretraining: The encoder, projector, and LLM undergo joint training using refined datasets, including re-captioned images and OCR data.
  3. Instruction Tuning: The model is trained to follow user instructions through grounded vision tasks, such as Optical Character Recognition (OCR) and Visual Question Answering (VQA).
  4. Full Multimodal Tuning: The final phase optimizes generalization for real-world scenarios by incorporating text-only examples, single images, multiple images, and video data.

To support this process, the Lab used 6.1 million instruction-tuning instances, including 842,000 text-only samples, drawing from datasets such as LLaVA Next, Cambrian, and AI2D to improve mathematical reasoning, document understanding, and visual grounding.

Multilingual Support and European Language Focus

Visual Salamandra integrates linguistic plurality directly into its multimodal instruction-tuned framework. By focusing on European languages, the model aims to close the resource gap for underrepresented languages in multimodal AI research, ensuring these languages benefit from the same level of instruction tuning and vision-task alignment as dominant languages.

Capabilities and Applications

Visual Salamandra is capable of comprehending and generating contextually accurate responses from diverse inputs. Key applications include:

  • Visual Question Answering (VQA): Providing context-aware responses to queries about images or videos.
  • Optical Character Recognition (OCR): Transcribing text from charts, scenes, and documents.
  • Document and Chart Understanding: Analyzing graphical content and complex visual documents with embedded text.
  • Mathematical Reasoning: Solving math problems that are visually grounded.
  • Instruction-based Image Interaction: Performing localization tasks and image captioning based on detailed instructions.
  • Video Analysis: The model's architecture supports future developments in event detection, multimodal storytelling, and video summarization.

Limitations and Ethical Considerations

Despite its capabilities, Visual Salamandra has known limitations:

  • Hallucinations: The model may generate plausible but incorrect answers when visual inputs are ambiguous.
  • OCR Complexity: Performance remains challenging when dealing with dense document layouts or highly complex OCR tasks.
  • Bias: While trained on filtered and licensed datasets, the model may still exhibit biases or inaccuracies.

Visual Salamandra is released under the Apache License, Version 2.0 for research and non-commercial use.

Sources