Aya Vision: Advancing Multilingual Multimodality with 8B and 32B Models

Cohere For AI has released the Aya Vision family, consisting of 8B and 32B parameter open-weight vision-language models (VLMs) designed to bring high-performance multimodal capabilities to 23 different languages. These models outperform several larger competitors in multilingual vision-language tasks, providing a strong foundation for image understanding and text generation across a diverse linguistic range.

Performance Benchmarks

Aya Vision models demonstrate superior performance compared to both similarly sized and significantly larger models across two primary benchmarks: AyaVisionBench and mWildVision.

Aya Vision 32B

Aya Vision 32B outperforms models more than twice its size. In pair-wise comparisons, it achieves win rates between 50% and 64% on AyaVisionBench and 52% to 72% on mWildVision (averaged across 23 languages) against the following models:

  • Llama-3.2 90B Vision
  • Molmo 72B
  • Qwen2.5-VL 72B

Aya Vision 8B

Aya Vision 8B is positioned as a compact, efficient model that leads its parameter class. It outperforms models such as Qwen2.5-VL 7B, Pixtral 12B, Gemini Flash 1.5 8B, Llama-3.2 11B Vision, Molmo-D 7B, and Pangea 7B, with win rates up to 79% on AyaVisionBench and 81% on mWildBench.

Architecture and Training Pipeline

Aya Vision utilizes a modular architecture combining a vision encoder, a connector, and a multilingual text decoder to process images of arbitrary resolutions.

Vision Processing and Token Compression

The models use SigLIP2-patch14-384 as the vision encoder. To handle high-resolution images, the system dynamically resizes and splits images into multiple tiles. To maintain efficiency and reduce latency, a Pixel Shuffle downsampling method is employed to compress the number of image tokens by 4x before they are passed to the LLM decoder.

Text Decoder Initialization

  • Aya Vision 8B: Initialized from Cohere Command R7B and post-trained using the Aya Expanse recipe, which includes diverse multilingual data, model merging, and preference training.
  • Aya Vision 32B: Initialized from Aya Expanse 32B to leverage its state-of-the-art multilingual performance.

Two-Stage Training Process

  1. Vision-Language Alignment: Only the vision-language connector is trained while the vision encoder and language model weights remain frozen. This maps image encoder features to the language model embedding space.
  2. Supervised Fine-Tuning (SFT): Both the connector and the language model are trained on diverse multimodal tasks across 23 languages.

Multilingual Data Enhancement

To overcome the scarcity of real-world multimodal data for underrepresented languages, Cohere For AI implemented a synthetic data pipeline:

  1. Synthetic Annotation: High-quality English datasets were used to generate synthetic annotations.
  2. Translation and Rephrasing: Data was translated into 23 languages and then rephrased to remove translation artifacts and ensure linguistic fluency.

This approach significantly boosted performance; for the 8B model, using synthetic annotations and scaled multilingual data increased the win rate against Pangea 7B from 40.9% to 58.1% on AyaVisionBench (a 17.2% gain).

Multimodal Model Merging

To ensure the models maintain high-quality conversational and generative capabilities while gaining vision understanding, Cohere For AI merged the base language model with the fine-tuned vision-language model. This technique improved the multimodal win rate on AyaVisionBench against Pangea 7B by 11.9%, reaching a total win rate of 70%.

Additionally, multimodal model merging allows Aya Vision models to excel in text-only tasks as measured by the mArenaHard datasets compared to other leading VLMs.

AyaVisionBench: A New Multilingual Benchmark

Cohere For AI has released AyaVisionBench, an open-ended vision-language benchmark covering 23 languages and 9 task categories with 135 image-question pairs per language. The benchmark evaluates capabilities in:

  • Image captioning and OCR
  • Chart and figure understanding
  • Document understanding and text transcription
  • Logical and mathematical reasoning
  • Converting screenshots to code
  • Identifying differences between two images
  • General visual question answering

The dataset was created using images from the Cauldron held-out test set to prevent training leakage. Questions were synthetically generated and then validated by human annotators to ensure they were truly dependent on the visual context.

Real-World Application

Aya Vision is designed for global accessibility and is currently available on WhatsApp, allowing users across different languages to interact with the model's multimodal capabilities on a widely used communication platform.

Sources