UCSC-VLAA/OpenVision

What it solves

OpenVision provides a family of open-source vision encoders designed for multimodal learning. It addresses the high cost and complexity of training advanced visual encoders by offering more efficient training objectives and scalable architectures that maintain or improve performance on benchmarks like OCR, TextVQA, and ChartQA.

How it works

The project has evolved through three versions:

• OpenVision (Original): Uses a combination of contrastive (CLIP-style) and generative (captioning) training objectives.
• OpenVision 2: Simplifies the process by removing the text encoder and contrastive loss, relying solely on a caption-only autoregressive generation objective. It utilizes dual-stage training (low to high resolution), synthetic captions from ReCap-DataComp-1B v2, and visual token masking to reduce memory footprint and training time.
• OpenVision 3: A unified visual encoder designed to handle both understanding and generation tasks.

Who it’s for

It is primarily for AI researchers and developers building multimodal models who need high-performance vision encoders and the infrastructure to train or fine-tune them, particularly those utilizing Google Cloud TPU infrastructure.

Highlights

• Significant Efficiency Gains: OpenVision 2 reduces training time by 1.5–2× and memory usage by ~1.8× compared to the original version.
• TPU Optimized: Specifically optimized for Google Cloud TPU training with support for model sharding.
• Scalable Architecture: Supports various ViT sizes, scaling up to 1B+ parameters.
• Comprehensive Model Zoo: Provides a wide range of pretrained weights for different resolutions and patch sizes.