EvolvingLMMs-Lab/LLaVA-OneVision-2

What it solves

LLaVA-OneVision-2 is a multimodal model designed to unify image, long-form video, and spatial understanding within a single architecture. It addresses the limitation where most open multimodal models are restricted to 2D single-image tasks or struggle with long-form video due to context window constraints and inefficient token usage.

How it works

The project utilizes a "codec-aligned" vision encoder (OneVision-Encoder) that mimics HEVC video compression. Instead of sampling frames uniformly, it keeps dense I-frames and only selects motion- and residual-rich patches from P-frames. This allows the model to cover three times the temporal range of standard sampling within the same token budget.

Crucially, the model uses a single unified encoder for all modalities—images, uniform frames, and codec-aligned tokens—using a shared position scheme, eliminating the need for task-specific adapters or separate tokenizers. Training follows a four-stage curriculum: bootstrapping video ability, instruction tuning, extending to long videos, and refining spatial and tracking skills.

Who it’s for

This project is for AI researchers and developers who need a high-performance, fully open multimodal model capable of reasoning over long videos, understanding 3D-aware spatial layouts, and processing high-resolution documents and OCR.

Highlights

• Unified Architecture: A single 8B model that handles images, long videos, and spatial reasoning without task-specific adapters.
• Codec-Aligned Encoding: Efficiently processes long-form video by focusing on motion-rich patches, increasing temporal coverage.
• Fully Open: Releases the entire pipeline, including encoder weights, training code, configurations, and full training logs.
• Comprehensive Datasets: Includes specialized datasets for dense video captioning and 3D-aware spatial reasoning.