EvolvingLMMs-Lab/LLaVA-OneVision-2
Fully Open Framework for Democratized Multimodal Training
What it solves
LLaVA-OneVision-2 is a multimodal model designed to unify image, long-form video, and spatial understanding within a single architecture. It addresses the limitation where most open multimodal models are restricted to 2D single-image processing or struggle with long-form video due to token budget constraints.
How it works
The project utilizes a codec-aligned vision encoder (OneVision-Encoder) that borrows concepts from HEVC. Instead of uniform frame sampling, it keeps dense I-frames and only motion- and residual-rich patches from P-frames. This allows the model to cover three times the temporal range of standard pipelines within the same token budget. All modalities—images, uniform frames, and codec-aligned tokens—are processed by the same encoder using a shared position scheme, removing the need for task-specific adapters.
Who it’s for
It is intended for developers and researchers in the multimodal AI space who need a fully open, reproducible framework for training and deploying models capable of long-video reasoning, 3D-aware spatial reasoning, and high-resolution document/OCR analysis.
Highlights
- Unified Architecture: One model handles images, long videos, and spatial reasoning without modality-specific routing.
- Codec-Aligned Encoding: Efficiently processes long videos by selecting only high-information patches.
- Fully Open: Releases the entire pipeline, including encoder weights, training code, configs, and full training logs.
- Comprehensive Datasets: Includes specialized datasets for dense video captioning and 3D-aware spatial reasoning.