EvolvingLMMs-Lab/LLaVA-OneVision-2

TL;DR

LLaVA‑OneVision‑2 is an open‑source 8‑billion‑parameter multimodal model that can understand images, long videos and 3‑D‑aware spatial data with a single vision encoder. The project releases the model weights, the novel “codec‑aligned” vision encoder, four training datasets, the full training code and logs, and scripts to reproduce the results on a single 8‑GPU node.

---

What it does

• Unified multimodal reasoning – a single LLM‑style model can take a still image, a short video, a long video, or a depth‑enhanced spatial scene and answer questions or follow instructions.
• Codec‑style video input – instead of sampling a few uniform frames, the encoder selects motion‑rich patches from HEVC‑style P‑frames, giving ~3× more temporal coverage for the same token budget.
• No per‑modality adapters – images, uniform‑frame video and codec‑aligned tokens all pass through the same OneVision‑Encoder with a shared (t, h, w) positional scheme.
• Instruction‑tuned – the released LLaVA‑OneVision‑2‑8B‑Instruct checkpoint has been fine‑tuned on a mixture of multimodal instruction data, including long‑form video captions (10‑15 min) and 3‑D spatial reasoning samples.

---

How it works (high‑level)

1. Codec‑aligned vision encoder – a transformer that ingests three kinds of visual tokens:
   • I‑frames (dense, full‑resolution patches)
   • P‑frames – only patches with significant motion or residual information are kept.
   • Uniform frames – supported for backward compatibility. This keeps the token count low (e.g., 54 tokens) while covering a much longer video span.
2. Single‑encoder architecture – the same encoder processes all visual modalities, eliminating the need for separate tokenizers or routing logic.
3. Four‑stage curriculum –
   • Bootstrap video ability with short captions.
   • Instruction‑tune on a large multimodal mix.
   • Extend to long‑video captions (10‑15 min).
   • Refine codec, spatial and tracking skills with dense long‑video sampling and 4 M spatial samples.
4. Training stack – built on Megatron‑LM, uses the released Docker image, Energon‑packed datasets (~340 GB), and provides full logs for reproducibility.

---

Who might use it

Use‑case	Why LLaVA‑OneVision‑2 helps
Research on video‑LLMs	Provides a publicly available 8B model with a novel codec‑aligned encoder and full training pipeline, enabling baseline comparisons and ablations.
Building multimodal assistants	One model handles images, videos and 3‑D spatial inputs without extra adapters, simplifying deployment.
Benchmarking multimodal benchmarks	The repo includes a lmms‑eval branch with exact evaluation wrappers, Docker env, and scripts to reproduce benchmark numbers.
Teaching multimodal ML	End‑to‑end code, data, and logs let students see a realistic large‑scale training workflow.

---

Getting started (quick‑start)

1. Clone the repo and navigate to the root.
2. Download the 340 GB bundle (lmms‑lab‑encoder/ov2_quickstart) – contains the Docker image, a Megatron checkpoint, the HF tokenizer, and the packed training dataset.
3. Run the provided script examples/llava_onevision2/quick_start_4b/quick_start.sh. It will:
   • Load the Docker image (llava_megatron:26.05).
   • Mount the bundle and an output directory.
   • Launch a detached container that starts the SFT run on 8 GPUs.
4. Monitor training via the TensorBoard links in the README or the logs that are uploaded to Hugging Face.
5. Inference – the model can be loaded with the standard vLLM or Hugging‑Face pipelines; a codec‑backend wrapper is provided in the vLLM docs link.

---

Limitations & open questions

• Hardware‑intensive – the full training bundle is ~340 GB and requires an 8‑GPU node with Docker and NVIDIA drivers.
• Model availability – only the 8B‑Instruct checkpoint is released; the 4B version is listed as “coming soon”.
• Codec‑specific preprocessing – to benefit from the codec‑aligned token budget you need to run the HEVC‑style patch selection step, which adds a preprocessing layer not present in standard video pipelines.
• Benchmarks limited to those reported – while the repo supplies evaluation scripts, performance on unseen tasks (e.g., real‑time video chat) is not yet demonstrated.

---

Citation

@article{llava_onevision2,
  title   = {LLaVA‑OneVision‑2: Fully Open Framework for Democratized Multimodal Training},
  author  = {Glint Lab and others},
  year    = {2026},
  url     = {https://arxiv.org/pdf/2605.25979},
}

---

All information above is taken directly from the repository’s README; no external assumptions have been added.