cambrian-mllm/cambrian-s

Cambrian-S: Towards Spatial Supersensing in Video

What it solves

Cambrian-S is designed to improve spatial reasoning in video understanding. While many multimodal large language models (MLLMs) struggle with spatial awareness, this project provides models and datasets specifically tuned for "spatial supersensing" in video, allowing AI to better understand the physical layout and spatial relationships of objects within a video sequence.

How it works

The project employs a four-stage training process:

  1. Vision-Language Alignment: Aligning visual and textual data.
  2. General Image Instruction Tuning: Training on images.
  3. General Video Instruction Tuning: Training on video data.
  4. Spatial Video Tuning: A final specialized stage for spatial understanding.

For the LFP (Latent Frame Prediction) version of the model, the fourth stage is modified to include a latent frame prediction objective to further enhance spatial capabilities.

Who it’s for

Researchers and developers working on video-based AI, multimodal LLMs, and applications requiring high-precision spatial reasoning in video (such as robotics or autonomous systems).

Highlights

  • Specialized Models: Offers a range of model sizes from 0.5B to 7B parameters, based on Qwen2.5 and SigLIP2.
  • VSI-590K Dataset: A curated video instruction-tuning dataset specifically focused on spatial understanding.
  • VSI-SUPER Benchmark: A dedicated evaluation suite to measure spatial supersensing capabilities.
  • Cambrian-S-3M: A collection of 3 million open-sourced video instruction tuning data points.
  • TPU Support: Training code is provided with support for TPU using TorchXLA.