MCG-NJU/VideoChat3

VideoChat3: a generalist video MLLM built for fine-grained motion, long-form reasoning, temporal grounding, and online proactive response.

What it solves

VideoChat3 addresses the challenge of creating a generalist video model capable of handling diverse tasks—such as fine-grained motion analysis, long-video reasoning, precise temporal grounding, and real-time proactive responses—without sacrificing efficiency or losing critical visual evidence.

How it works

It is a 4B parameter Multimodal Large Language Model (MLLM) that utilizes two key technical innovations to optimize video processing:

  1. I3D-ViT: This component provides 16x spatiotemporal compression to reduce redundant visual tokens while keeping essential evidence.
  2. Adaptive Frame Resolution: The model dynamically increases frame resolution only when closer visual inspection is required, allowing for evidence-aware streaming.

It was trained on three large-scale datasets: Academic2M, LV116K, and OL617K.

Who it’s for

This project is designed for AI researchers and developers working on video understanding, multimodal LLMs, and applications requiring efficient processing of both short-form motion and hour-long video content.

Highlights

  • Generalist Capabilities: A single model that handles motion, long-form reasoning, temporal grounding, and online proactive responses.
  • Token Efficiency: Uses I3D-ViT to compress visual data without losing spatiotemporal detail.
  • Adaptive Perception: Dynamically adjusts resolution based on the need for visual detail.
  • Fully Open: Publicly provides model weights, training code, and complete training datasets.

Related

  • Project
    mbzuai-oryx/Video-ChatGPT

    A video conversation model that combines LLMs with a spatiotemporal visual encoder to enable detailed, natural language discussions about video content.

  • Project
    MME-Benchmarks/Video-MME-v2

    A robust benchmark for evaluating video understanding in MLLMs, using a three-level progressive difficulty scale and grouped non-linear scoring to measure true temporal reasoning.

  • Project
    OpenGVLab/VideoChat-Flash

    VideoChat-Flash is a multimodal large language model designed for efficient long-context video understanding, capable of processing videos up to three hours long using hierarchical compression.

  • Project
    OpenGVLab/Ask-Anything

    A family of multimodal large language models designed for chat-centric video and image understanding, enabling natural language conversations about visual content.

  • Project
    facebookresearch/VLM3

    VLM³ is a framework that enables standard Vision Language Models to perform complex 3D vision tasks like depth estimation and camera pose estimation without needing specialized architectures or data augmentations.