OpenGVLab/Ask-Anything

[CVPR2024 Highlight][VideoChatGPT] ChatGPT with video understanding! And many more supported LMs such as miniGPT4, StableLM, and MOSS.

What it solves

Ask-Anything (the VideoChat family) provides a way to have natural language conversations about the contents of videos and images. It solves the challenge of complex video understanding, allowing users to ask questions and receive detailed answers based on the video's visual content.

How it works

The project implements several versions of VideoChat, utilizing different Large Language Model (LLM) backends. It offers an end-to-end chatbot approach where the model is instruction-tuned for video and image chatting. It also supports explicit communication with external LLMs like ChatGPT, StableLM, and MOSS, as well as implicit communication with Vicuna via MiniGPT-4.

Who it’s for

Researchers and developers working on multimodal AI, specifically those interested in video understanding, video-language models, and the-end end conversational AI for visual media.

Highlights

  • Multiple Model Variants: Includes VideoChat, VideoChat2, and VideoChat3, with specialized versions like VideoChat2_HD for high-resolution data and VideoChat2_phi3 for speed.
  • Instruction Tuning: Uses a large-scale dataset of 2 million diverse instruction data for effective tuning.
  • Broad LLM Support: Compatible with various backends including ChatGPT, StableLM, MOSS, and Vicuna.
  • Comprehensive Benchmarking: Includes MVBench, a comprehensive benchmark for video understanding.
  • Long Video Support: Incorporates LangChain and Whisper to handle videos longer than one minute.

Related

  • Project
    MCG-NJU/VideoChat3

    VideoChat3 is a 4B generalist video MLLM designed for efficient video understanding, covering everything from subtle motion and long-form reasoning to live stream proactive responses.

  • Project
    bytedance/SALMONN

    A suite of advanced multi-modal LLMs that provide large language models with generic hearing abilities and integrated audio-visual perception.

  • Project
    bytedance/vidi

    A family of Large Multimodal Models (LMMs) for video understanding and creation, enabling tasks like temporal retrieval, spatio-temporal grounding, and automated video editing.

  • Project
    OpenMOSS/MOSS-VL

    MOSS‑VL is an open‑weight 11 B‑parameter family of video‑language models that can understand and converse about live video streams in real time. It uses a cross‑attention architecture with timestamped frames and a 3‑D rotary embedding (XRoPE) to achieve low‑latency, interruptible dialogue, proactive silence, and dynamic correction. Three variants are released (Realtime, Instruct, Base), with quantized checkpoints for single‑GPU use, integration with FlashAttention‑3, SGLang, ms‑swift, and LoRA fine‑tuning. The project includes benchmarks showing state‑of‑the‑art streaming performance and provides scripts for both realtime and offline inference.

  • Project
    MME-Benchmarks/Video-MME-v2

    A robust benchmark for evaluating video understanding in MLLMs, using a three-level progressive difficulty scale and grouped non-linear scoring to measure true temporal reasoning.