mbzuai-oryx/Video-ChatGPT

[ACL 2024 🔥] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.

What it solves

Video-ChatGPT addresses the challenge of detailed video understanding. It allows users to have meaningful, natural language conversations about the content of videos, moving beyond simple classification or short-answer question answering.

How it works

The model combines a Large Language Model (LLM) with a pretrained visual encoder that has been adapted for spatiotemporal video representation. This allows the system to process video frames and temporal sequences to generate text responses based on the visual input.

Who it’s for

Researchers and developers working on multimodal AI, video analysis, and conversational AI who need a system capable of complex video reasoning, action recognition, and temporal understanding.

Highlights

  • VideoInstruct100K: A dataset of 100,000 high-quality video-instruction pairs used for training.
  • Quantitative Evaluation Framework: The first dedicated framework for benchmarking video-based conversational models.
  • Broad Capabilities: Capable of video reasoning, creative generation, spatial and temporal understanding, and action recognition.
  • State-of-the-Art Performance: Outperforms other conversational models like Video LLaMA and Video Chat on several zero-shot QA benchmarks.

Related

  • Project
    bcurts/agentchattr

    agentchattr is a cross‑platform local chat server that lets you and multiple AI coding agents converse in real‑time. By @‑mentioning an agent in the web UI, the server injects a prompt into the agent’s terminal (via MCP), the agent reads the chat and replies automatically, enabling hands‑free coordination, multi‑agent workflows, jobs, rules, sessions, and a rich UI with channels, pins, decision cards, scheduling, voice typing, and more.

  • Project
    vanloctech/youwee

    An open-source GUI for yt-dlp that provides video downloading from 1,800+ sites with integrated AI video summaries, natural language editing, and AI-powered subtitle generation.

  • Project
    kyutai-labs/unmute

    A system that enables text LLMs to listen and speak by wrapping them in low-latency speech-to-text and text-to-speech models.

  • Project
    ATH-MaaS/Ovis

    Ovis is a Multimodal Large Language Model (MLLM) architecture that structurally aligns visual and textual embeddings to enable high-resolution perception and reflective reasoning.