bytedance/vidi

The official repo for "Vidi: Large Multimodal Models for Video Understanding and Editing"

What it solves

Vidi addresses the challenge of comprehensive video understanding and creation. It provides a unified framework for tasks ranging from precise temporal retrieval (finding specific time ranges in a video based on a text query) and spatio-temporal grounding (locating objects within specific timeframes) to high-level video analysis and automated editing.

How it works

Vidi is a family of Large Multimodal Models (LMMs) designed for Video Understanding and Editing (VUE). It processes video inputs to perform several distinct operations:

  • Understanding: It can identify specific clips via retrieval, draw bounding boxes around objects (grounding), generate chapter titles, identify highlights, and answer questions about video content (VQA/Thinking).
  • Creation: Through its "Vidi-Edit" capability, the model can take multiple uploaded videos and automatically generate an edited video complete with a storyline, music, and effects.
  • Evaluation: The project includes several benchmarks (VUE-PLOT, VUE-STG, VUE-TR-V2) to measure performance in plot understanding, spatio-temporal grounding, and temporal retrieval.

Who it’s for

  • AI Researchers: Those working on multimodal models, video analysis, and automated video editing.
  • Developers: Users looking to integrate advanced video retrieval and grounding capabilities into applications.
  • Content Creators: People needing automated tools for video summarization, highlighting, and editing.

Highlights

  • Multi-task Capability: Supports grounding, retrieval, chaptering, highlighting, and VQA.
  • Automated Editing: Capable of generating a full edited video from raw clips including music and effects.
  • Comprehensive Benchmarks: Provides specialized evaluation sets for spatio-temporal grounding and plot understanding.
  • Model Weights: Offers access to Vidi-7B and Vidi1.5-9B models with finetuning code.

Related

  • Project
    MME-Benchmarks/Video-MME-v2

    A robust benchmark for evaluating video understanding in MLLMs, using a three-level progressive difficulty scale and grouped non-linear scoring to measure true temporal reasoning.

  • Project
    OpenGVLab/Ask-Anything

    A family of multimodal large language models designed for chat-centric video and image understanding, enabling natural language conversations about visual content.

  • Project
    OpenGVLab/InternVideo

    A series of video foundation models and large-scale datasets designed for multimodal video understanding, long-context modeling, and contextual reasoning.

  • Project
    PaddlePaddle/ERNIE

    A family of large-scale multimodal MoE models and development toolkits providing state-of-the-art performance in text and vision-language understanding.

  • Dispatch
    Hugging Face Vision Language Models 2025 Update

    Hugging Face provides a comprehensive overview of the 2024-2025 evolution of Vision Language Models (VLMs), highlighting trends in any-to-any architectures, reasoning models, and the rise of Vision-Language-Action (VLA) models for robotics.