HUANGCHIHHUNGLeo/claude-real-video

Let Claude (or any LLM) actually watch a video — scene-aware, deduplicated frames + transcript, from a URL or local file. Runs locally, MIT.

What it solves

Many LLMs cannot natively "see" video files or only process them by reading transcripts or sampling frames at fixed intervals (e.g., 1 frame per second). This often leads to missing fast cuts or wasting tokens on static scenes. claude-real-video allows any LLM to effectively "watch" a video by extracting only the most meaningful visual changes and providing a timestamped transcript, all while keeping the processing local to the user's machine.

How it works

The tool processes videos locally using a pipeline of specialized utilities:

  1. Frame Extraction: Uses ffmpeg to pull frames based on scene-change detection rather than fixed intervals.
  2. Deduplication: Employs a sliding-window deduplication process to remove near-identical frames, ensuring only unique visual information is sent to the LLM.
  3. Transcription: Uses OpenAI's Whisper (or faster-whisper and mlx-whisper for optimized performance) to generate timestamped transcripts.
  4. Packaging: Outputs a clean folder containing the keyframes, a MANIFEST.txt file, and the transcript, which the user can then upload to an LLM like Claude, ChatGPT, or Gemini.
  5. Integration: It can be installed as a skill for coding agents (Claude Code, Cursor, etc.) or run as an MCP server, allowing agents to trigger the video analysis process directly.

Who it’s for

  • AI Power Users: Those who want to give LLMs visual context from videos without uploading raw files to cloud providers.
  • Researchers and Analysts: People needing to analyze long meetings, tutorials, or screen recordings where specific visual cues (like code or slides) are critical.
  • Developers: Those building AI agents that need the ability to "watch" and reason about video content.

Highlights

  • Local Processing: Video analysis, frame extraction, and transcription happen on your machine; only the final selected frames/text are shared with the LLM.
  • Smart Sampling: Uses scene-change detection and adaptive sampling to capture fast cuts and slow morphs while collapsing static slides into single frames.
  • ** uma Grid View**: Can tile frames into 3x3 contact sheets to help models follow motion and progression with fewer tokens.
  • Advanced Audio Features: Supports local speaker diarization to label different speakers in the transcript.
  • Memory System: Indexes all watched videos locally in a SQLite database, allowing users to search across their entire video library using crv-ask.
  • MCP Server Support: Integrates directly with MCP-compatible clients like Claude Desktop and Cursor.

Related

  • Dispatch
    claude-real-video: Enabling LLMs to Watch Videos via Scene-Aware Frame Extraction

    claude-real-video is an MIT-licensed local tool that allows any LLM to analyze videos by extracting scene-aware, deduplicated frames and transcripts instead of relying on fixed-interval sampling.

  • Project
    taoufik123-collab/claude-watch

    A Claude skill that enables the AI to watch videos from URLs or local files by extracting key frames and transcripts for multimodal analysis.

  • Project
    jordanrendric/claude-video-vision

    A Claude Code plugin that enables the AI to analyze videos and YouTube URLs by extracting frames and transcribing audio via multiple backends.

  • Project
    bradautomates/claude-video

    A plugin for AI agents that converts videos from URLs or local files into timestamped frames and transcripts for visual analysis.

  • Project
    WEIFENG2333/VideoCaptioner

    An all-in-one video subtitle tool that uses LLMs for speech recognition, subtitle optimization, and translation to automate the video captioning process.