HUANGCHIHHUNGLeo/claude-real-video

Let Claude (or any LLM) actually watch a video — scene-aware, deduplicated frames + transcript, from a URL or local file. Runs locally, MIT.

What it solves

Most LLMs cannot "see" videos natively or they rely on fixed-interval sampling (e.g., one frame per second), which often misses fast cuts or wastes tokens on static scenes. claude-real-video provides a local processing pipeline that extracts only the most meaningful frames based on scene changes and deduplication, reducing token costs while improving the AI's visual understanding of the video content.

How it works

The tool processes videos locally using the following pipeline:

  1. Fetch & Extract: Uses yt-dlp for URLs and ffmpeg to grab frames at every scene change and a minimum density floor.
  2. Deduplication: Employs a sliding-window deduplication system with three channels (global pixel difference, settled-local detection for small UI changes, and an action channel) to ensure only unique shots are sent to the LLM.
  3. Transcription: Prioritizes existing subtitles; otherwise, it uses OpenAI Whisper (or faster-whisper) for local speech-to-text.
  4. Packaging: Outputs a folder containing the keyframes, a timestamped JSON file, and a transcript, which can then be uploaded to an LLM. It also supports tiling frames into contact sheets (--grid) to help models follow motion.
  5. Integration: Can be run as a CLI, a local web UI (crv-web), or an MCP server for direct integration with agents like Claude Desktop and Cursor.

Who it’s for

  • AI Power Users: Those who want to analyze videos with LLMs without uploading raw files to cloud services.
  • Developers: Users of coding agents (Claude Code, Cursor) who want their agents to be able to "watch" videos via a skill/plugin.
  • Researchers: People needing a high-quality, deduplicated keyframe extractor for video analysis.

Highlights

  • Local Processing: All frame extraction and transcription happen on the user's machine; only the final selected data is shared with the LLM.
  • Intelligent Sampling: Uses scene-change detection instead of fixed intervals to capture fast cuts and collapse static slides.
  • Advanced Dedup: Prevents redundant frames from being sent even after cutaways (A-B-A cuts).
  • Agent Integration: Ships as an MCP server and a plugin for various agent hosts.
  • Flexible Input: Supports YouTube, Instagram, TikTok, and local files.
  • Memory System: Locally indexes all watched videos via SQLite, allowing users to search across their entire video library (crv-ask).

Related

  • Dispatch
    claude-real-video: Enabling LLMs to Watch Videos via Scene-Aware Frame Extraction

    claude-real-video is an MIT-licensed local tool that allows any LLM to analyze videos by extracting scene-aware, deduplicated frames and transcripts instead of relying on fixed-interval sampling.

  • Project
    taoufik123-collab/claude-watch

    A Claude skill that enables the AI to watch videos from URLs or local files by extracting key frames and transcripts for multimodal analysis.

  • Project
    jordanrendric/claude-video-vision

    A Claude Code plugin that enables video understanding by extracting frames and transcribing audio via local or cloud backends.

  • Project
    WEIFENG2333/VideoCaptioner

    An all-in-one video subtitle tool that uses LLMs for speech recognition, subtitle optimization, and translation to automate the video captioning process.

  • Project
    JayWebtech/autoshorts

    A local-first desktop app that uses AI to identify viral moments in long-form video or audio and automatically crops them into vertical short-form clips.