claude-real-video: Enabling LLMs to Watch Videos via Scene-Aware Frame Extraction

Overview

claude-real-video is a local processing tool designed to enable Large Language Models (LLMs) like Claude, ChatGPT, and Gemini to "watch" videos by converting them into a format the models can actually process. Instead of relying on simple transcripts or fixed-interval frame sampling, the tool extracts only the most meaningful frames based on scene changes and deduplicates them to optimize context window usage.

Why Scene-Aware Extraction Matters

Most AI video analysis pipelines, including Gemini's native capabilities, typically sample frames at a fixed interval (e.g., 1 frame per second). This approach creates two primary inefficiencies:

1. Over-sampling: Static content, such as a 10-minute screencast, results in hundreds of near-identical frames that waste tokens.
2. Under-sampling: Fast-cut reels may have critical visual changes occur between samples, causing the model to miss key information.

claude-real-video solves this by using scene-change detection combined with a density floor (ensuring at least one frame is captured every N seconds). It further employs a sliding-window deduplication process using real pixel difference (downscaled RGB) rather than perceptual hashes. This ensures that if a video cuts from Shot A to Shot B and back to Shot A, the repeated shot is not sent to the model a second time.

Technical Workflow

The tool processes video through a six-step pipeline:

1. Fetch: Downloads content via yt-dlp for URLs or accesses local files.
2. Extract: Uses ffmpeg select to grab every scene change and maintain the density floor.
3. Dedup: Compares frames against a sliding window of previously kept frames to remove redundancies.
4. Text: Extracts existing subtitles (.srt/.vtt) if available; otherwise, it uses OpenAI's Whisper for audio transcription.
5. Audio (Optional): Can save the full original soundtrack as an .m4a file for models capable of native audio processing (e.g., GPT-4o).
6. Manifest: Generates a MANIFEST.txt file that summarizes the extracted data for the LLM.

Installation and Usage

System Requirements

ffmpeg and ffprobe are required for frame extraction and audio processing.

• macOS: brew install ffmpeg
• Linux: sudo apt install ffmpeg
• Windows: winget install Gyan.FFmpeg

Setup

pip install claude-real-video              # Core functionality
pip install "claude-real-video[whisper]"   # Adds audio transcription

Common Commands

• Process a URL: crv "https://www.youtube.com/watch?v=..."
• Local file with English transcript: crv lecture.mp4 -o out --lang en
• Frames only: crv clip.mp4 --no-transcribe
• Login-gated content: crv "https://..." --cookies cookies.txt

Key Configuration Options

Flag	Default	Description
--scene	0.30	Sensitivity for scene-change detection (lower = more frames).
--fps-floor	1.0	Minimum frame capture rate (one frame every N seconds).
--max-frames	150	Hard limit on the total number of frames extracted.
--dedup-threshold	8	Percentage of pixel change required to count a frame as new.
--dedup-window	4	Number of previous frames to compare against for deduplication.
--keep-audio	off	Saves the full soundtrack for audio-capable models.

Community Insights and Trade-offs

While claude-real-video provides a structured way to feed visual data to LLMs, community discussions highlight several considerations:

• Token Cost: Some users note that sending many frames to models like Claude can be expensive in terms of token consumption compared to using native VLMs (Vision Language Models) or specialized libraries like vlm-run.
• Privacy: While the processing happens locally, users should remember that once the extracted frames are uploaded to a cloud-based LLM (like Claude or ChatGPT), that data is then transmitted to the provider.
• Visual Limitations: Some users have found that LLMs still struggle with high-precision tasks, such as mapping specific sprites from a video to a sprite sheet, or inferring complex motion design and timing without explicit textual descriptions.
• Orchestration: Some users report that ChatGPT's internal agents may already perform similar frame extraction and interpolation tasks automatically, though claude-real-video provides explicit local control over this process.