jordanrendric/claude-video-vision

What it solves

This project provides a perception layer that allows Claude Code to "watch" and understand video content. It bridges the gap between a text-based LLM and video files or YouTube URLs by converting visual and auditory data into formats Claude can process.

How it works

The plugin acts as an MCP (Model Context Protocol) server that processes videos using the following pipeline:

1. Visual Extraction: It uses ffmpeg to extract frames as images (JPEG, PNG, or WebP) based on parameters like FPS and resolution.
2. Audio Processing: It transcribes audio using one of three backends: Gemini API, local Whisper (whisper.cpp or openai-whisper), or OpenAI API. For YouTube videos, it prioritizes manual subtitles, then auto-captions, then audio transcription.
3. Data Delivery: It sends the extracted frames and timestamped transcriptions directly to Claude, allowing the model to interpret the visual and auditory events.
4. Adaptive Sampling: Claude can dynamically adjust the extraction settings (e.g., increasing resolution for a specific timestamp) based on the user's query.

Who it’s for

Developers and users of Claude Code who need to analyze video files, summarize YouTube tutorials, or debug visual bug reports without manually transcribing or screenshotting the footage.

Highlights

• Multimodal Input: Combines direct image frames with timestamped audio text.
• YouTube Integration: Native support for downloading and processing YouTube videos via yt-dlp.
• Flexible Backends: Supports cloud-based APIs for speed or fully local Whisper processing for privacy and cost.
• Dynamic Control: Allows Claude to adjust FPS, time ranges, and resolution on the fly to focus on specific moments.