jordanrendric/claude-video-vision

Give Claude the ability to watch and understand videos — Claude Code plugin with frame extraction and multimodal audio analysis

What it solves

It gives Claude Code the ability to "watch" and understand video content. Since LLMs cannot natively process raw video files, this project acts as a perception layer that converts video and audio into formats Claude can interpret: a series of images (frames) and timestamped text transcripts.

How it works

The plugin uses an MCP (Model Context Protocol) server to process videos. It employs ffmpeg to extract visual frames and utilizes one of three audio backends—Gemini API, local Whisper, or OpenAI API—to transcribe speech. For YouTube videos, it uses yt-dlp to download the content and prioritize existing subtitles before falling back to transcription. Claude then receives these images and transcripts to answer user questions, automatically adjusting the frame rate and resolution based on the specific request.

Who it’s for

Developers using Claude Code who need to analyze video files, screen recordings, or YouTube tutorials directly within their coding environment.

Highlights

  • Flexible Audio Backends: Supports cloud-based APIs (Gemini, OpenAI) or fully offline local processing via whisper.cpp.
  • YouTube Integration: Direct support for YouTube URLs with automatic metadata and caption retrieval.
  • Adaptive Extraction: Dynamically changes FPS and resolution based on the user's query (e.g., high resolution for specific timestamps, low FPS for long lectures).
  • Interactive Setup: Includes a /setup-video-vision wizard to handle configuration and dependency checks.

Related

  • Project
    taoufik123-collab/claude-watch

    A Claude skill that enables the AI to watch videos from URLs or local files by extracting key frames and transcripts for multimodal analysis.

  • Project
    jkawamoto/mcp-youtube-transcript

    An MCP server that allows AI agents to retrieve transcripts and metadata from YouTube videos, featuring pagination to handle long videos.

  • Project
    HUANGCHIHHUNGLeo/claude-real-video

    A local video processing pipeline that extracts meaningful keyframes and transcripts for LLMs, replacing fixed-interval sampling with scene-change detection and deduplication.

  • Project
    pmarreck/yt-transcriber

    A CLI tool that transcribes YouTube videos or local media files using Whisper, with optional AI-powered summarization and translation.

  • Dispatch
    claude-real-video: Enabling LLMs to Watch Videos via Scene-Aware Frame Extraction

    claude-real-video is an MIT-licensed local tool that allows any LLM to analyze videos by extracting scene-aware, deduplicated frames and transcripts instead of relying on fixed-interval sampling.