bradautomates/claude-video

Give Claude the ability to watch any video. /watch downloads, extracts frames, transcribes, hands it all to Claude.

What it solves

This project provides a way for AI agents (like Claude Code or Codex) to "watch" videos by converting them into a format the agents can understand: timestamped frames and transcripts. It allows agents to analyze video content, summarize visual events, and provide timestamped evidence from both local files and URLs.

How it works

The tool operates using two primary engines:

  1. Gemini Engine: Sends the video directly to Google's agentic video model for analysis and relays the timestamped answer.
  2. Local Engine: Processes the video on the user's machine using ffmpeg and yt-dlp to extract frames and transcripts. It handles transcription through several options: native captions, local inference via WhisperX (managed in a separate Python environment), or cloud APIs like Groq and OpenAI.

Who it’s for

Developers and users of local AI agents (Claude Code, Codex, and other Agent Skills hosts) who need their agents to process and analyze video evidence.

Highlights

  • Multi-Engine Support: Choose between high-performance cloud analysis (Gemini) or private local processing.
  • Flexible Transcription: Supports native captions, local WhisperX transcription, and cloud-based Whisper APIs (Groq/OpenAI).
  • Agent Integration: Native plugin support for Claude Code (Desktop, VS Code, Terminal) and Codex.
  • Customizable Detail: Offers different sampling modes (efficient, balanced, token-burner) to control the number of frames extracted based on the agent's token budget.
  • Smart Frame Selection: Uses scene change detection and RGB thumbnail deduplication to avoid redundant frames.

Related

  • Project
    gastownhall/gastown

    Gas Town is an open‑source Go‑based workspace manager that lets you run many AI coding agents (Claude, Copilot, Codex, Gemini, etc.) on multiple Git projects while persisting every step in a Git‑backed ledger. A central AI coordinator (the *Mayor*) creates *convoys* of work items (*beads*), hands them to worker agents (*Polecats*), and a set of watchdog services (Witness, Deacon, Dogs) keep the system healthy. Completed work is merged via a Bors‑style queue (*Refinery*), and an optional federation layer (*Wasteland*) lets different installations share tasks. Installable via Homebrew, `go install`, or Docker, it provides a CLI (`gt`) for creating rigs, crews, convoys, and for monitoring progress.

  • Project
    microsoft/SkillOpt

    SkillOpt is a Microsoft‑open‑source framework that optimises an LLM‑agent’s skill prompt as a trainable document using a deep‑learning‑style loop, delivering measurable performance gains without altering the model’s weights.

  • Project
    Huanshere/VideoLingo

    An AI-powered video translation and dubbing tool that automates speech recognition, subtitle translation, and voiceover generation in a single pipeline.

  • Project
    microsoft/unilm

    A comprehensive collection of foundation models and architectures from Microsoft focusing on large-scale self-supervised pre-training across text, vision, speech, and multimodal tasks.