kitlangton/scrawl

Scrawl – Capture‑and‑Explain for AI Agents

What it does

• Lets you record a short screen session by pressing a global shortcut (⌥⇧S). While recording you can talk, click, scroll and draw on‑screen.
• When you stop, Scrawl bundles:
  1. A contact‑sheet image of key frames with your ink overlays.
  2. Full‑resolution PNGs of each key frame.
  3. A time‑aligned transcript and interaction timeline (summary.md).
  4. A JSON manifest (scrawl.json) describing windows, cursor paths, auto‑cropped regions, and where the capture should be sent.
• The bundle can be opened in a native review window, sent to an OpenCode (Claude‑style) session, or exported as an MP4.

Why it matters for AI

• The bundle is a structured representation of a visual‑plus‑verbal task that an LLM‑agent can ingest. The agent reads the contact sheet and transcript, then can request deeper views (scrawl zoom …) or individual frames.
• Scrawl ships an agent skill (bunx skills add kitlangton/scrawl) that registers the scrawl command with Claude Code, OpenCode, Codex, etc., so agents can automatically fetch and explore the captured context.

Key features

Feature	What you get
One‑click recording	Global hot‑key starts/stops capture; optional hold‑to‑choose area, window or iPhone screen.
Audio + transcription	Uses Apple Speech (or Hex) to produce a synced transcript without needing microphone permission beyond the initial grant.
Live annotation	Draw red strokes, numbered pins, undo with ⌘Z – all baked into the final frames.
Auto‑crop	After a screen‑wide capture Scrawl analyses cursor movement, clicks, and ink to automatically crop each keyframe to the relevant window(s).
Agent integration	scrawl zoom, scrawl frames, scrawl send, scrawl export commands let an LLM retrieve focused frames, send the bundle to an OpenCode session, or create a shareable video.
Dev‑site routing	If the capture ends on a local dev site, Scrawl can auto‑detect the project’s OpenCode session (via a tiny SDK script or by inspecting the local git repo) and send the bundle there.
Cross‑platform	Native macOS app (Apple Silicon, macOS 26+), previewable Windows build (preview, no auto‑updates).
CLI utilities	All operations (preview, inspect, frames, export, autocrop, mic, devices, etc.) are available via cargo run -- … for power users and automation.
Self‑updating	macOS version updates via Sparkle; Windows version is a manual zip.

How you use it

1. Install the app (Homebrew cask on macOS or zip on Windows).
2. Add the agent skill (bunx skills add kitlangton/scrawl).
3. Press ⌥⇧S, demonstrate the problem (talk, click, draw), press again.
4. The bundle appears in ~/Downloads (or a custom folder).
5. In an LLM chat, paste the bundle or run scrawl send … to let the agent inspect it.

Typical workflow for an AI‑assistant

• User records a UI bug walkthrough.
• Agent receives the bundle, reads the transcript, looks at the contact‑sheet, and asks for a zoom on a specific moment (scrawl zoom demo.scrawl --at "error dialog").
• Agent analyses the zoomed frames, suggests a fix, and optionally exports a short video to share with the team.

Technical notes

• Permissions: screen‑recording, microphone, accessibility, and (optional) camera for iPhone mirroring.
• Auto‑crop algorithm runs in Rust (src/autocrop.rs) and can be tuned via a local JSON config or a live web UI (scrawl autocrop … --serve).
• All configuration lives in ~/.config/scrawl/config.json.

Who might find it useful

• Developers who want to give LLM agents concrete visual context for UI debugging, onboarding, or documentation.
• Teams using Claude‑style “code‑assistant” agents that support custom skills.
• Anyone needing a quick, searchable record of a screen‑plus‑voice walkthrough.

---

All details are taken directly from the project's README; no additional features are inferred.