bigarrow CLI: Visual Guidance for macOS AI Agents

TL;DR – What bigarrow does and why it matters

bigarrow is a single‑binary macOS command‑line tool that draws click‑through arrows, boxes and optional text on top of any window or screen coordinate, then disappears automatically. It requires no screen‑recording or accessibility permissions for drawing, works across multiple displays, Spaces and full‑screen apps, and returns precise exit codes so AI agents can reliably signal success or failure. By giving agents a way to point rather than click, bigarrow bridges the last human‑only step in many automated workflows.

Core functionality – three commands agents need

Conclusion: Agents only need three simple commands to point, track, and clean up arrows.

Command	Purpose	Example
`bigarrow point`	Draw a single arrow to a UI element (by label) or screen coordinate, optionally with a caption.	`bigarrow point --element "Allow" --app "System Settings" --text "Franz, click Allow"`
`bigarrow start` / `bigarrow stop`	Keep an arrow visible until the agent explicitly stops it or a timeout expires.	`bigarrow start --window "Safari:Inbox" --text "This window" && bigarrow stop`
`bigarrow point --at X,Y`	Target a raw coordinate when label‑based targeting is unavailable (e.g., non‑accessible web pages).	`bigarrow point --at 760,500 --text "Franz, click HERE"`

All commands accept --duration, --json output, and a full set of styling flags (shape, colour, size, etc.). Exit codes are deterministic: 0 success, 2 bad input, 3 target not found, 4 missing permission.

Why pointing, not clicking?

Conclusion: Pointing respects user agency and avoids security pitfalls while still guiding the human.

Permission dialogs – macOS prompts (e.g., “Allow”) can be highlighted without the agent taking the click.
Sensitive actions – 2FA, CAPTCHAs, payment confirmations, legal checkboxes remain under human control.
Window disambiguation – With many Chrome windows open, --window "Google Chrome:Pull request" selects the correct tab, then points.
Audio cue – --say reads the caption aloud, pulling the user back to the desk.
Remote assistance – Instead of describing “the other gear icon”, an agent can point directly.
Documentation – --png renders the arrow into a PNG for static docs or screencasts.
Debugging – --dry-run --json reports the computed target without drawing, useful for coordinate verification.

The tool deliberately does not click, type, or capture screenshots; it is a pure visual cue.

Installation and integration

Conclusion: bigarrow can be installed via Homebrew or built from source, and includes a ready‑made Claude Code / Codex skill.

brew install franzenzenhofer/tap/bigarrow
bigarrow install-skill   # registers the skill for Claude Code (~/.claude/skills) and Codex (~/.agents/skills)

From source: swift build -c release (requires Xcode 16+, macOS 14+). The resulting binary lives at .build/release/bigarrow.

Styling options – making the arrow noticeable

Conclusion: A rich set of visual parameters ensures the arrow stands out on any background.

Shapes: bend, straight, zigzag (urgent), spiral (loops once).
Styles: arrow (default), ring, box (border‑only).
Sizes: S, M, L.
Corners: round or sharp.
Colours: Named palette (red, orange, …) or custom #RRGGBB.
Borders: shadow (default), white-black, black.
Close button: --close-button adds an X that removes the arrow on click.
Dynamic behaviour: --follow tracks a moving window, --until-click ends on a click, --say triggers speech synthesis.

The project includes a gallery script that renders every combination off‑screen and zooms into the joint where the shaft meets the sign, ensuring a seamless visual appearance.

Permissions model – why no screen‑recording?

Conclusion: Drawing uses a transparent overlay window that macOS allows without extra entitlements; only target‑discovery features need Accessibility.

Drawing – No Screen Recording or Accessibility permission required.
Target lookup – --element, elements, --until-click, and --window rely on Accessibility, which macOS grants to the shell process (Terminal, iTerm, VS Code, Claude) rather than to bigarrow itself.
bigarrow doctor – Reports which app holds the necessary permissions and exits with code 4 if they are missing.

This design prevents the tool from silently capturing screen content while still enabling precise UI element identification.

Robustness across macOS environments

Conclusion: The arrow works on multiple displays, full‑screen apps, Stage Manager, and Spaces, and gracefully handles display disconnection.

Supports negative coordinates for displays left/above the primary screen.
Unplugging a display while an arrow is present causes the arrow to disappear politely.
Verified on macOS 15, 26, and 27 via CI.
CPU impact is minimal – ~1.4 % on a CI runner, with Core Animation handling rendering.

Real‑world use cases highlighted by the author

Conclusion: bigarrow addresses concrete pain points in AI‑augmented workflows.

Permission prompts – “Click Allow” dialogs that agents can’t press.
2FA / CAPTCHAs – Agents point, humans confirm.
Window selection – Disambiguate among dozens of Chrome windows.
Guided onboarding – Step‑by‑step tours with start/stop semantics.
Remote help – Point to the correct UI element for a colleague.
Documentation – Generate annotated screenshots automatically.
Coordinate debugging – Visual confirmation of Accessibility or Peekaboo coordinates.

Community feedback – synthesis of top comments

Conclusion: Users appreciate the concept but raise concerns about safety, usability, and broader applicability.

"I tried to read the ‘Does it need Screen Recording or Accessibility?’ part, but it’s slopped … what’s stopping it from drawing a box that hides the ‘decline’ button?" – @hn8726 (clarifies that drawing never blocks UI elements; only the sign itself is clickable.)

"An arrow solves ‘where do I click’ but not ‘should I click’. It may reduce the human’s decision‑making role." – @tilemarch (highlights the ethical boundary; the tool intentionally stops short of automating the click.)

"This would be great for helping my mom download and print a document over the phone." – @arshxyz (suggests a non‑agent use case for assisted remote help.)

"I would pay good money for an iPad version that launches the correct app and guides the user through a bank transfer." – @lbreakjai (indicates demand for a cross‑platform UI‑assistant.)

"The README is geared toward technical people; a simpler UI could broaden adoption." – @usrbinbash (points out documentation accessibility.)

Overall, the community sees value in visual guidance but stresses the need for clear boundaries to preserve user consent.

Test coverage and reliability

Conclusion: Extensive automated testing validates geometry, rendering, and interaction behaviour.

87 unit tests covering geometry, joint smoothness, golden‑image comparison, and integration with Accessibility and Peekaboo fixtures.
17 visual behaviour checks in CI on real macOS window server: click‑through, focus preservation, multi‑display handling, CPU usage, etc.
Agent skill test – A fresh Claude Code agent successfully located a Chrome “Reload” button using only the skill, exposing a bug that was added as a regression test.

Limitations and open questions

Conclusion: bigarrow is not a general screen‑annotation tool and cannot replace UI automation.

It cannot click, type, or capture screenshots.
Web page element targeting works in Electron apps and Chrome only when launched with --force-renderer-accessibility or VoiceOver enabled.
The tool relies on Accessibility for label‑based targeting; without it, coordinates must be supplied manually.
No iOS/iPadOS support yet – community interest is evident from comments.

Future directions hinted by the repo

Conclusion: The author plans to expand documentation, refine the skill format, and explore additional visual cues.

Detailed PLAN.md, DECISIONS/, and RESEARCH/ directories outline architecture and risk assessment.
Prior art includes Peekaboo visualizer and Nameplate; bigarrow fills the gap of arrow‑plus‑label overlays.
Potential extensions could include richer animations, multi‑agent coordination, or cross‑platform ports.

Bottom line

bigarrow provides a lightweight, permission‑free way for AI agents to point users at UI elements on macOS, turning ambiguous “click this button” prompts into clear visual cues while preserving user agency and system security. Its minimal footprint, robust testing, and flexible styling make it a practical addition to any agent‑driven workflow.