browser-use/macos-harness

What it solves

It provides a minimal, thin interface that allows an LLM agent to control a Mac computer. Instead of relying on pre-built, app-specific tools or rigid frameworks, it gives the agent raw primitives to interact with any application, the browser, and the local filesystem.

How it works

The project runs as a single persistent Python process that connects an LLM to macOS using several low-level mechanisms:

• Visuals and Input: It uses CGWindow for screenshots and CGEvent to send keyboard and coordinate input directly to a specific application's process ID (PID) without moving the user's physical cursor.
• Accessibility: It exposes raw Apple Accessibility (AX) and Apple Events for deeper interaction when visual data is insufficient.
• Browser Control: It integrates with Browser Harness to control a real, logged-in web browser via the Chrome DevTools Protocol (CDP).
• System Access: It provides direct access to the local filesystem via Path and the system shell via subprocess.

Who it’s for

Developers building AI agents that need to perform complex, multi-app tasks on macOS without being restricted by a set of predefined plugins or tools.

Highlights

• No Pre-defined Tools: The agent writes its own Python logic mid-task using raw primitives (see, key, type, click, ax, script).
• Non-Intrusive: Captures background windows and sends input without bringing apps to the foreground or moving the physical mouse pointer.
• ** uma animated, click-through pointer for visual feedback.
• Self-Bootstrapping: Designed to be installed and configured by the agent itself using a simple set of instructions.