shhivv/third-hand

What it solves

Third Hand is a macOS assistant that allows users to control various applications through a chat interface. It eliminates the need to manually navigate menus and buttons by translating natural language commands into direct app actions, such as playing music on Spotify or searching for specific content within an app.

How it works

The system uses a combination of tools to interact with the macOS environment:

• arc-cua: A driver that reads and controls apps in the background via macOS Accessibility APIs, allowing it to interact with hidden or minimized windows.
• Apple Vision: Performs local text recognition on the screen to handle cases where app controls are not accessible.
• ChatGPT: Acts as the planner, generating a sequence of simple steps (clicking, typing, scrolling) based on the request and the app's current state.
• Jev: A specialized tool used to resolve ambiguities when a control label doesn't match exactly what the planner suggests.
• Execution: The app executes these steps one by one and verifies the result of each action before proceeding.

Who it’s for

Users of Apple Silicon Macs (macOS 14+) who want an AI-driven assistant to automate tasks across different macOS applications using natural language.

Highlights

• Background Execution: Tasks can be run in the background (Shift+Enter), leaving the pointer and front app undisturbed.
• @mentions: Users can tag specific apps in chat threads to direct commands to them, which also launches the app if it is not already open.
• Threaded Conversations: Maintains history within threads, allowing for follow-up commands like "now play the second one."
• Control-Space Shortcut: Quickly opens a new thread with the current active app already tagged.
• Local Processing: Screen text recognition is handled locally via Apple Vision, ensuring screenshots are not uploaded.