mediar-ai/mcp-server-macos-use

What it solves

This project provides a way for AI models to interact directly with macOS applications. It bridges the gap between a Large Language Model (LLM) and the operating system's user interface by allowing the AI to see the UI structure and perform actions like clicking, typing, and opening apps.

How it works

It is implemented as a Model Context Protocol (MCP) server written in Swift. It uses the MacosUseSDK and macOS accessibility APIs to traverse the accessibility tree of an application (which tells the AI what elements are on screen) and simulate user inputs. It communicates with MCP-compatible clients, such as Claude Desktop, via standard input/output (stdio).

Who it’s for

Developers and AI enthusiasts who want to give their AI agents the ability to control their Mac, enabling the automation of tasks across different desktop applications.

Highlights

• UI Traversal: Can scan the accessibility tree of applications to understand the current state of the interface.
• Application Control: Supports opening applications, clicking specific coordinates, typing text, and pressing keyboard keys with modifiers.
• State Refresh: Includes a tool to refresh the UI traversal without performing any action to update the AI's context.
• Customizable Actions: Offers optional parameters to control animation, delays, and whether to only see visible elements.