awlevin/typesafe-computer-use

What it solves

It provides a cost-effective and fast way to automate a Mac computer using plain English goals. Unlike frontier models that send full screenshots to expensive LLMs for every step, this project reduces costs and latency by using a small, specialized classifier for decision-making and only calling a writing model when free-text input is required.

How it works

The system operates in a loop that processes the screen and determines the next action:

1. Perception: It captures the screen, uses Vision OCR to read text, and walks the macOS accessibility tree to find labeled controls. It employs a caching system that only re-reads changed tiles of the screen to save on OCR costs.
2. Decision: It sends the processed screen state to a TypeSafe decision model, which chooses an action (e.g., clicking an item, using a browser, or typing text) from a set of mutually exclusive options.
3. Action: It executes the deterministic action via macOS APIs (Quartz, AppleScript, AXPress).
4. Text Generation: A separate "writer" model is only invoked for specific tasks like composing text for a field or proposing a URL.
5. Final Answer: Once the goal is reached or the loop stops, a stronger model reads the final screen to provide the result.

Who it’s for

Developers and power users on macOS (14+) who want to build or use computer-automation agents that are significantly cheaper and faster than those relying solely on large multimodal models.

Highlights

• Extreme Cost Efficiency: Up to 300x cheaper than using frontier models for multi-step tasks.
• Low Latency: Decision steps take a fraction of the time compared to full-model planning.
• Hybrid Perception: Combines OCR with the accessibility tree to identify both text and non-text controls.
• Off-screen Control Support: Can interact with labeled controls that are present in the accessibility tree but not currently visible on screen.
• Deterministic State: Rebuilds reasoning (like date parsing) as deterministic code to avoid relying on expensive model reasoning.