yliust/Tactile

What it solves

Tactile addresses the fragility and inefficiency of "computer-use" agents that rely solely on screenshots and pixel-coordinate guessing. By prioritizing accessibility semantics over raw vision, it reduces the need for constant screenshot reasoning and coordinate retries, making agent interactions with software more reliable and precise.

How it works

Tactile implements a tiered operating strategy (an "operating ladder") that agents use to interact with a UI:

1. Accessibility Semantics: The agent first attempts to read the accessibility tree to identify element roles, names, states, and hierarchy to perform actions.
2. OCR-Grounded Coordinates: If semantics are missing, the agent uses system OCR to find visible text and its exact coordinates as a text-based fallback.
3. Native Visual Use: As a final resort, the agent falls back to its own visual reasoning and screenshot-based coordinate prediction.

This process extends to verification, where the agent confirms task success by checking accessibility states first, then OCR text changes, and finally visual screenshots.

Who it’s for

Developers building AI agents capable of operating desktop software and those interested in improving software accessibility for both humans and AI.

Highlights

• Accessibility-First Approach: Leverages existing OS accessibility APIs to give agents a "sense of touch" for software structure.
• Multi-layered Fallback: Provides a structured path from semantic data to OCR and finally to raw vision.
• MCP Support: Includes a dedicated macOS Model Context Protocol (MCP) server for faster and easier integration.
• Accessibility Advocacy: Encourages software developers to improve accessibility metadata, benefiting both AI agents and human users with disabilities.