awizemann/harness

AI-driven user testing for iOS Simulator, macOS apps, and web apps. Write a goal in plain language; an LLM agent drives the UI and reports friction. macOS 14+, Swift 6.

What it solves

Harness automates user testing by simulating real human behavior rather than relying on rigid, scripted UI tests. It helps developers identify UX friction—such as dead ends, ambiguous labels, or unresponsive controls—by having an AI agent attempt to complete specific goals within an application.

How it works

A user provides a goal in plain language (e.g., "sign up and create a list") and a persona (e.g., "first-time user"). An LLM agent then interacts with the target application by reading screenshots and executing actions like clicking, typing, or scrolling. The system uses "Set-of-Mark" targeting, which overlays numbered badges on interactive elements so the agent can click by ID rather than imprecise pixel coordinates. It supports local inference via Ollama or cloud providers like Anthropic, OpenAI, and Google.

Who it’s for

  • iOS/macOS Developers: To test native apps on simulators or desktop environments.
  • Web Developers: To test web applications via an embedded browser view.
  • QA Engineers: To perform autonomous user-flow testing and receive detailed reports on success, failure, and friction points.

Highlights

  • Multi-platform support: Drives iOS Simulators, macOS apps, and web apps.
  • Real-user simulation: Focuses on UX friction and goal completion rather than scripted paths.
  • Set-of-Mark targeting: Uses numbered overlays on interactive elements to ensure precise agent interaction.
  • Flexible inference: Supports local execution via Ollama (using vision models like Qwen3-VL) or high-end cloud LLMs.
  • Rich artifacts: Generates a replayable sequence of actions, success/failure summaries, and timestamped friction reports.