awizemann/harness

AI-driven user testing for iOS Simulator, macOS apps, and web apps. Write a goal in plain language; an LLM agent drives the UI and reports friction. macOS 14+, Swift 6.

What it solves

Harness replaces scripted UI tests with AI-driven user simulation. Instead of writing rigid test scripts that break when a UI element moves, developers can define a goal in plain language and a user persona, allowing an AI agent to navigate the app and identify real-world UX friction, dead ends, and unresponsive controls.

How it works

Harness uses an LLM agent that reads screenshots and interacts with the target application via a "Set-of-Mark" system (numbered overlays on interactive elements). The agent performs actions like clicking, typing, and scrolling based on the visual state of the app. It supports three target environments:

  • iOS Simulator: Driven via xcodebuild and WebDriverAgent.
  • macOS Apps: Driven via AX actions and CGEvent for contained input, capturing windows via CGWindowListCreateImage.
  • Web Apps: Driven via an embedded WKWebView using JS-synthesized events and snapshots.

The system supports both cloud-based LLMs (Anthropic, OpenAI, Google) and local inference via Ollama (e.g., Qwen3-VL, Llama 3.2 Vision).

Who it’s for

It is designed for macOS developers who want to perform automated user testing and UX auditing on iOS, macOS, or web applications without writing manual test scripts.

Highlights

  • Multi-platform support: Works across iOS simulators, native macOS apps, and web apps.
  • Set-of-Mark targeting: Agents click by ID rather than pixels, improving reliability.
  • Local & Cloud AI: Compatible with local vision models via Ollama and major cloud providers.
  • MCP Server integration: Allows external agents to drive the tool for autonomous runs or step-level UI sessions.
  • Session State Management: Supports injecting and exporting web session state (cookies/localStorage) to handle authenticated sessions.

Related

  • Project
    browser-use/macos-harness

    A minimal harness that gives LLM agents raw primitives to control macOS applications, browsers, and files without requiring app-specific tools.

  • Project
    vostride/agent-qa

    An agentic QA harness that allows users to write web and mobile tests in natural language and uses self-healing AI agents to adapt to UI changes.

  • Project
    fufankeji/deepseek-harness-studio

    A zero-code desktop application for DeepSeek Harness that provides a visual interface for managing AI agents, plugins, and local model inference.

  • Project
    QoderAI/better-harness

    An open-source framework for analyzing and improving the AI coding agent work loop by turning project and session evidence into prioritized workflow improvements.

  • Project
    AgentiLoop/Agent

    A native macOS app that provides an autonomous AI agent capable of controlling the desktop, managing codebases, and automating system tasks across 18 LLM providers.