ZSeven-W/dsh-ios

DSH iOS Simulator – an AI‑agent‑driven iOS device view

What it is – A plugin (@zseven-w/dsh-ios) for the DeepSeek Harness (DSH) conversation platform that lets an LLM‑driven agent (or a user) control an iOS Simulator or a real iPhone from inside a chat window. The device’s screen is streamed as an MJPEG video into a persistent right‑hand panel, and a set of 22 JSON‑based agent tools let the model boot devices, build/run Xcode projects, tap UI elements, read logs, inspect processes, and even hot‑reload SwiftUI previews.

---

Core ideas

Idea	How it works
Live visual feedback	The simulator (or a USB‑connected iPhone) streams video through a signed, loop‑back‑only URL (/_dsh/dsh-ios/*). The browser never talks directly to the simulator port; all bytes are proxied by the DSH web‑server.
Agent‑friendly tools	Each capability is exposed as a plain‑JSON tool (e.g. ios_sim_boot, ios_sim_interact, ios_sim_ui_tree). The LLM can call a tool, get structured output, and decide the next step, enabling fully autonomous UI automation.
Real‑device support	When a physical iPhone is attached via USB, the plugin builds and launches WebDriverAgent on the phone, tunnels its control and MJPEG ports over the USB‑mux loopback, and re‑uses the same UI panel and tool set.
Safety gates	All taps are verified either by accessibility counters, OCR text, or explicit expect_* parameters. Real‑device taps are blocked unless the target is uniquely identified.
SwiftUI hot‑reload	ios_sim_preview creates a disposable host app, builds the SwiftUI preview as a dylib, and hot‑swaps it into the running simulator in ~2‑5 s, so the agent can see UI changes instantly.

---

Main feature groups

Group	Representative tools	What you can do
Device lifecycle	ios_sim_devices, ios_sim_boot, ios_sim_shutdown	List simulators, start/stop them, stream their screen.
App management	ios_sim_list_apps, ios_sim_launch_app, ios_sim_build_run	Discover installed apps, launch them, or build a project from source and run it on the simulator or a real phone.
Interaction	ios_sim_interact, ios_sim_tap_element, ios_sim_tap_text, ios_sim_find_text, ios_sim_tap_row	Tap, type, press hardware buttons, scroll, or gesture using either accessibility identifiers, OCR‑detected text, or list‑row coordinates.
UI inspection	ios_sim_ui_tree, ios_sim_ui_rows	Dump the accessibility hierarchy or a higher‑level list view, with parsed counters for list‑style apps.
Diagnostics	ios_sim_logs, ios_sim_processes, ios_sim_backtrace, ios_sim_leaks, ios_sim_app_info	Pull unified logs, enumerate processes, capture a backtrace, run Xcode’s leak detector, or query app bundle info.
Preview	ios_sim_preview	Hot‑reload SwiftUI previews without relaunching the app.
Real‑device bridge	ios_real_start_wda	Build and launch WebDriverAgent on a connected iPhone, exposing the same control & video streams as the simulator.

---

How to get started

1. Install the npm package

   npm i -g @zseven-w/dsh-ios   # or add to your project’s dependencies
   

2. Run a DSH session – start DeepSeek Harness (the host that runs the conversation UI). The plugin is automatically discovered if the DSH version is 0.1.5‑rc.1 or newer.
3. Boot a simulator (example JSON tool call):

   { "tool": "ios_sim_boot", "args": { "udid": "iPhone-14‑Simulator" } }
   

   The right‑hand panel will appear with a live MJPEG view of the booted device.
4. Interact – use ios_sim_interact to tap, type, or press Home, or call ios_sim_ui_tree to get an accessibility map and then ios_sim_tap_element to press a button by label.
5. Run your own code – point ios_sim_build_run at an Xcode project or Swift package, or start a SwiftUI preview with ios_sim_preview.
6. Real iPhone – plug the phone, run ios_real_start_wda with its UDID, then the same interaction tools work on the physical device.

---

Limitations & gotchas

• macOS‑only for simulator control – the tools that talk to simctl/serve‑sim require a Mac. On other OSes the tools register but return an explanatory error.
• USB‑device requires an unlocked phone and the user must grant “Identify‑before‑tap” permission; every tap on a real device is verified to avoid accidental actions.
• Loopback‑only transport – all traffic stays on 127.0.0.1; the plugin cannot be used across a network without additional tunnelling.
• Performance – deep accessibility snapshots on a real phone can take several seconds (up to ~30 s for an uncapped tree). List‑row extraction (ios_sim_ui_rows) is similarly heavy; use lighter tools (ios_sim_find_text, ios_sim_ui_tree with a shallow depth) first.
• Hot‑reload preview – only works for SwiftUI previews; it builds a temporary host app in ~/Library/Caches/dsh-ios and watches source files. One preview session at a time.
• Security – signed URLs expire after 10 minutes; the DSH server validates Host, Fetch‑Metadata, and HMAC on every request.

---

Who might use this

• LLM‑powered AI agents that need to test iOS UI, run automated UI‑regression checks, or demonstrate app behavior inside a chat.
• Developers who want a quick visual feedback loop (simulator or real device) while iterating on SwiftUI code via a conversational interface.
• QA engineers looking for a programmatic, JSON‑based way to drive iOS UI without writing XCUITest scripts.

---

Quick reference table (selected tools)

Tool	Primary purpose	Key arguments
ios_sim_devices	List available simulators & connected iPhones	–
ios_sim_boot	Start a simulator and begin streaming	udid (or device name)
ios_sim_interact	Generic tap/gesture/keyboard actions	action, x, y, text, …
ios_sim_ui_tree	Dump accessibility hierarchy	udid, max_depth, filter
ios_sim_tap_element	Tap an element by label/identifier	identifier / label, optional verification flags
ios_sim_find_text	OCR current screen	query, min_confidence
ios_sim_tap_row	Tap inside a list/feed row with counter verification	index, x, y, expect_count
ios_sim_preview	SwiftUI hot‑reload preview	packagePath, action (start/status/stop)
ios_real_start_wda	Launch WebDriverAgent on a USB iPhone	udid

---

Bottom line

DSH iOS Simulator turns an iOS simulator—or a real iPhone—into a live, image‑free visual endpoint that an AI agent can see and control through simple JSON calls. By keeping all media behind signed loop‑back URLs and providing safety‑checked interaction tools, it makes autonomous iOS UI automation feasible inside a conversational AI workflow.