raulvidis/hermes-android

Android device control for hermes-agent — bridge app + Python toolset

What it solves

This project provides a way for AI agents to remotely control Android devices, effectively giving an agent "hands" to interact with mobile apps and hardware. It allows an agent to perform tasks like opening apps, sending messages, and capturing screenshots without requiring complex network configurations like VPNs or port forwarding.

How it works

The system consists of two main parts: an Android bridge app (written in Kotlin) and a Python toolset that runs on a server. The phone connects outward via a WebSocket to a relay server, which then communicates with the AI agent via HTTP. This architecture allows the phone to work behind any NAT. Pairing is handled via a simple 6-character code.

Who it’s for

Developers building AI agents (specifically those using hermes-agent) who want their agents to have the-real-world interaction capabilities of a mobile device, as well as users interested in automating Android Automotive OS (AAOS) car head units.

Highlights

  • Extensive Toolset: Includes 42 specialized tools for actions like tapping, swiping, typing, reading the accessibility tree, and managing notifications.
  • Hardware Integration: Supports recording audio from the microphone, accessing GPS location, and making phone calls or sending SMS.
  • Crossover Support: Compatible with Android Automotive OS (AAOS) for controlling car head units.
  • NAT Traversal: The phone initiates the connection to the server, eliminating the need for USB cables or complex network setups.

Related

  • Project
    rounak/PhoneAgent

    An experimental mobile automation project that enables AI agents to control iOS and Android devices through an in-app agent or a JSON-RPC bridge.

  • Project
    droidrun/mobile-harness

    A Python API and Markdown-based instruction set that enables AI agents to control Android and iOS devices locally or in the cloud.

  • Project
    ghost-in-the-droid/android-agent

    An open-source framework that gives AI agents a mobile body, enabling them to control Android and iOS devices via a unified tool surface and MCP server.

  • Project
    minitap-ai/mobile-use

    An open-source AI agent that automates Android and iOS devices using natural language to perform tasks and scrape data from apps.

  • Project
    anywhere-labs/Agents-Anywhere

    A remote control system that lets you manage and interact with AI coding agents (like Codex and Claude Code) running on your computer from a mobile device or web browser.