xinzezhu/Open-AutoGLM-Android

What it solves

This project provides a way to automate smartphone operations using natural language commands. It eliminates the need for a computer or ADB (Android Debug Bridge) by leveraging Android's Accessibility Services to perform actions on the device directly.

How it works

The app uses Vision Language Models (VLMs) like AutoGLM or Alibaba's GUI-plus to analyze screen content and plan actions. It captures screenshots of the current screen, sends them to the AI model via API, and then executes the resulting commands (such as tapping, swiping, or typing) through the Android Accessibility Service API.

Who it’s for

Android users who want to automate repetitive tasks across different apps using simple text instructions, as well as developers interested in integrating VLMs with mobile device automation.

Highlights

• No ADB Required: Operates entirely on-device via Accessibility Services.
• Natural Language Control: Users can describe tasks in plain text (e.g., "Open WeChat and send a message to Zhang San").
• Multi-step Tasks: Supports conversational interaction and remembers context to complete complex workflows.
• VLM Integration: Compatible with Zhipu AI's AutoGLM and Alibaba's GUI-plus models.
• Comprehensive Action Set: Supports launching apps, tapping, typing, swiping, long-pressing, and double-tapping.