askui/python-sdk
Enable AI to control your desktop, mobile and HMI devices
What it solves
Traditional UI automation is often fragile because it relies on brittle selectors (like XPath or CSS) that break whenever a layout changes or a label is updated. AskUI replaces this with a vision-based approach, allowing AI agents to interact with interfaces based on how they look and what they say rather than their underlying code structure.
How it works
The SDK combines vision-based automation with AI-powered agents. It allows users to find UI elements using text, images, or natural language descriptions. It supports two primary modes: single-step UI commands and agentic intent-based instructions where the AI determines the necessary steps to achieve a goal. The framework is model-agnostic, supporting providers like Anthropic Claude, Google Gemini, and custom models via the Model Context Protocol (MCP).
Who it’s for
It is designed for developers and testers building RPA (Robotic Process Automation) workflows, automating desktop applications across Windows, MacOS, and Linux, or testing mobile applications on Android and iOS.
Highlights
- Multi-platform support: Works across desktop (Windows, Linux, MacOS) and mobile (Android).
- Model flexibility: Compatible with various LLMs including Claude and Gemini, or custom-integrated models.
- Vision-first interaction: Locates elements via visual appearance and natural language instead of selectors.
- Extensible capabilities: Supports adding custom tools and capabilities to agents.
- Cost optimization: Includes caching to reduce expensive AI API calls.