HITsz-TMG/KnowAct

What it solves

KnowAct-GUIClaw is a personal GUI assistant designed to automate tasks across multiple platforms, including desktop (macOS, Linux, Windows), Android, iOS, and HarmonyOS. It addresses the challenge of long-horizon GUI automation by providing a framework that allows agents to learn and evolve their capabilities through memory and skill acquisition.

How it works

The system operates on a "Know–Route–Act–Reflect" loop. It uses multimodal models to perceive the screen and execute actions. The framework includes a self-evolving mechanism where it can infer and validate Android shortcuts, extract reusable skills from successful trajectories, and maintain a GUI memory bank and a permission policy to guide future actions.

Who it–s for

It is intended for developers and researchers building GUI agents who need a cross-platform automation tool that can integrate with various vision-language models and supports both a full agent host (via nanobot) or a standalone CLI.

Highlights

• Cross-Platform Support: Works across Android, iOS, HarmonyOS, and major desktop operating systems.
• Self-Evolving Capabilities: Can automatically discover, validate, and promote shortcuts to a shared skill file.
• High Performance: Achieves state-of-the-art results on the MobileWorld benchmark using open-source models.
• Flexible Deployment: Available as a complete agent host with a WebUI or as a lightweight standalone CLI tool.
• Customizable Profiles: Includes pre-configured agent profiles to match the action formats of various model families (e.g., Qwen, Seed, Venus).