google-research/android_world

What it solves

AndroidWorld provides a standardized environment for developing and testing autonomous agents capable of controlling a computer (specifically Android devices). It addresses the need for a reproducible benchmark to evaluate how well AI agents can navigate real-world apps and perform complex tasks on a mobile operating system.

How it works

The project runs on a live Android emulator. It includes a benchmark of 116 hand-crafted tasks across 20 different apps. To ensure the agents aren't just memorizing specific paths, the environment uses dynamic task instantiation, which randomly generates parameters to create millions of unique variations of these tasks. It also integrates the MiniWoB++ web benchmark, rendering web elements as native Android UI widgets to test an agent's ability to use universal mobile interface components.

Who it’s for

It is designed for AI researchers and developers building autonomous agents, specifically those focusing on computer control, mobile UI navigation, and general-purpose AI assistants.

Highlights

• Diverse Task Suite: 116 tasks across 20 real-world apps.
• Dynamic Variations: Millions of unique task permutations via random parameter generation.
• Reliable Evaluation: Uses durable reward signals for consistent benchmarking.
• Extensible Architecture: Supports adding new tasks and custom agent implementations via a base agent class.
• Broad Compatibility: Open environment allowing access to millions of Android apps and websites.
• Deployment Flexibility: Includes experimental Docker support for consistent setup across different environments.