google-deepmind/android_env

What it solves

AndroidEnv provides a standardized way to turn an Android device (or emulator) into a Reinforcement Learning (RL) environment. It allows researchers to train agents to navigate the Android OS and any installed application, enabling the study of agents that can operate in real-world digital contexts using only visual and touch inputs.

How it works

The library wraps a simulated Android device, exposing it as an RL environment. Agents interact with the device through a universal action interface that mimics a human touchscreen: sending localized touch and lift events. The environment returns RGB pixel observations of the screen and rewards based on custom task definitions (e.g., rewards for successfully sending an email or achieving a score in a game).

Who it’s for

It is designed for AI researchers focusing on Reinforcement Learning, specifically those interested in agents that can interact with complex, real-time user interfaces and learn to perform tasks across diverse Android applications.

Highlights

• Real-world context: Uses an unaltered Android OS, meaning agents learn on a system used by billions of people.
• Hybrid action space: Combines continuous coordinates for location and discrete signals for touch/lift events to enable complex gestures like swiping and pinching.
• Pixel-based observations: Agents receive RGB values of the screen, with options for downsampling.
• Real-time simulation: The environment dynamics run in real-time and do not wait for the agent to deliberate.
• Flexible task definition: Supports the creation of custom tasks for any Android app, including games and productivity tools.