Purewhiter/mobilegym

What it solves

MobileGym addresses the limitations of real-device and emulator environments for training and evaluating mobile GUI agents. It solves three primary "walls":

1. Unreadable state: Unlike real devices where accessibility trees often hide critical data (like account balances or chat history), MobileGym provides the entire environment as a structured JSON snapshot, allowing for deterministic verification.
2. Unwritable state: It eliminates the difficulty of resetting or cloning app states (which are often hidden in encrypted databases or server backends) by allowing users to reset, inject, and clone state across hundreds of parallel instances.
3. Irreversible side effects: It provides a safe, sandboxed environment where agents can perform high-stakes actions (like transferring money) without real-world consequences, enabling scalable online Reinforcement Learning (RL).

How it works

MobileGym is a browser-hosted mobile simulation platform. It uses a three-layer architecture consisting of a Benchmark Layer (Python/Playwright for task templates and judges), an Apps Layer (simulated apps with declarative navigation), and an OS Runtime layer.

Key mechanisms include:

• Programmable State: The environment is treated as a single JSON blob that can be captured, configured, and restored.
• Deterministic Judges: Instead of relying on stochastic VLM judges, it uses programmatic check functions that read the structured state directly to provide sub-millisecond verdicts.
• Parallel Rollouts: It can run up to 256 parallel instances on a single server using a lightweight footprint (approx. 400 MB RAM per instance).
• Sim-to-Real Transfer: The platform is designed for behavioral fidelity, allowing agents trained in simulation to transfer their gains to real hardware.

Who it’s for

Researchers and developers building mobile GUI agents, specifically those focusing on online RL, trustworthy evaluation, and agents that interact with account-bound or backend-dependent applications.

Highlights

• 28 Simulated Apps: Includes research surrogates for popular apps like WeChat, Alipay, Bilibili, and Spotify, as well as system apps.
• 416 Task Templates: Provides a wide array of parameterized tasks for testing and training.
• High Scalability: Supports 256 parallel instances on one server with fast cold-starts (~3 seconds).
• Verifiable Outcomes: Uses state-diffing to detect unexpected side effects that real-device pipelines cannot see.
• AnswerSheet Protocol: Agents fill structured forms to prevent chain-of-thought leakage from gaming metrics.
• Sim-to-Real Validated: Demonstrated a 95.1% retention of training gains when transferring from simulation to a real device.