Hugging Face ScreenSuite Release

Hugging Face has introduced ScreenSuite, a unified evaluation framework designed to benchmark the performance of GUI (Graphical User Interface) agents. By consolidating 13 different benchmarks, ScreenSuite provides a standardized way to evaluate how Vision Language Models (VLMs) perceive, navigate, and execute tasks across desktop, mobile, and web environments.

Unified Evaluation of GUI Agent Capabilities

ScreenSuite organizes GUI agent abilities into four primary categories to provide a granular understanding of model performance. The suite integrates the following benchmarks:

Perception and Grounding

These benchmarks evaluate a model's ability to correctly perceive information on a screen and understand the precise positioning of elements for accurate clicking.

  • ScreenQA-Short & ScreenQA-Complex: Mobile environment evaluation.
  • ScreenSpot-v2 & ScreenSpot-Pro: Desktop environment evaluation.
  • WebSRC & VisualWebBench: Web environment evaluation.

Single-Step Actions

These benchmarks test the ability to solve instructions correctly using a single action.

  • Showdown-clicks: Web environment.
  • AndroidControl: Mobile environment.
  • Multimodal-Mind2web: Web environment.

Multi-Step Agents

These benchmarks assess higher-level goal achievement through a sequence of actions. Because these require virtual environments, ScreenSuite provides support for E2B desktop remote sandboxes and custom Dockerized containers for Ubuntu and Android virtual machines.

  • AndroidWorld (including MobileMiniWob): Mobile environment.
  • OSWorld: Desktop environment.
  • BrowseComp: Web environment.
  • GAIA-Web: Web environment.
  • Mind2Web-Live: Web environment.

Technical Implementation and Vision-Only Approach

ScreenSuite is built for modularity and consistency, utilizing the smolagents framework for agent execution and orchestration in online benchmarks.

A key technical distinction of ScreenSuite is its vision-only approach. Unlike many existing benchmarks that provide models with accessibility trees or DOM (Document Object Model) metadata, ScreenSuite relies solely on visual input. This design choice is intended to mirror human interaction with interfaces, making the evaluation more realistic and challenging. For example, in the Multimodal Mind2Web benchmark, ScreenSuite replaces element-name-based multi-choice selection with click precision within bounding boxes based on vision alone.

VLM Performance Rankings

Hugging Face evaluated several leading VLMs using ScreenSuite, including the Qwen-2.5-VL series (3B to 72B), UI-Tars-1.5-7B, Holo1-7B, and GPT-4o. The results generally align with industry reports, though scores may differ from other sources due to the strict vision-only requirement, which increases task difficulty.

Deployment and Usage

ScreenSuite can be deployed locally via its GitHub repository. The setup requires cloning the repository with submodules and installing the package using uv. Users can execute evaluations via run.py or examples/run_benchmarks.py for parallel model evaluation. Note that multi-step benchmarks require a bare-metal machine to run the necessary desktop and mobile environment emulators.

Sources