Holo1 and Surfer-H: Open-Source Action VLMs for GUI Automation

H Company has introduced Holo1, a family of open-source Action Vision Language Models (VLMs), and Surfer-H, a web-native agent designed for human-like browser interaction. This release provides a cost-efficient path to high-performance web automation by combining precise UI localization with a modular agent architecture.

Holo1 Action VLMs

Holo1 is a family of Action VLMs specifically engineered for deep web UI understanding and precise element localization. The models are based on the Qwen2.5-VL architecture and are fully compatible with the Hugging Face transformers library.

Model Variants and Performance

The family consists of two primary sizes:

  • Holo1-3B: A smaller, efficient version of the Action VLM.
  • Holo1-7B: The larger variant, which achieves 76.2% average accuracy on common UI localization benchmarks, the highest performance recorded among small-size models.

The WebClick Benchmark

Alongside the models, H Company released WebClick, a new multimodal localization benchmark hosted on the Hugging Face Hub. WebClick contains 1,639 human-like UI tasks designed to evaluate a model's ability to accurately localize elements within a graphical user interface.

Surfer-H Web Agent

Surfer-H is a modular web-native agent that utilizes the Holo1 family of open-weights models to automate complete web tasks. Unlike agents that rely on brittle wrappers or custom APIs, Surfer-H interacts with the browser purely as a human user would, performing actions such as reading, thinking, clicking, scrolling, typing, and validating.

Modular Architecture

Surfer-H is composed of three independent components:

  1. Policy Model: Responsible for planning and driving the agent's overall behavior.
  2. Localizer Model: Utilizes Holo1 to understand visual UIs for precise interaction.
  3. Validator Model: Confirms whether the assigned tasks have been completed successfully.

Efficiency and Benchmarks

Surfer-H establishes a new Pareto frontier for cost-efficient web navigation on the WebVoyager benchmark. It achieves 92.2% accuracy on real-world web tasks with an operational cost of only $0.13 per task.

Technical Implementation

Holo1 models can be integrated using the AutoModelForImageTextToText and AutoProcessor classes from the transformers library. The models accept an image and a text instruction (e.g., "Select July 14th as the check-out date") and output a specific click position in the format Click(x, y), where x and y represent pixels from the left and top edges of the image, respectively.

Sources