microsoft/OmniParser

What it solves

OmniParser addresses the difficulty vision-based AI agents have in interacting with graphical user interfaces (GUIs). It solves the problem of accurately identifying and grounding actions on specific screen elements from a screenshot, which is often a challenge for general-purpose vision models like GPT-4V.

How it works

OmniParser parses screenshots into structured elements by combining two main components: an interactive region detector (using models like YOLOv9-E) to find clickable areas and an icon functional description model to provide captions for those elements. This process converts a raw image into a structured map that a Large Language Model (LLM) can use to generate precise coordinates and actions.

Who it’s for

It is designed for developers building GUI agents or "computer-use" agents that need to navigate software interfaces (such as Windows 11) using pure vision rather than relying on underlying accessibility trees or metadata.

Highlights

• Structured Parsing: Converts raw screenshots into easy-to-understand elements for better grounding.
• OmniTool Integration: Includes a tool to control Windows 11 VMs using various models like OpenAI (4o/o1/o3-mini), DeepSeek (R1), Qwen (2.5VL), or Anthropic Computer Use.
• Interactability Prediction: Version 1.5 adds the ability to predict whether a detected screen element is actually interactable.
• State-of-the-Art Performance: Achieves high results on grounding benchmarks like Screen Spot Pro and the Windows Agent Arena.