kyegomez/ScreenAI

What it solves

ScreenAI is designed to understand user interfaces (UI) and infographics, allowing a model to process both visual elements and text to interpret screen content.

How it works

The model uses a vision-language architecture. It takes an image and text as input, processes the image through patch sizes and a Vision Transformer (ViT), embeds and concatenates the data, and then passes it through a series of attention and feed-forward networks (FFN), including cross-attention and self-attention mechanisms, to produce an output.

Who it’s for

Developers and researchers working on UI automation, accessibility tools, or any application that requires a machine understanding of graphical user interfaces and infographics.

Highlights

• Implementation of the research paper "A Vision-Language Model for UI and Infographics Understanding".
• Multi-modal input processing (image and text).
• Customizable model parameters such as patch size, image size, and depth of the encoder and decoder.