OSU-NLP-Group/UGround

🎯 What is UGround?

UGround is an open‑source research project that tackles visual grounding for graphical‑user‑interface (GUI) agents. Given a screenshot and a natural‑language description (e.g., “click the Send button”), the model predicts the (x, y) coordinates of the referenced UI element. The code, pretrained weights, training data, and evaluation suites are all released, and the work was accepted as an oral paper at ICLR 2025.

📚 Main Contributions

Component	What it provides
Model family	Three sizes of a Qwen2‑VL‑based visual‑language model (2 B, 7 B, 72 B) fine‑tuned for GUI grounding (named UGround‑V1).
Training data	Over 1 M GUI screenshots with bounding‑box annotations (the UGround‑V1 dataset) hosted on Hugging Face. A separate “box‑only” version is also available.
Evaluation suite	Ready‑to‑run scripts for four established GUI‑grounding benchmarks – ScreenSpot, Multimodal‑Mind2Web, OmniAct, and AndroidControl – plus a live‑agent benchmark (Mind2Web‑Live‑SeeAct‑V, AndroidWorld‑SeeAct‑V).
Demo	A Hugging Face Spaces web demo that lets you upload an image and a description and returns the predicted coordinate.
Paper & results	Full ICLR 2025 paper, a pre‑print on arXiv, and a leaderboard showing state‑of‑the‑art performance (e.g., 89.4 % average on ScreenSpot with the 72 B model).

🚀 Getting Started (Inference)

1. Install the required Python packages

pip install "git+https://github.com/huggingface/transformers@21fac7abba2a37fae86106f87fcf9974fd1e3830"
pip install accelerate qwen-vl-utils "vllm==0.6.1"

2. Run the model with vLLM (high‑throughput serving)

# Example for the 7‑B checkpoint
vllm serve osunlp/UGround-V1-7B \
    --api-key <your‑token> \
    --dtype float16

The repository recommends float16 for more stable decoding.

3. Build the grounding prompt

def format_openai_template(description, base64_image):
    return [{
        "role": "user",
        "content": [
            {"type": "image_url",
             "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}},
            {"type": "text",
             "text": f"""
Your task is to help the user identify the precise coordinates (x, y) of a specific area/element/object on the screen based on a description.

- Return a single string "x, y" that points to the centre of the described element.
- If the description is ambiguous, choose the most plausible element.
- The model outputs coordinates in the range [0, 1000); scale them by (width/1000, height/1000) to get pixel values.

Description: {description}

Answer:"""}
        ]
    }]

Send the messages list to the vLLM OpenAI‑compatible endpoint with temperature=0 to get deterministic coordinates.

📂 Repository Layout (high‑level)

train/ – scripts for fine‑tuning Qwen2‑VL on the UGround data.
offline_evaluation/ – code and results for the four static benchmarks (ScreenSpot, Multimodal‑Mind2Web, OmniACT, AndroidControl).
online_experiments/ – links to external repos that run the model in interactive settings (Mind2Web‑Live‑SeeAct‑V, AndroidWorld‑SeeAct‑V).
README.md – this overview, links to weights, data, demo, and citation.

📊 Performance Highlights (ScreenSpot benchmark)

Model	Size	Avg. Score
UGround‑V1‑2B (Qwen2‑VL)	2 B	77.7
UGround‑V1‑7B (Qwen2‑VL)	7 B	86.3
UGround‑V1‑72B (Qwen2‑VL)	72 B	89.4
Claude (Computer‑Use)	–	82.9

The 72 B checkpoint surpasses all prior models on the standard ScreenSpot setting and matches the best scores in the agent setting when paired with GPT‑4o as the planner.

🌐 Resources & Links

Homepage: https://osu-nlp-group.github.io/UGround
Paper (arXiv): https://arxiv.org/abs/2410.05243
Model weights (Hugging Face): https://huggingface.co/collections/osunlp/uground-677824fc5823d21267bc9812
Demo (HF Spaces): https://huggingface.co/spaces/orby-osu/UGround
Training data: https://huggingface.co/datasets/osunlp/UGround-V1-Data (box‑only version also available)
Leaderboard: https://gui-agent.github.io/grounding-leaderboard/

📜 Citation

@inproceedings{gou2024uground,
  title={Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents},
  author={Boyu Gou and Ruohan Wang and Boyuan Zheng and Yanan Xie and Cheng Chang and Yiheng Shu and Huan Sun and Yu Su},
  booktitle={International Conference on Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=kxnoqaisCT}
}

TL;DR: UGround provides a ready‑to‑use, high‑performing visual‑language model for locating UI elements from natural language, complete with data, weights, benchmarks, and a live demo—making it a solid building block for any research or product that needs GUI‑aware agents.