Smol2Operator release: turning a small VLM into an open‑source agentic GUI coder

TL;DR

Smol2Operator introduces a post‑training recipe that turns the lightweight vision‑language model SmolVLM2‑2.2B‑Instruct into an agentic GUI coder capable of perceiving and interacting with graphical user interfaces, and all code, data, and the resulting model are released open‑source.

Introduction – A Small VLM Can Become a GUI Agent

The Hugging Face team demonstrates that a 2.2 B‑parameter vision‑language model, initially lacking any GUI grounding, can be transformed into a functional GUI agent through a two‑phase supervised fine‑tuning pipeline. The result, smolagents/SmolVLM2‑2.2B‑Instruct‑Agentic‑GUI, can locate UI elements, generate low‑level actions (click, type, scroll), and perform multi‑step reasoning on screenshots. Open‑sourcing the entire training recipe, data‑processing tools, and the final model enables reproducibility and invites further research on lightweight GUI agents.

1. Unified Action Space – Standardising Heterogeneous GUI Datasets

The Problem of Inconsistent Action Formats

Existing GUI automation datasets use disparate function signatures, naming conventions, and coordinate systems, which prevents a single model from learning across them.

The Unified Solution

The authors built a transformation pipeline that parses any function call, normalises parameter ordering, and maps all actions to a common API. Key components include:

  • utils/function_parser.py – extracts and normalises function calls from raw text.
  • preprocessing/action_conversion.py – converts mobile and PyAutoGUI actions into a unified set (e.g., click, double_click, type).
  • Normalised coordinates (0‑1 range) replace pixel values, ensuring consistency across image resolutions.

Example Transformation

# Original
pyautogui.click(x=0.8102, y=0.9463)
mobile.swipe(from_coord=[0.581, 0.898], to_coord=[0.601, 0.518])

# Unified
click(x=0.8102, y=0.9463)
swipe(from_coord=[0.581, 0.898], to_coord=[0.601, 0.518])

Action Space Converter – Customising the Vocabulary

utils/action_space_converter.py lets users map the unified actions to any domain‑specific API (e.g., converting click to touch). It supports configurable function‑level and parameter‑level mappings, value transformations, and validation.

Released Datasets

The pipeline produces two new Hugging Face datasets in the unified format:

  • smolagents/aguvis-stage-1
  • smolagents/aguvis-stage-2 These are directly usable for training GUI agents.

2. Phase 1 – From Zero to Perception

Training Data

Phase 1 fine‑tunes on smolagents/aguvis-stage-1, which pairs user instructions with concrete actions expressed as code (e.g., { "user": "click on more button", "assistant": "click(x=0.8875, y=0.2281)" }). The loss is computed only on the assistant’s action output.

Image‑Resolution & Coordinate Ablations

Image size Coordinate type ScreenSpot‑v2 ↑
384 px Normalised 31.28 %
764 px Normalised 32.32 %
1152 px Normalised 33.72 %
1152 px Pixel 4.32 %
The best configuration uses 1152 px resolution with normalised coordinates.

Results

Training for two epochs on the optimal configuration raised ScreenSpot‑v2 accuracy from 0 % (baseline) to 41.27 %, a +41 % absolute gain, confirming that the model learned to ground visual elements to actions.

3. Phase 2 – From Perception to Cognition

Training Data

Phase 2 uses smolagents/aguvis-stage-2, which adds explicit reasoning tags (` click(x=0.41, y=0.178) 

### Results
Fine‑tuning the Phase 1 checkpoint for two more epochs increased ScreenSpot‑v2 performance to **61.71 %**, demonstrating that reasoning‑oriented data further improves GUI grounding. The same two‑phase recipe applied to a 460 M‑parameter nanoVLM achieved ~58 % on ScreenSpot‑v2, establishing state‑of‑the‑art performance for that model size.

---

## 4. Open‑Source Release – Everything Needed to Replicate
* **Training recipe** – `recipe.ipynb` (uses the TRL library).
* **Datasets** – `smolagents/aguvis-stage-1` and `smolagents/aguvis-stage-2`.
* **Trained model** – `smolagents/SmolVLM2-2.2B-Instruct-Agentic-GUI`.
* **Pre‑processing utilities** – function parser, action conversion system, and action‑space converter (all linked in the blog).
* **Demo Space** – an interactive Hugging Face Space (`A-Mahla/Smol2Operator`) showcases end‑to‑end task execution.
All resources are publicly available under permissive licenses.

---

## 5. Implications and Future Directions
The work proves that high‑quality, reasoning‑rich GUI data can endow even modest VLMs with reliable perception and agentic capabilities using only supervised fine‑tuning. This lowers the barrier for building domain‑specific GUI assistants and suggests that further gains may come from reinforcement‑learning or Direct Preference Optimization methods, which could enable continual learning from interaction.

---

## 6. What’s Next?
The authors anticipate exploring RL and DPO to improve real‑time adaptation and reasoning depth. Community contributions—new datasets, custom action vocabularies, or larger base models—can extend the approach to mobile, web, and specialized enterprise interfaces.

---

**Takeaway:** By unifying heterogeneous GUI action data and applying a two‑phase supervised fine‑tuning regimen, Hugging Face turned a 2.2 B‑parameter vision‑language model into an open‑source, agentic GUI coder, and released the full stack for the research community to build upon.

Sources