OthersideAI/self-operating-computer

What it solves

Performing computer tasks automatically often requires writing scripts that break when the interface changes. This framework lets a multimodal model watch the screen and take actions just like a human—clicking, typing, scrolling—so it can handle dynamic or unfamiliar interfaces without custom code.

How it works

The model receives a screenshot of the current screen and a text objective. It then outputs a series of mouse and keyboard actions, which the framework executes. The framework includes multiple model integrations (GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL, LLaVa) and special modes:

• OCR mode: uses optical character recognition to create a clickable map of text elements, improving accuracy.
• Set-of-Mark (SoM) mode: uses a simple object detector to mark buttons and enhance visual grounding. Voice input is also supported via the --voice flag.

Who it's for

Developers and researchers who want to experiment with computer-use agents, test different multimodal models, or build on an early open-source example of this capability. It requires an API key for the chosen model and works on Mac, Windows, and Linux (with X server).

Highlights

• One of the first open-source frameworks for full computer-use (Nov 2023).
• Multiple model backends: GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL, and LLaVa via Ollama.
• OCR mode maps text to coordinates for reliable clicking.
• Set-of-Mark prompting improves visual grounding.
• Voice input for hands‑free objective entry.
• Simple install and run: pip install and the operate command.