Open-Interface: a cross-platform computer automation tool that uses LLMs to simulate mouse and keyboard inputs

What it solves

Open Interface allows users to control their computer using natural language requests. It eliminates the need for manual navigation by automating keyboard and mouse inputs to perform complex tasks across various applications.

How it works

The system operates in a loop: it sends the user's goal and a screenshot of the current screen to an LLM backend (such as GPT-4o or Gemini). The LLM determines the necessary steps, which are then passed through an interpreter and an executer to simulate the required mouse and keyboard actions. The app continuously takes updated screenshots to assess progress and course-correct as needed.

Who it’s for

It is designed for users on macOS, Windows, and Linux who want to automate repetitive computer tasks or perform actions in software without manual input.

Highlights

• Cross-platform support: Compatible with macOS, Windows, and Linux.
• Flexible LLM backends: Supports GPT-4o, Google Gemini, and other OpenAI-style APIs (e.g., Llava).
• Visual feedback loop: Uses real-time screenshots to monitor progress and adjust actions.
• Broad application: Capable of tasks ranging from solving Wordle to creating meal plans in Google Docs.