AmberSahdev/Open-Interface

Control Any Computer Using LLMs.

What it solves

Open Interface allows users to control their computer using natural language requests. It eliminates the need for manual navigation by automating keyboard and mouse inputs to perform complex tasks across various applications.

How it works

The system operates in a loop: it sends the user's goal and a screenshot of the current screen to an LLM backend (such as GPT-4o or Gemini). The LLM determines the necessary steps, which are then passed through an interpreter and an executer to simulate the required mouse and keyboard actions. The app continuously takes updated screenshots to assess progress and course-correct as needed.

Who it’s for

It is designed for users on macOS, Windows, and Linux who want to automate repetitive computer tasks or perform actions in software without manual input.

Highlights

  • Cross-platform support: Compatible with macOS, Windows, and Linux.
  • Flexible LLM backends: Supports GPT-4o, Google Gemini, and other OpenAI-style APIs (e.g., Llava).
  • Visual feedback loop: Uses real-time screenshots to monitor progress and adjust actions.
  • Broad application: Capable of tasks ranging from solving Wordle to creating meal plans in Google Docs.

Related

  • Project
    remorses/usecomputer

    usecomputer is a cross‑platform CLI (and TypeScript library) that lets AI agents take screenshots, map coordinates, and perform mouse/keyboard actions (click, drag, type, scroll) on macOS, Linux and Windows. It uses a native Zig backend, provides a coord‑map system for accurate screen‑to‑image mapping, streams global input events via SSE, and supports Kitty Graphics Protocol for agent‑friendly image delivery.

  • Project
    oculix-org/SikuliX1

    A computer vision-based automation tool that identifies and interacts with screen elements using image recognition to simulate mouse and keyboard input.

  • Project
    theJayTea/WritingTools

    Writing Tools is a free, open‑source desktop app for Windows, Linux and macOS that lets you invoke an LLM anywhere on your computer with a hot‑key. It can proofread, rewrite, change tone, translate, summarise web pages or YouTube transcripts, and even chat with the model. You can use cloud APIs (Gemini, OpenAI, Anthropic, etc.) or run local models via Ollama, llama.cpp or Apple‑silicon MLX. The app is privacy‑focused (no telemetry, local storage of API keys) and licensed under GPL‑v3.

  • Project
    go-vgo/robotgo

    A cross-platform Golang library for desktop automation and AI computer use, enabling programmatic control of the mouse, keyboard, and screen.

  • Project
    pmbstyle/Alice

    An open-source AI desktop companion that integrates voice interaction, system-level computer control, and long-term memory to assist with daily tasks and creative work.