AmberSahdev/Open-Interface

Control Any Computer Using LLMs.

What it solves

Open Interface allows users to control their computer using natural language requests. It eliminates the need for manual navigation by automating keyboard and mouse inputs to perform complex tasks across various applications.

How it works

The system operates in a loop: it sends the user's goal and a screenshot of the current screen to an LLM backend (such as GPT-4o or Gemini). The LLM determines the necessary steps, which are then passed through an interpreter and an executer to simulate the required mouse and keyboard actions. The app continuously takes updated screenshots to assess progress and course-correct as needed.

Who it’s for

It is designed for users on macOS, Windows, and Linux who want to automate repetitive computer tasks or perform actions in software without manual input.

Highlights

  • Cross-platform support: Compatible with macOS, Windows, and Linux.
  • Flexible LLM backends: Supports GPT-4o, Google Gemini, and other OpenAI-style APIs (e.g., Llava).
  • Visual feedback loop: Uses real-time screenshots to monitor progress and adjust actions.
  • Broad application: Capable of tasks ranging from solving Wordle to creating meal plans in Google Docs.

Related

  • Project
    remorses/usecomputer

    usecomputer is a cross‑platform CLI (and TypeScript library) that lets AI agents take screenshots, map coordinates, and perform mouse/keyboard actions (click, drag, type, scroll) on macOS, Linux and Windows. It uses a native Zig backend, provides a coord‑map system for accurate screen‑to‑image mapping, streams global input events via SSE, and supports Kitty Graphics Protocol for agent‑friendly image delivery.

  • Project
    oculix-org/SikuliX1

    SikuliX1 is a historical, read-only archive of a computer vision-based automation tool that allows users to interact with graphical interfaces by identifying elements visually on the screen.

  • Project
    theJayTea/WritingTools

    Writing Tools is a free, open‑source desktop app for Windows, Linux and macOS that lets you invoke an LLM anywhere on your computer with a hot‑key. It can proofread, rewrite, change tone, translate, summarise web pages or YouTube transcripts, and even chat with the model. You can use cloud APIs (Gemini, OpenAI, Anthropic, etc.) or run local models via Ollama, llama.cpp or Apple‑silicon MLX. The app is privacy‑focused (no telemetry, local storage of API keys) and licensed under GPL‑v3.

  • Project
    go-vgo/robotgo
  • Project
    pmbstyle/Alice

    A desktop AI companion with voice interaction, system-level tool access, and long-term memory that can be run locally or via cloud APIs.