oculix-org/SikuliX1

SikuliX version 2.0.0+ (2019+)

What it solves

It provides a way to automate graphical user interfaces (GUIs) across Windows, macOS, and Linux without needing access to the application's source code, DOM, or accessibility APIs.

How it works

The tool uses computer vision via OpenCV to identify visual elements on the screen. Once a GUI element is recognized through image recognition, the software simulates mouse and keyboard inputs to interact with it.

Who it’s for

Users who need to automate tasks on any visible screen element regardless of the operating system or the underlying software architecture.

Highlights

  • Uses OpenCV for image-based element identification.
  • Works across multiple operating systems (Windows, macOS, Linux).
  • Operates based on the principle: "If you can see it, you can automate it."

Related

  • Project
    LazoVelko/neverclick

    A desktop application that uses computer vision to let users perform mouse actions via their keyboard, reducing the need for a physical mouse.

  • Project
    ok-oldking/ok-script

    A computer vision-based Python automation framework for Windows apps, games, and Android devices that supports OCR and template matching.

  • Project
    AmberSahdev/Open-Interface

    An AI-powered computer automation tool that uses LLMs to self-drive a PC by simulating keyboard and mouse inputs based on visual feedback.

  • Project
    remorses/usecomputer

    usecomputer is a cross‑platform CLI (and TypeScript library) that lets AI agents take screenshots, map coordinates, and perform mouse/keyboard actions (click, drag, type, scroll) on macOS, Linux and Windows. It uses a native Zig backend, provides a coord‑map system for accurate screen‑to‑image mapping, streams global input events via SSE, and supports Kitty Graphics Protocol for agent‑friendly image delivery.

  • Project
    opencv/opencv

    OpenCV is an open-source computer vision library that provides the tools and algorithms necessary to build AI-powered visual interpretation software.