oculix-org/SikuliX1

SikuliX version 2.0.0+ (2019+)

What it solves

It provides a way to automate graphical user interfaces (GUIs) across Windows, macOS, and Linux without needing access to the application's source code, DOM, or accessibility APIs.

How it works

The tool uses computer vision via OpenCV to identify visual elements on the screen. Once a GUI element is recognized through image recognition, the software simulates mouse and keyboard inputs to interact with it.

Who it’s for

Users who need to automate tasks on any visible screen element regardless of the operating system or the underlying software architecture.

Highlights

  • Uses OpenCV for image-based element identification.
  • Works across multiple operating systems (Windows, macOS, Linux).
  • Operates based on the principle: "If you can see it, you can automate it."

Related

  • Project
    LazoVelko/neverclick

    A desktop application that uses computer vision to let users perform mouse actions via their keyboard, reducing the need for a physical mouse.

  • Project
    ok-oldking/ok-script

    A computer vision-based Python automation framework for Windows applications, games, and Android devices that uses OCR and template matching for UI interaction.

  • Project
    AmberSahdev/Open-Interface

    An AI-powered computer automation tool that uses LLMs to self-drive a PC by simulating keyboard and mouse inputs based on visual feedback.

  • Project
    remorses/usecomputer

    A desktop automation CLI and library for AI agents to perform screenshots, mouse control, and keyboard synthesis across macOS, Linux, and Windows.

  • Project
    opencv/opencv

    OpenCV is an open-source computer vision library that provides tools and algorithms for processing images and video to enable AI-driven visual interpretation.