simular-ai/Agent-S

Agent S: the state-of-the-art open source computer use agents

What it solves

Agent S is designed to automate desktop and web applications by interacting with the graphical user interface (GUI) exactly as a human would—using a mouse, keyboard, and screen. It eliminates the need for application-specific APIs or custom scripting for every app, allowing a natural-language task to be executed across any software on macOS, Windows, and Linux.

How it works

The framework uses a combination of a main generation model (such as GPT-5 or Claude) and a grounding model (such as UI-TARS) to translate natural language instructions into precise GUI actions. It captures screenshots of the screen, analyzes them, and then executes clicks, typing, and scrolling. It also features a reflection agent to assist the worker agent and an optional local coding environment that allows the agent to execute Python and Bash code for data processing or system automation tasks.

Who it’s for

This project is intended for AI researchers studying operating system agents, developers looking to automate their own desktop workflows, and contributors to open-source computer-use agents.

Highlights

  • Cross-Platform Support: Works on Windows, macOS, and Linux.
  • Human-Level Performance: Agent S3 surpassed human-level performance on the OSWorld benchmark (72.60%).
  • Model Agnostic: Supports models from OpenAI, Anthropic, Gemini, and open-weight providers via vLLM or Open Router.
  • Integrated Code Execution: Can execute arbitrary Python and Bash code locally to handle complex data manipulation or system tasks.

Related

  • Project
    injaneity/pi-computer-use

    A Pi extension that enables AI agents to interact with desktop applications by observing UI elements and performing actions like clicking and typing across macOS, Windows, and Linux.

  • Project
    aaif-goose/goose

    An open-source, general-purpose AI agent available as a desktop app, CLI, and API that handles coding, research, and automation tasks.

  • Project
    xlang-ai/OSWorld-V2

    OSWorld-V2 is a benchmark for evaluating AI agents' ability to operate real-world desktop environments using virtual machines and gated task datasets.

  • Project
    TurixAI/TuriX-CUA

    TuriX is an open-source computer-use agent that allows AI models to perform real actions on a desktop GUI across macOS, Windows, and Linux without needing app-specific APIs.

  • Project
    openai/openai-cua-sample-app

    A collection of reference implementations for computer-use agents that can operate web browsers and desktop software by executing code, capturing screenshots, and simulating user inputs.