simular-ai/Agent-S

Agent S: an open agentic framework that uses computers like a human

What it solves

Agent S provides a way for AI agents to operate a computer's graphical user interface (GUI) just as a human would. It eliminates the need for custom API integrations or per-app scripting by allowing an agent to perceive the screen and interact with it using mouse, keyboard, and scrolling actions across various desktop and web applications.

How it works

The framework uses a combination of a main generation model (such as GPT-5) and a specialized grounding model (such as UI-TARS) to translate natural-language instructions into executable GUI actions. It captures screenshots of the screen, analyzes them, and determines the necessary clicks or keystrokes to complete a task. It can also be configured with a local coding environment to execute Python and Bash code for tasks involving data processing or system automation.

Who it’s for

This project is designed for researchers studying OS agents, developers looking to automate their own desktop workflows, and contributors to open-source computer-use AI.

Highlights

  • Cross-Platform Support: Works on macOS, Windows, and Linux.
  • Human-Level Performance: Agent S3 was the first computer-use agent to surpass human performance on the OSWorld benchmark (72.60%).
  • Model Flexibility: Compatible with models from OpenAI, Anthropic, Gemini, and open-weight providers via vLLM or Open Router.
  • Integrated Coding Environment: Optional capability to execute arbitrary Python and Bash code locally for complex system tasks.

Related

  • Project
    injaneity/pi-computer-use

    A Pi extension that enables AI agents to operate desktop applications on macOS, Windows, and Linux by interacting with the GUI when APIs are unavailable.

  • Project
    X-PLUG/MobileAgent

    A family of multi-platform GUI agents and foundation models that automate tasks across mobile, desktop, and web interfaces using visual perception and planning.

  • Project
    erthorpabar/pabar_agent_runtime

    An AI agent runtime framework that integrates chat, RAG, and agentic loops to enable professional automation tasks across desktop and mobile platforms.

  • Project
    AgentiLoop/Agent

    A native macOS app that provides an autonomous AI agent capable of controlling the desktop, managing codebases, and automating system tasks across 18 LLM providers.

  • Project
    TurixAI/TuriX-CUA

    TuriX is an open-source computer-use agent that allows AI models to perform real actions on a desktop GUI across macOS, Windows, and Linux without needing app-specific APIs.