Agent-S: an autonomous GUI agent framework that surpasses human performance on OSWorld for cross-platform computer use

What it solves

Agent S is designed to overcome the difficulty of creating AI agents that can interact with computer graphical user interfaces (GUIs) as naturally and effectively as humans. It provides a framework for autonomous computer use across Windows, macOS, and Linux, enabling agents to perform complex tasks by interpreting screen content and executing actions.

How it works

The system uses an Agent-Computer Interface (ACI) to translate high-level agent decisions into executable commands. It typically employs a dual-model architecture: a main generation model (such as GPT-5) for high-level reasoning and a specialized grounding model (such as UI-TARS) to map those intentions to precise screen coordinates and actions. The framework can also be extended with a local coding environment, allowing the agent to execute Python and Bash code for tasks like data processing or system automation that are more efficient than GUI interaction.

Who it’s for

This project is for developers and researchers building autonomous GUI agents, automation engineers looking to replace manual workflows with AI, and those interested in state-of-the-art computer-use agents (CUA).

Highlights

• Human-Level Performance: Agent S3 has surpassed human-level performance on the OSWorld benchmark (72.60%).
• Cross-Platform Support: Works across Windows, macOS, and Linux.
• Zero-Shot Generalization: Demonstrates strong ability to function on new environments like WindowsAgentArena and AndroidWorld without specific training.
• Hybrid Interaction: Combines GUI interaction with the ability to execute local code for complex system tasks.