bytedance/UI-TARS-desktop

The Open-Source Multimodal AI Agent Stack: Connecting Cutting-Edge AI Models and Agent Infra

What it solves

It enables computers and browsers to be controlled using natural language instructions. By treating the user interface (GUI) as a visual input, it allows an AI agent to perform complex tasks—like booking flights, adjusting software settings, or checking GitHub issues—without requiring manual API integrations for every single application.

How it works

The project consists of two main components:

  1. Agent TARS: A multimodal AI agent stack that integrates Vision-Language Models (VLMs) with a CLI and Web UI. It uses a hybrid strategy to control browsers via visual grounding, DOM analysis, or a combination of both. It also integrates with Model Context Protocol (MCP) servers to connect to real-world tools.
  2. UI-TARS Desktop: A native application that uses the UI-TARS model (and Seed-1.5-VL/1.6 series) to provide precise mouse and keyboard control based on screenshots and visual recognition.

Who it’s for

  • Developers looking to build GUI automation agents using the provided SDK.
  • Users who want to automate repetitive desktop or browser tasks using natural language.
  • Researchers interested in multimodal agents and visual grounding for GUI interaction.

Highlights

  • Hybrid Browser Control: Supports GUI-based visual grounding and DOM-based interaction.
  • Remote Operation: Ability to remotely control computers and browsers.
  • MCP Integration: Built on the Model Context Protocol to easily mount external tools.
  • Cross-Platform: Works across Windows, MacOS, and web browsers.
  • Local Processing: UI-TARS Desktop offers private and secure local processing.

Related

  • Project
    X-PLUG/MobileAgent

    A family of multi-platform GUI agents and foundation models that automate tasks across mobile, desktop, and web interfaces using visual perception and planning.

  • Project
    Eynzof/Hermes-CN-Desktop

    A native Windows and macOS desktop client for Hermes Agent, providing an integrated UI and runtime management for AI agents.

  • Project
    EKKOLearnAI/hermes-studio

    A desktop app and web console that provides a visual interface and local runtime for managing Hermes Agent, featuring visual workflows, multi-platform deployment, and local model management.

  • Project
    lahfir/agent-desktop

    A native Rust-based desktop automation CLI that enables AI agents to interact with any application via OS accessibility trees, featuring token-efficient UI traversal and session tracing.

  • Project
    erthorpabar/pabar_agent_runtime

    An AI agent runtime framework that integrates chat, RAG, and agentic loops to enable professional automation tasks across desktop and mobile platforms.