microsoft/UFO

UFO³: Weaving the Digital Agent Galaxy

What it solves

UFO 是一个跨单一或多个设备自动化用户界面的框架。它解决了在不同操作系统（Windows、Linux、Android）和应用程序之间执行复杂多步骤工作流的难题，超越了简单的顺序任务执行，实现了协同的“星系”代理。

How it works

The project consists of two primary components:

  • UFO³ Galaxy: A multi-device orchestration framework. It uses a ConstellationAgent to decompose user requests into a Directed Acyclic Graph (DAG) of tasks. A TaskOrchestrator then assigns these tasks to the most suitable devices based on platform and resource capabilities, executing them asynchronously via a secure WebSocket-based Agent Interaction Protocol (AIP).
  • UFO² Desktop AgentOS: A specialized agent for Windows automation. It integrates deeply with Windows UIA, Win32, and WinCOM to perform hybrid actions (GUI clicks and API calls). It can operate as a standalone tool or as a device agent within the Galaxy framework.

Who it’s for

  • Developers building cross-platform automation workflows.
  • Power users looking to automate complex tasks across Windows, Linux, and Android devices.
  • AI researchers focusing on GUI agents and multi-agent orchestration.

Highlights

  • Cross-Device Orchestration: Coordinates tasks across heterogeneous platforms (Windows, Linux, Android).
  • Dynamic DAG Planning: Decomposes tasks into executable graphs that can evolve based on execution feedback.
  • Deep Windows Integration: Uses a hybrid of visual and UIA detection for robust Windows OS control.
  • Efficiency: Features speculative multi-action predictions to reduce LLM calls by up to 51%.
  • MCP Integration: Supports the Model Context Protocol (MCP) for rapid tool augmentation of device agents.