microsoft/UFO

UFO³: Weaving the Digital Agent Galaxy

What it solves

UFO 是一個跨單一或多個裝置自動化使用者介面的框架。它解決了在不同作業系統（Windows、Linux、Android）與應用程式之間執行複雜多步工作流程的困難，超越了簡單的順序任務執行，達成協調的「星系」代理。

How it works

The project consists of two primary components:

  • UFO³ Galaxy: A multi-device orchestration framework. It uses a ConstellationAgent to decompose user requests into a Directed Acyclic Graph (DAG) of tasks. A TaskOrchestrator then assigns these tasks to the most suitable devices based on platform and resource capabilities, executing them asynchronously via a secure WebSocket-based Agent Interaction Protocol (AIP).
  • UFO² Desktop AgentOS: A specialized agent for Windows automation. It integrates deeply with Windows UIA, Win32, and WinCOM to perform hybrid actions (GUI clicks and API calls). It can operate as a standalone tool or as a device agent within the Galaxy framework.

Who it’s for

  • Developers building cross-platform automation workflows.
  • Power users looking to automate complex tasks across Windows, Linux, and Android devices.
  • AI researchers focusing on GUI agents and multi-agent orchestration.

Highlights

  • Cross-Device Orchestration: Coordinates tasks across heterogeneous platforms (Windows, Linux, Android).
  • Dynamic DAG Planning: Decomposes tasks into executable graphs that can evolve based on execution feedback.
  • Deep Windows Integration: Uses a hybrid of visual and UIA detection for robust Windows OS control.
  • Efficiency: Features speculative multi-action predictions to reduce LLM calls by up to 51%.
  • MCP Integration: Supports the Model Context Protocol (MCP) for rapid tool augmentation of device agents.