microsoft/UFO

UFO³: Weaving the Digital Agent Galaxy

What it solves

UFO は単一または複数デバイス上のユーザーインターフェース自動化のためのフレームワークです。Windows、Linux、Android といった異なる OS やアプリケーションにまたがる複雑なマルチステップワークフローの実行の難しさを解決し、単純な順次タスク実行を超えてエージェントの「銀河」的な協調を実現します。

How it works

The project consists of two primary components:

  • UFO³ Galaxy: A multi-device orchestration framework. It uses a ConstellationAgent to decompose user requests into a Directed Acyclic Graph (DAG) of tasks. A TaskOrchestrator then assigns these tasks to the most suitable devices based on platform and resource capabilities, executing them asynchronously via a secure WebSocket-based Agent Interaction Protocol (AIP).
  • UFO² Desktop AgentOS: A specialized agent for Windows automation. It integrates deeply with Windows UIA, Win32, and WinCOM to perform hybrid actions (GUI clicks and API calls). It can operate as a standalone tool or as a device agent within the Galaxy framework.

Who it’s for

  • Developers building cross-platform automation workflows.
  • Power users looking to automate complex tasks across Windows, Linux, and Android devices.
  • AI researchers focusing on GUI agents and multi-agent orchestration.

Highlights

  • Cross-Device Orchestration: Coordinates tasks across heterogeneous platforms (Windows, Linux, Android).
  • Dynamic DAG Planning: Decomposes tasks into executable graphs that can evolve based on execution feedback.
  • Deep Windows Integration: Uses a hybrid of visual and UIA detection for robust Windows OS control.
  • Efficiency: Features speculative multi-action predictions to reduce LLM calls by up to 51%.
  • MCP Integration: Supports the Model Context Protocol (MCP) for rapid tool augmentation of device agents.