microsoft/UFO

UFO³: Weaving the Digital Agent Galaxy

What it solves

UFO는 단일 또는 다중 장치에서 사용자 인터페이스를 자동화하기 위한 프레임워크입니다. Windows, Linux, Android와 같은 서로 다른 운영 체제와 애플리케이션에 걸쳐 복잡하고 다단계 워크플로를 실행하는 어려움을 해결하며, 단순한 순차 작업 실행을 넘어 에이전트들의 "갤럭시"와 같은 협업을 구현합니다.

How it works

The project consists of two primary components:

  • UFO³ Galaxy: A multi-device orchestration framework. It uses a ConstellationAgent to decompose user requests into a Directed Acyclic Graph (DAG) of tasks. A TaskOrchestrator then assigns these tasks to the most suitable devices based on platform and resource capabilities, executing them asynchronously via a secure WebSocket-based Agent Interaction Protocol (AIP).
  • UFO² Desktop AgentOS: A specialized agent for Windows automation. It integrates deeply with Windows UIA, Win32, and WinCOM to perform hybrid actions (GUI clicks and API calls). It can operate as a standalone tool or as a device agent within the Galaxy framework.

Who it’s for

  • Developers building cross-platform automation workflows.
  • Power users looking to automate complex tasks across Windows, Linux, and Android devices.
  • AI researchers focusing on GUI agents and multi-agent orchestration.

Highlights

  • Cross-Device Orchestration: Coordinates tasks across heterogeneous platforms (Windows, Linux, Android).
  • Dynamic DAG Planning: Decomposes tasks into executable graphs that can evolve based on execution feedback.
  • Deep Windows Integration: Uses a hybrid of visual and UIA detection for robust Windows OS control.
  • Efficiency: Features speculative multi-action predictions to reduce LLM calls by up to 51%.
  • MCP Integration: Supports the Model Context Protocol (MCP) for rapid tool augmentation of device agents.