vercel-labs/agent-desktop

Computer use CLI for AI agents

What it solves

It enables AI agents to interact with desktop applications by reading and manipulating the operating system's accessibility tree, rather than relying on unreliable pixel-based screen matching.

How it works

The tool provides a CLI and C library that generates a JSON snapshot of the current desktop state. Each interactive element in the snapshot is assigned a unique reference ID. An agent can then perform actions—such as clicking, typing, or scrolling—by targeting these specific IDs. The system is designed for a loop: observe the UI, act on a reference, and observe again to verify the result.

Who it’s for

It is built for developers creating AI agents that need to perform tasks within desktop applications on macOS, Windows, or Linux.

Highlights

  • Operates via the native OS accessibility tree for precise control.
  • Headless by default, allowing actions without moving the mouse or stealing window focus.
  • Provides a structured JSON interface for easy integration into agent loops.
  • Includes over 50 operational commands for managing apps, windows, and input.
  • Supports both CLI usage and direct integration via a C library (FFI).

Related

  • Project
    botpress/botpress

    Botpress Cloud is an open‑source platform for building AI‑powered chatbots. The repo contains the CLI, SDK, client libraries, example bots, and a collection of public integrations that can be deployed to Botpress Cloud workspaces. Developers use `bp init` to scaffold a bot or integration, write TypeScript with the SDK, and deploy with `bp deploy`. The system is designed for developers who want programmatic control over chatbot creation, with optional publishing of integrations to the Botpress Hub. All code is MIT‑licensed.

  • Project
    aiming-lab/AutoResearchClaw

    An autonomous, collaborative research pipeline that turns a research idea into a conference-ready academic paper with real citations and sandbox experiments.

  • Project
    akitaonrails/ai-memory

    ai-memory is a long-term memory system for AI coding agents, storing memory as git-backed Markdown, working across agents, projects, machines, and teams, with zero-LLM default and optional LLM features.

  • Project
    livekit/agents

    LiveKit Agents is a Python SDK for building real‑time, multimodal voice agents (LLM‑driven assistants) that run on LiveKit servers. It provides unified adapters for STT/LLM/TTS, job scheduling, WebRTC/telephony integration, turn detection, MCP tool support, and a built‑in test framework.