ScreenEnv Release: Deploying Full Stack Desktop Agents

Hugging Face has introduced ScreenEnv, a Python library designed to create isolated Ubuntu desktop environments within Docker containers. This tool allows developers to deploy and test GUI agents (Computer Use agents) in a sandboxed setting where the agent can see, click, and interact with real applications.

Sandboxed Desktop Environments for GUI Agents

ScreenEnv provides a complete virtual desktop session that can be fully controlled via code. Unlike traditional automation frameworks that may be brittle or require complex VM setups, ScreenEnv leverages Docker to provide an environment that is isolated, reproducible, and deployable in under 10 seconds. It supports both AMD64 and ARM64 architectures.

Key capabilities of the environment include:

  • Full Desktop Control: Automation of mouse and keyboard, window management, application launching, file operations, and terminal access.
  • Session Management: The ability to record entire sessions and manage the desktop experience holistically.
  • Docker Native: Elimination of complex VM requirements by using Docker containers for isolation.

Integration Approaches

ScreenEnv offers two distinct methods for integrating with agent frameworks and backend systems:

Direct Sandbox API

For developers requiring fine-grained control or those using custom agent frameworks, the Direct Sandbox API allows for programmatic control. This includes the ability to launch applications, write text, and capture screenshots directly through the Sandbox class.

MCP Server Integration

For AI systems that support the Model Context Protocol (MCP), ScreenEnv provides an MCPRemoteServer. This allows AI agents to connect to the desktop environment and control it via MCP tools, such as calling a "screenshot" tool to receive image bytes of the current screen state.

Building Desktop Agents with smolagents

ScreenEnv is natively compatible with smolagents, enabling the creation of custom Desktop Agents. The process involves three primary steps:

  1. Model Selection: Users can choose a backend Vision Language Model (VLM) using connectors for OpenAI, Hugging Face Inference Endpoints (e.g., Qwen2.5-VL-7B-Instruct), local Transformers models, or other providers via LiteLLM (e.g., Claude 3.5 Sonnet).
  2. Agent Definition: By inheriting from DesktopAgentBase, developers can implement the _setup_desktop_tools method to define the agent's action space. Example tools include click(x, y), write(text), press(key), open(file_or_url), and launch_app(app_name).
  3. Execution: The agent can be assigned a specific task—such as opening LibreOffice to write and save a report—and executed within a defined Sandbox environment with a specific resolution (e.g., 1920x1080).

Future Roadmap

While currently focused on Ubuntu, ScreenEnv aims to expand its support to include Android, macOS, and Windows. The goal is to enable cross-platform GUI automation and the creation of reproducible, sandboxed environments specifically for benchmarking and evaluation.

Sources