rylena/sshdesk

What it solves

SSHDESK provides a way to access a full interactive remote desktop entirely through a single SSH session, eliminating the need for dedicated remote desktop protocols (like VNC or RDP), web browsers, or additional open network ports. It allows users to view and control a remote graphical interface directly inside a standard terminal emulator.

How it works

The system works by capturing the remote host's screen and rendering it as visual data sent over an SSH PTY. It uses a forced-command configuration in OpenSSH to launch the desktop session. Depending on the terminal capabilities, it delivers high-resolution "real-pixel" tiles via the Kitty graphics protocol or falls back to lower-resolution color-cells for ANSI terminals.

Input (keyboard and mouse) is sent back through the same SSH session. On Linux, it integrates with various display stacks including X11, GNOME Wayland (via PipeWire), and other Wayland compositors (via ydotool). It also supports macOS and Windows as hosts.

Who it’s for

• Remote Administrators: Those who want graphical access to a remote machine without configuring complex firewall rules or additional services.
• AI Agents: The project includes a dedicated sshdesk-agent command set that allows AI agents to take screenshots, move the mouse, and type, enabling "computer use" capabilities over SSH.
• Terminal Power Users: Users who whom prefer staying within their terminal environment for all remote management tasks.

Highlights

• Zero Extra Ports: Operates entirely over the existing SSH port.
• Cross-Terminal Compatibility: Supports high-fidelity graphics for Kitty, Ghostty, and WezTerm, with universal fallbacks for PuTTY and mobile clients.
• Adaptive Performance: Features latest-frame scheduling to drop stale frames and reduce latency, with adaptive scaling for slow networks.
• Agent-Ready: Provides a structured JSON API (sshdesk-remote) for AI agents to interact with the desktop.
• Multi-Platform Host Support: Works across Linux (X11/Wayland), macOS, and Windows.