milind-soni/tiptour-macos

What it solves

TipTour is a macOS menu bar companion that allows users to control their desktop via AI. It provides a way to perform desktop actions—such as clicking, typing, and scrolling—either through real-time voice conversations with Gemini or by typing click-based tasks for a local agent (JEV).

How it works

TipTour operates in two distinct modes using a shared local grounding and action engine:

• Gemini Realtime Mode: Uses voice and screenshots to let users talk about their screen and request desktop actions. It supports a wide range of actions including clicking, typing, shortcuts, opening apps/URLs, and creating Apple Notes.
• JEV Text Mode: A text-based mode where users type a task. JEV identifies controls from locally detected screen labels and locations to execute and validate a sequence of click-based actions (up to 12 actions per task).

Who it’s for

macOS users (version 14.2+) who want to AI-driven desktop automation and control using their own API keys.

Highlights

• Dual-Mode Interaction: Switch between voice-driven multimodal interaction (Gemini) and text-driven click automation (JEV).
• Local Detection: JEV processes screen labels and locations locally to maintain privacy.
• Privacy-Focused: API keys are stored in the macOS Keychain, and there is no hosted key proxy.
• Contextual Focus: Users can manually paint over an area of the screen to provide Gemini with specific focus context.