ScreenMind: Local AI Screen Memory Powered by Gemma 4

ScreenMind is an open-source, 100% local AI memory system that captures and analyzes screen activity to create a searchable history of a user's digital life. By leveraging the multimodal capabilities of Gemma 4 E2B, ScreenMind provides a privacy-centric alternative to cloud-based screen-aware AI, operating entirely on the user's machine without network dependencies.

Local Multimodal Intelligence with Gemma 4 E2B

ScreenMind utilizes the Gemma 4 E2B model via llama.cpp to handle vision, audio, and reasoning tasks on hardware with as little as 4GB of VRAM. The system is architected to use all three modalities of the model:

  • Vision Analysis: Every screenshot is processed to return structured JSON containing the application name, activity category, mood classification, and a detailed inventory of visible elements and layout regions.
  • Audio Processing: The native audio encoder in Gemma 4 E2B is used for transcribing voice memos and meeting audio (Zoom, Teams, Meet) without requiring external dependencies like Whisper.
  • Reasoning: The model generates daily summaries, powers a conversational RAG (Retrieval-Augmented Generation) chat interface, and executes automation agents.

To balance performance and resource usage, ScreenMind offers three analysis modes: Accurate (76s with deep thinking), Balanced (40s), and Fast (~12s using a no-thinking prefill trick).

Privacy-First Architecture

ScreenMind is designed to address the privacy concerns associated with screen-recording AI by ensuring no data ever leaves the local machine. Key security features include:

  • Local Execution: Zero cloud dependencies and no telemetry.
  • Sensitive Data Filtering: Automatic redaction of credit card numbers, SSNs, API keys, and passwords before storage.
  • Encryption: AES encryption for screenshots using Fernet and the OS keyring.
  • Access Control: A session-based dashboard PIN lock with configurable auto-lock timeouts.
  • Incognito Mode: A one-click toggle to pause all recording and analysis.

Technical Pipeline and Stack

ScreenMind employs a multi-model pipeline to transform raw pixels into searchable intelligence. The workflow follows this sequence: Screenshot $\rightarrow$ EasyOCR (text extraction) $\rightarrow$ Gemma 4 E2B (understanding) $\rightarrow$ MiniLM-L6-v2 (semantic embeddings) $\rightarrow$ SQLite + FTS5 (storage and indexing).

The Tech Stack

Layer Technology Purpose
AI Model Gemma 4 E2B Vision, Audio, and Reasoning
Inference llama-server GGUF inference with OpenAI-compatible API
OCR EasyOCR Raw text extraction from screens
Embeddings all-MiniLM-L6-v2 384-dim vectors for semantic search
Database SQLite (WAL) + FTS5 Concurrent reads and full-text search
Backend FastAPI Async REST API and system orchestration
Frontend Vanilla JS/CSS Dark glassmorphism SPA dashboard

Automation and Extensibility

Beyond simple recording, ScreenMind includes an agent platform that allows users to build automations based on their screen data using two methods:

  1. AI Agents (.md): Users can write prompts in plain English. For example, a "Daily Focus Report" agent can analyze screen activity to calculate deep work hours and distraction scores.
  2. Python Plugins (.py): Developers can use the ScreenMindSDK to access activities, manage persistent state, and trigger LLM calls.

ScreenMind also implements a Model Context Protocol (MCP) Server, enabling its screen history to be exposed to AI tools such as Claude Desktop, Cursor, and VS Code. Available tools include search_screen, get_recent_activity, and get_daily_summary.

Performance Optimizations

To maintain a low system footprint while running continuously in the background, ScreenMind implements several optimization strategies:

  • Smart Capture: Uses content-change detection rather than a fixed timer to reduce unnecessary screenshots.
  • Per-App pHash Cache: A 3-tier caching system uses perceptual hashing to avoid redundant inference calls for static screens.
  • GPU Priority: Chat requests instantly cancel in-flight background analysis to free the GPU in under one second.
  • Auto-Pause: The system automatically stops capturing when heavy applications (e.g., games or 3D software) are detected.

Community Perspective

While the project positions itself as a privacy-focused alternative to Microsoft Recall, community discussion highlights the polarizing nature of screen-aware AI. One user noted that the backlash against Recall suggests a lack of general demand for such features, regardless of the privacy implementation. Additionally, some users have questioned the reasoning capabilities of the Gemma E2B model for general text generation, though ScreenMind utilizes it specifically for structured vision and audio tasks.

Sources