SCM (Screen Memories) AI Search for macOS

SCM (Screen Memories) is a local-first AI search utility for macOS that allows users to search every photo and every frame of video within specified folders using local vision models. By running all inference on the machine, SCM ensures that media never leaves the device, providing a private alternative to cloud-based media indexing.

Five Specialized Search Modes

SCM implements five distinct search modes to handle different types of media data, ensuring that users can find specific moments or text regardless of the format.

Semantic File and Scene Search

SCM uses vision models to rank results by cosine similarity against image embeddings.

  • Files Mode: Searches whole photos and videos by meaning, utilizing vision ranking with filename and phrase boosts.
  • Scenes Mode: Segments videos into shots and embeds them, allowing users to jump directly to a specific timecode within a video rather than just finding the file.

OCR and Dialogue Search

For literal text retrieval, SCM bypasses vision models to ensure speed and offline availability.

  • OCR Mode: Uses Tesseract to match query tokens literally against visible text in images and frames. It supports English and 35 additional toggleable languages.
  • Dialogue Mode: Uses OpenAI's Whisper (tiny.en or base.en) to perform exact literal retrieval over video transcripts. It offers three tiers of matching: exact lines, exact words within an 8-second window, and any words spoken within the same video.

Local LLM Integration (Opt-in)

SCM includes an optional "Ask" mode powered by a llama.cpp sidecar. This allows users to chat with their local library using a RAG (Retrieval-Augmented Generation) approach, where the model answers questions based on extracted dialogue, OCR text, and filenames. Supported models include Qwen3 1.7B (default) and Llama 3.2 3B.

Technical Architecture and Pipeline

SCM is built using Electron and Bun, utilizing a variety of specialized workers for indexing and inference.

Vision Model Selection

Users can switch between four ONNX Runtime models depending on their needs for speed versus detail:

  • CLIP ViT-L/14@336: The default for real-world video scene search.
  • SigLIP-2-B/16: Optimized for the fastest bulk imports.
  • SigLIP-2-L/16@256: High-detail model for small objects and on-screen text.
  • SigLIP-B/16@384: Maximum detail.

Video Processing Pipeline

To handle the computational load of video indexing, SCM uses ffmpeg to scan for shot boundaries and build a segment plan. Users can choose from five sampling presets ranging from "Eco" (60 seconds per point) to "Ultra Pro" (2.5 seconds per point). Each segment embeds its midpoint frame and stores a poster for quick retrieval.

Library Management and Privacy

SCM employs a "privacy by construction" philosophy. All weights (Hugging Face, Tesseract CDN, GitHub) are downloaded once and verified via SHA-256. Media is copied into an app-managed library and streamed from disk. The system uses SHA-256 content hashing for deduplication, ensuring that renamed files are not re-indexed unnecessarily.

Community Insights and Considerations

Discussion among users on Hacker News highlighted several technical trade-offs and potential improvements for the tool:

  • Indexing Performance: Users noted that frame sampling rates are critical for performance. One developer mentioned that sampling 12k videos at one frame per second could take days, emphasizing the importance of SCM's configurable sampling presets.
  • OCR Frameworks: Some users suggested that Apple's native Vision framework would be more performant and accurate than Tesseract for macOS-specific applications.
  • Alternative Tools: Immich was mentioned as a cross-platform alternative that provides similar AI-powered photo and video search capabilities.
  • Resource Consumption: There were questions regarding the time and compute costs associated with processing large libraries (e.g., 12,000 videos), which SCM addresses through its background re-embedding process and cached shot plans.

"Since this is for the mac you really should be using apple's vision framework for OCR. It smokes tesseract in both speed and accuracy."

Installation and Requirements

SCM requires macOS 12+ and the Bun package manager. It can be installed via Homebrew using the following commands:

brew tap allenv0/scm
brew trust allenv0/scm
brew install --cask allenv0/scm/scm

Sources

Related

  • Project
    rtk-ai/icm

    A long-term memory system for AI coding agents that stores project decisions and preferences in a local SQLite file, shared across multiple AI tools.

  • Project
    v-modal/vmodal_sdk_swift_iphoneduo

    A Swift SDK for iOS and macOS that enables multimodal semantic search and indexing of video content using natural language, speech, and imagery.

  • Project
    kennss/SiliconScope

    SiliconScope is a native macOS system‑monitor for Apple‑Silicon that visualises CPU, GPU, Neural Engine, memory‑bandwidth, power, temperatures, battery, network, disk and per‑process stats without sudo. It adds AI‑focused views (ANE usage, bottleneck classification, token‑per‑second benchmark) and, from version 4, can monitor remote Macs, Linux‑GPU boxes or cloud VMs via an encrypted agent, presenting everything in a unified SwiftUI dashboard and menu‑bar suite.

  • Dispatch
    Sentence Transformers v5.4: Multimodal Embedding and Reranker Models

    Sentence Transformers v5.4 introduces multimodal embedding and reranker models for encoding and comparing text, images, audio, and video.

  • Dispatch
    Open-Source OCR Models Guide – Choosing, Running, and Extending Modern Vision-Language OCR

    Hugging Face announced a comprehensive guide to open-weight OCR models, detailing their capabilities, benchmark performance, cost efficiency, and how to run them locally or via hosted endpoints.