akitaonrails/FrankSherlock

Intelligent File Searcher that can search for descriptions in media files like photos, for example "receipt from market of $42.00" using AI based classification

What it solves

Frank Sherlock is a local-only, AI-powered image cataloging and search tool designed for NAS (Network Attached Storage) users. It solves the problem of organizing and searching through massive libraries of images, PDFs, and videos without sending data to the cloud, providing a private way to classify, index, and find media files based on content rather than just filenames.

How it works

The application uses a multi-stage pipeline to process media:

  1. Discovery and Indexing: It performs a read-only scan of directories, generating fingerprints and thumbnails for new or modified files.
  2. AI Classification: It utilizes local vision LLMs (via Ollama) to describe and classify images, identify anime/manga characters, and detect documents.
  3. Text Extraction: It uses Surya OCR (with a vision LLM fallback) to extract text from documents, receipts, and screenshots.
  4. Face Recognition: It employs native ONNX models (SCRFD and ArcFace) to detect faces, create embeddings, and cluster them into identifiable people.
  5. Search: All extracted metadata, OCR text, and facial clusters are indexed into a SQLite database with FTS5 for fast full-text search.

Who it’s for

Users with large local media collections (especially on NAS) who prioritize privacy and want an AI-driven way to organize and find photos, documents, and videos using natural language search and facial recognition.

Highlights

  • Local-only processing: All AI models run on the user's machine via Ollama and ONNX, ensuring no data leaves the device.
  • Incremental scanning: Optimized for 100k+ files, allowing the app to resume interrupted scans and skip unchanged files.
  • Duplicate detection: Identifies exact duplicates via SHA-256 and near-duplicates using a combination of perceptual hashing (dHash) and LLM description similarity.
  • Face clustering: Automatically groups faces into people, allowing users to name and merge them and search by face:name.
  • Broad format support: Handles JPEG, PNG, GIF, BMP, WebP, TIFF, PDF, and various video formats (MP4, MKV, AVI, etc.).

Related

  • Project
    stereolabs/zed-unity

    A Unity plugin that integrates ZED cameras' depth sensing, spatial mapping, and object detection capabilities into the Unity engine for AR/MR development.

  • Project
    openclimatefix/graph_weather

    A PyTorch library implementing various graph neural networks for weather forecasting and data assimilation, including models like GenCast and Aurora.

  • Project
    arthurflor23/handwritten-text-recognition

    A TensorFlow-based framework for handwritten text recognition and synthesis, featuring extensive data augmentation and support for multiple standard HTR datasets.

  • Project
    TIGER-AI-Lab/Pixel-Reasoner

    Pixel Reasoner is a research framework that adds visual‑manipulation operations (zoom‑in, select‑frame, etc.) to a Vision‑Language Model, training it via instruction‑tuning and curiosity‑driven reinforcement learning. The repo supplies installation guides, scripts for both SFT and RL stages, pretrained 7 B checkpoints, datasets, and evaluation pipelines for image and video benchmarks.