ssrajadh/sentrysearch

What it solves

SentrySearch enables semantic search over video footage, allowing users to find specific moments by typing a natural language description or providing a reference image. It eliminates the need to manually scrub through hours of video to find a particular event, automatically trimming the most relevant segment into a separate clip.

How it works

The tool splits videos into overlapping chunks and generates embeddings for each chunk using a multimodal model (Google Gemini, Alibaba Qwen, or a local Qwen3-VL model). These vectors are stored in a local ChromaDB database. When a user searches, the query (text or image) is embedded into the same vector space and matched against the stored video embeddings using cosine similarity. The top match is then automatically trimmed from the original file using ffmpeg.

Who it’s for

It is designed for anyone managing large amounts of video footage, such as dashcam owners or security camera operators, who need to quickly locate specific visual events.

Highlights

• Flexible Backends: Supports cloud APIs (Gemini, DashScope) and local execution (PyTorch or MLX for Apple Silicon) for privacy and offline use.
• Multimodal Search: Search using either natural language text or reference images.
• Anomaly Detection: A highlights feature that identifies and trims the most statistically anomalous clips in an index.
• VLM Reranking: An optional --rerank flag that uses a Vision Language Model to refine the initial embedding results for higher accuracy.
• Hardware Optimized: Includes specific optimizations for Apple Silicon (MLX) and NVIDIA GPUs (4-bit quantization) to reduce memory and processing time.