Indexing 669 GB of GoPro Footage Locally Using M1 Max and ML Models

Local ML Indexing for Large Video Libraries

Developer iliashad successfully indexed 668.68 GB of GoPro video footage (comprising 628 videos and approximately 15 hours and 13 minutes of duration) using an Apple M1 Max computer and local open-source machine learning models. The project aims to solve the common problem of manually scrubbing through hours of footage to find specific interesting moments for editing.

Technical Pipeline and Performance

The indexing process follows a structured pipeline to convert raw video into searchable metadata:

  1. Frame Analysis: The system divides videos into separate scenes, typically at a rate of one frame per second (1fps).
  2. Multi-Modal Analysis: The pipeline utilizes several ML models to extract different types of data:
    • Face Recognition: Identifying specific people using custom face data.
    • Object Detection: Identifying items and elements within the scene.
    • On-Screen Text (OCR): Extracting text visible in the video.
    • Scene Description: Generating textual descriptions of what is happening in the scene.
    • Transcription: Converting audio to text using OpenAI Whisper.
  3. Vector Database Storage: The extracted information is stored across three vector database collections, including video location metadata, camera names, recognized faces, detected objects, and scene descriptions.
  4. Advanced Mode: For higher quality descriptions, the system can utilize the Qwen2.5-VL-7B-Instruct model, though this increases indexing time.

Performance Metrics:

  • Total Processed Frames: 57,537 frames.
  • Total Compute Time: 67 hours, 40 minutes, and 42 seconds.

Integration and Workflow

The primary goal of the system is to streamline the post-production workflow. Once a moment is indexed and searched via semantic query, the system allows the user to send the best clips directly to a DaVinci Resolve timeline for final editing.

Community Insights and Counterpoints

Discussion among the community highlighted several technical considerations and alternative approaches to local video indexing:

  • Hardware and Acceleration: Users noted that while the M1 Max is capable, processing 67+ hours for 15 hours of footage is a significant time investment. Some suggested that spinning up GPU instances on AWS or Google Cloud could accelerate the process for larger libraries.
  • Alternative Software: Some users pointed to existing tools like DaVinci Resolve 21's built-in "AI IntelliSearch" and Jumper, which provide similar local indexing and NLE (Non-Linear Editor) integrations.
  • Practicality vs. Complexity: One critic argued that the most reliable way to capture highlights is to mark chapters manually during recording using the camera's power button, noting that AI-driven indexing is "insanely complex and inefficient" compared to human marking for this specific use case.
  • Technical Limitations: Community members pointed out that frame-level embedding can miss action-related searches that require temporal context (the movement between frames) rather than just static image analysis.

"I’d like to see embedding of actual video clips become practical in this type of workflow. Frame level embedding it covering a lot, but can miss out on a lot of action related searches."

  • Comparison to Other Projects: Other developers shared similar experiences, such as the Framedex project, which also uses local ML techniques to index years of personal video footage.

Sources