Indexing a Year of Video Locally: Leveraging Gemma 4 31B on M1 Max Hardware

For many videographers and content creators, the archive is a growing liability. Footage accumulates across SSDs and cloud drives faster than it can be edited, resulting in a mountain of IMG_*.mov and DJI_*.mp4 files that are effectively invisible. The bottleneck isn't the editing software—it's the index. Most AI video editors solve the second problem (the cut) while assuming the first problem (the labeling) is already solved.

To solve this, a custom local indexing pipeline was built to transform a year of raw footage from the Maasai Mara into a searchable, English-language database. By leveraging local LLMs and a "sidecar" file architecture, it is possible to make a massive archive queryable without uploading terabytes of data to the cloud.

The Architecture of a Local Index

Building a local-first index requires balancing computational cost with the need for exhaustive metadata. The goal was to ensure that a single "vision pass" over a clip captured every possible data point needed for future editing, avoiding the need to re-process files.

The Pipeline

The indexing process follows a multi-stage pipeline for every clip:

  1. Metadata Extraction: ffprobe handles basic file metadata, while exiftool extracts GPS coordinates and altitude.
  2. Geocoding: GPS data is reverse-geocoded via Nominatim to provide human-readable location names.
  3. Frame Extraction: ffmpeg extracts five evenly-spaced frames at 1920px to serve as visual samples for the vision model.
  4. Transcription: WhisperX provides word-level alignment and speaker diarization across 97 languages.
  5. Facial Recognition: insightface detects faces and stores 512-dimensional ArcFace embeddings in a centralized SQLite database for cross-archive person queries.
  6. Vision Analysis: A vision model (Gemma 4 31B locally via LM Studio) analyzes the frames, transcripts, and folder context to generate a structured description.
  7. Sidecar Generation: The final output is written to a .description.md file living alongside the original clip.

Why Sidecars?

Rather than using a central database, the system uses plain-text Markdown sidecars. This ensures the index is grep-able, survives the failure of any specific indexing tool, and travels with the data when files are moved between physical drives.

Pushing the Hardware: M1 Max and 50GB of Swap

One of the most surprising results of this project was the performance of a 2021 MacBook Pro M1 Max (64GB RAM). Running Gemma 4 31B Q4 in LM Studio pushed the machine to its absolute limits, resulting in over 50GB of swap usage during peak bulk runs.

While high swap usage is often cautioned against due to SSD wear, the unified memory architecture and high memory bandwidth of Apple Silicon make this viable for short-term, intensive bursts. As one community member noted, "on x86 that much swap would make inference unusably slow."

This demonstrates a critical point for local AI: hardware that was considered "high-end" five years ago is now the floor for running capable 30B+ parameter models, provided the user is willing to let the fans spin and the swap file grow.

Technical Lessons and Bug Triage

Developing this system using Claude Code as an orchestrator revealed several critical lessons in AI-native engineering:

  • Defensive API Calls: When using fast-moving libraries like WhisperX, constructor signatures can change. Implementing signature introspection (trying one keyword argument and falling back to another) provides cheap insurance against breaking changes.
  • Silent Failures in CLIs: Non-interactive CLI modes often return error messages as successful string responses (exit code 0). Robust scripting requires string-matching for specific failure phrases (e.g., "I need permission") rather than relying on exit codes.
  • The Danger of Union-Types: Prompting a model to return "an integer or the string 'many'" creates downstream parsing headaches. It is always better to enforce a strict type (e.g., always an integer) and handle estimation in the prompt.
  • Permissive Culling: In photography, culling is aggressive. In video memories, it must be permissive. Criteria should focus on "non-recordings" (lens caps, test clips) rather than "imperfect captures" (motion blur), as the blur often defines the "vibe" of a memory.

Key Insights on Local LLMs

Enums vs. Instructions

To prevent confabulation (hallucinations), structured schemas are far superior to open-ended instructions. For example, forcing a model to choose from a specific list of lighting conditions (golden_hour | bright_daylight | nighttime) proved more accurate than asking for a prose description of the light, which occasionally led the model to describe a nighttime scene as "brightly lit."

The Two-Tier Scaling Model

Local 31B models with structured prompts can close the gap to cloud models like Claude 3.5 Sonnet for the bulk of the work. The most efficient scaling strategy is a two-tier approach:

  1. Local Pass: Run a local model for bulk indexing of thousands of clips.
  2. Cloud Pass: Use high-end cloud APIs only for the small percentage of clips flagged for review or those requiring extreme nuance.

Conclusion

By shifting the focus from the editor to the index, the process of managing a massive video archive becomes tractable. Once the archive is queryable in plain English—"show me handheld interior clips from Mara, golden hour, with people, longer than 8 seconds"—the actual editing becomes a thin layer of orchestration. The index is the prerequisite that most AI video tools are skipping, but it is the only way to make a lifetime of footage truly useful.

Sources