smacke/ffsubsync

What it solves

FFsubsync is a language-agnostic tool that automatically synchronizes subtitles with a video or audio file, ensuring that subtitles appear at the correct starting point and align with the spoken dialogue.

How it works

The tool uses a three-step process to align subtitles and audio:

1. Discretization: It breaks both the audio stream and the subtitles into 10ms windows.
2. Speech Detection: It determines if speech is present in each window. For subtitles, this is based on whether a subtitle is active; for audio, it uses a Voice Activity Detector (VAD) such as WebRTC or Silero VAD.
3. Alignment: It treats the speech patterns as binary strings and aligns them using a Fast Fourier Transform (FFT) to efficiently find the best-scoring offset, reducing the complexity from O(n^2) to O(n log n).

Who it’s for

Users who have subtitles that are out of sync with their video files, regardless of the language of the content, and those who need to perform bulk synchronization of subtitle files.

Highlights

• Flexible Reference: Syncs against a video/audio file or another correctly-synced subtitle file.
• Browser-based version: Offers a client-side sync tool using ffmpeg.wasm, so files never leave the user's machine.
• Robust Encoding: Automatically detects legacy character encodings (e.g., Windows-1251, GBK) to prevent crashes.
• Advanced Sync Options: Includes an experimental piecewise alignment mode (--split-penalty) for handling mid-file breaks like commercial cuts.
• Integration: Can be used as a command-line tool, a Docker image, or a Python library.
• Whisper Integration: Supports using Whisper.cpp transcripts as a reference for videos without existing subtitles.