baxtree/subaligner

What is Subaligner?

Subaligner is an open‑source Python tool that synchronises subtitle files with their corresponding video or audio tracks. It can align existing subtitles, generate new subtitles by transcribing the media, and even translate them, all using modern machine‑learning models (e.g., OpenAI Whisper, HuggingFace Transformers) and classic audio‑processing libraries.

---

Core Capabilities

Capability	How it works (as described in the README)	Typical command example
Single‑stage alignment	Fast, global shift estimation (lower latency).	subaligner -m single -v video.mp4 -s subtitle.srt
Dual‑stage alignment	Two‑pass process: a global offset followed by per‑segment fine‑tuning (higher latency, higher accuracy).	subaligner -m dual -v video.mp4 -s subtitle.srt
Transcription	Calls Whisper (or another LLM‑based speech‑to‑text model) to create subtitles from audio.	subaligner -m transcribe -v video.mp4 -ml eng -mr whisper -mf small -o subtitle.srt
Translation	After transcription (or on an existing subtitle) it can translate using models such as Helsinki‑NLP, Facebook‑MBart, or M2M100.	subaligner -m dual -v video.mp4 -s subtitle.srt -t eng,spa -tr helsinki-nlp
Manual shifting	Simple offset addition/subtraction in seconds.	subaligner -m shift --subtitle_path subtitle.srt -os 5.5
Batch processing	Align many videos/subtitles in a directory tree.	subaligner_batch -m dual -vd videos/ -sd subtitles/ -od aligned_subtitles/
Custom model training	Train a new alignment model on your own video‑subtitle pairs.	subaligner_train -vd VIDEO_DIRECTORY -sd SUBTITLE_DIRECTORY -tod TRAINING_OUTPUT_DIRECTORY

---

Installation Overview

1. System dependency – FFmpeg must be present (apt-get install ffmpeg or brew install ffmpeg).
2. Python package – pip install subaligner (recommended inside a virtual environment).
3. Optional extras:
   • subaligner[llm] – pulls in Whisper and translation models.
   • subaligner[stretch] – adds forced‑alignment utilities (requires eSpeak).
   • subaligner[dev] / subaligner[harmony] – development and full‑feature sets.
4. Docker – a ready‑made image (docker pull baxtree/subaligner) lets you run the CLI without local installs.

---

Quick‑Start Example

# Align an existing English SRT to a local MP4 (dual‑stage for best accuracy)
subaligner -m dual -v video.mp4 -s subtitle.srt -o subtitle_aligned.srt

The command reads the video, extracts audio features, predicts a global offset, then refines each subtitle line. The result is written to subtitle_aligned.srt.

---

How It Works (High‑Level Architecture)

1. Feature extraction – audio is processed with librosa and TensorFlow‑based models to obtain embeddings.
2. Global alignment – a trained neural model predicts a single time shift for the whole file.
3. Parallel per‑segment alignment – the video is split into short chunks; each chunk is aligned independently, allowing fine‑grained correction.
4. Optional transcription – Whisper (or another LLM) converts speech to text when -m transcribe is chosen.
5. Optional translation – HuggingFace Transformers models translate the transcript or original subtitles.

---

When to Use Which Mode

Situation	Recommended mode
You need a fast fix and subtitles are only slightly out of sync	single (global shift)
Precise timing is critical (e.g., for dubbing or captioning)	dual (global + per‑segment)
No subtitle exists – you need a new transcript	transcribe (choose Whisper model size via -mf)
You have a transcript in one language and need a translation	dual/single + -t src,tgt + -tr <model>
You want to process many files automatically	subaligner_batch

---

Extensibility

• Custom models – train on your own domain data with subaligner_train and then point the CLI to the new model.
• Prompt engineering – the -ip flag lets you prepend a global prompt to Whisper, and --use_prior_prompting feeds the previous subtitle line as context for the next one.
• Word‑level timestamps – add --word_time_codes to get a JSON output where each word has its own start/end time, useful for downstream editing tools.

---

Community & Support

• Documentation – https://subaligner.readthedocs.io (auto‑generated from the repo).
• CI / Test coverage – GitHub Actions badge shows continuous integration; Codecov badge indicates test coverage.
• Citation – a Zenodo DOI is provided for academic use.
• Contributions – install with the dev extra to get testing and linting tools.

---

TL;DR

Subaligner is a Python‑based CLI (and Docker image) that aligns, creates, and translates subtitles using a combination of classic audio processing, TensorFlow models, and large‑language‑model transcription/translation back‑ends. It works with virtually any subtitle or media format, offers both quick‑global and high‑precision dual‑stage alignment, and can be extended by training your own models.