readbeyond/aeneas

aeneas is a Python/C library and a set of tools to automagically synchronize audio and text (aka forced alignment)

What it solves

aeneas is designed to solve the problem of "forced alignment," which is the process of automatically synchronizing a text file (containing a list of fragments) with an audio file that narrates that text. This allows users to create a precies time-stamped map of when each piece of text is spoken in an audio recording.

How it works

The tool uses a combination of Text-to-Speech (TTS) engines (such as eSpeak, AWS Polly, or Nuance) to synthesize a reference audio version of the text. It then compares this synthesized audio with the actual narration using MFCC (Mel-frequency cepstral coefficients) and DTW (Dynamic Time Warping) to determine the exact time intervals for each text fragment.

Who it’s for

It is primarily intended for researchers, digital publishers (e.g., for EPUB 3), and creators of closed captioning (SRT, VTT) who need to align text and audio at various levels of granularity, from single words to full paragraphs.

Highlights

  • Wide Format Support: Outputs synchronization maps in numerous formats including JSON, SRT, VTT, SMIL, and TextGrid.
  • Multilingual Support: Confirmed working for 38 different languages.
  • Flexible Granularity: Supports alignment from word-level to paragraph-level.
  • Robustness: Designed to be resilient against background noise, sporadic spikes, and minor mispronunciations.
  • Batch Processing: Ability to process multiple audio/text pairs via job containers.
  • Integration: Includes wrappers for multiple TTS engines and supports audio downloads from YouTube.

Related

  • Project
    MahmoudAshraf97/ctc-forced-aligner

    A Python package for efficient forced alignment of audio and text using Hugging Face CTC models, supporting over 1,100 languages.

  • Project
    strob/gentle

    A robust forced-aligner built on Kaldi that aligns speech audio with text transcripts to provide precise word-level timing.

  • Project
    smacke/ffsubsync

    A language-agnostic tool that automatically synchronizes subtitles with video or audio files using voice activity detection and FFT-based alignment.

  • Project
    zh-plus/openlrc

    A Python library that transcribes audio using faster-whisper and uses LLMs to translate and polish the text into synchronized .lrc subtitles.

  • Project
    Asad-Ismail/MoneyPrinterTurbo-Extended

    An automated video generation tool that creates scripts, synthesizes voiceovers with local voice cloning, and matches relevant video clips with synchronized word-level subtitles.