m-bain/whisperX

What it solves

WhisperX improves upon OpenAI's Whisper by providing significantly faster transcription speeds, more accurate word-level timestamps, and integrated speaker diarization. It addresses the common issues of utterance-level timestamps that can be off by several seconds and the lack of native batching in the original Whisper model.

How it works

WhisperX uses a multi-stage pipeline to process audio:

1. Transcription: It utilizes a faster-whisper backend for batched inference, allowing for speeds up to 70x real-time.
2. Alignment: To achieve precise word-level timestamps, it uses a phoneme-based ASR model (like wav2vec2.0) for forced alignment of the transcription to the audio.
3. Diarization: It integrates pyannote-audio to partition the audio stream and assign speaker ID labels to the transcript.
4. Preprocessing: Voice Activity Detection (VAD) is used to reduce hallucinations and enable efficient batching without degrading Word Error Rate (WER).

Who it’s for

This tool is designed for developers and researchers who need high-speed, time-accurate speech-to-text transcription with speaker identification for long-form audio recordings.

Highlights

• Extreme Speed: Up to 70x real-time transcription using large-v2.
• Word-Level Precision: Accurate timestamps via forced phoneme alignment.
• Speaker Identification: Integrated speaker diarization to label who is speaking.
• Efficient Memory: Requires less than 8GB of GPU memory for large-v2 with beam size 5.
• Multilingual Support: Supports multiple languages including English, French, German, Spanish, and Italian, with automatic alignment model selection.