m-bain/whisperX

What it solves

WhisperX addresses the limitations of OpenAI's original Whisper model, specifically the lack of native batching for fast inference and the inaccuracy of utterance-level timestamps. It provides a way to achieve high-speed transcription with precise, word-level timing and the ability to identify different speakers in an audio stream.

How it works

WhisperX uses a multi-stage pipeline to improve transcription:

1. Batched Inference: It utilizes the faster-whisper backend to perform transcription much faster than real-time (up to 70x for large-v2).
2. VAD Preprocessing: Voice Activity Detection (VAD) is used to identify speech segments, which reduces hallucinations and enables efficient batching without degrading word error rate (WER).
3. Forced Alignment: To fix timestamp inaccuracies, it uses phoneme-based ASR models (like wav2vec2.0) to align the orthographic transcription to the audio, generating precise word-level segments.
4. Speaker Diarization: It integrates pyannote-audio to partition the audio stream and assign speaker ID labels to the transcribed text.

Who it’s for

This tool is for developers and researchers who need high-performance automatic speech recognition (ASR) with precise timing and speaker identification, particularly for long-form audio recordings.

Highlights

• Extreme Speed: Up to 70x real-time transcription using batched inference.
• Word-Level Accuracy: Uses wav2vec2 alignment for precise timestamps instead of utterance-level estimates.
• Speaker Identification: Built-in support for speaker diarization to label who is speaking.
• Low Memory Footprint: Requires less than 8GB of GPU memory for the large-v2 model with specific settings.
• Multilingual Support: Supports multiple languages via automatically selected phoneme ASR alignment models.