MahmoudAshraf97/whisper-diarization

What it solves

It solves the problem of identifying "who spoke when" in an audio recording. While standard transcription tools can turn speech into text, they often fail to distinguish between different speakers in a conversation, making it difficult to follow the dialogue in the resulting transcript.

How it works

The project implements a multi-stage pipeline to process audio:

1. Vocal Extraction: Uses source separation to isolate vocals from background noise to improve embedding accuracy.
2. Transcription: Generates text using OpenAI Whisper.
3. Alignment: Uses ctc-forced-aligner to correct and align timestamps, reducing time-shift errors.
4. Voice Activity Detection (VAD): Employs MarbleNet to detect speech segments and exclude silences.
5. Speaker Embedding: Uses TitaNet to extract unique speaker embeddings for each segment.
6. Association: Maps the identified speakers to the specific words and timestamps, then uses punctuation models for final realignment.

Who it’s for

Researchers and developers who need high-accuracy speaker diarization (identifying different speakers) combined with automated speech-to-text transcription.

Highlights

• Hybrid Architecture: Combines Whisper for ASR with NVIDIA NeMo (MarbleNet and TitaNet) for diarization.
• Timestamp Correction: Integrates forced alignment to minimize errors caused by time shifts.
• Parallel Processing: Includes an experimental parallel mode (diarize_parallel.py) for systems with high VRAM to speed up processing.
• Customizable: Supports various Whisper models, manual language selection, and batch size adjustments.