linto-ai/whisper-timestamped

What it solves

Whisper models from OpenAI provide high-quality multilingual speech recognition but lack precise word-level timestamps, typically only offering approximate segment-level timing (often with 1-second accuracy). whisper-timestamped solves this by predicting accurate word timestamps and providing more precise estimations of speech segments.

How it works

The project implements an approach based on Dynamic Time Warping (DTW) applied to cross-attention weights. This allows the system to align words to the audio signal on the fly after each speech segment is decoded, often without requiring additional inference steps. It also assigns confidence scores to each word and segment.

Who it’s for

Developers and researchers needing high-precision timing for transcribed text, such as for creating accurate subtitles, audio-to-text alignment, or analyzing speech patterns across multiple languages.

Highlights

• Word-Level Precision: Predicts exact start and end times for individual words.
• Confidence Scoring: Assigns a confidence score to every word and segment.
• memory Efficient: Designed to process long audio files with minimal additional memory overhead compared to standard Whisper.
• VAD Integration: Supports Voice Activity Detection (via Silero, Auditok) to prevent hallucinations during periods of silence.
• Multilingual Support: Compatible with any version of openai-whisper and supports fine-tuned models from the Hugging Face Hub.