Overtone Captions: A Multi-Format Captioning System for Responsive Video

Overview

Overtone is a captioning system designed to solve the problem of rigid, one-size-fits-all subtitles by generating responsive captions that adapt to different screen aspect ratios. By storing word-level timings and speaker data in a single .overtone.json file, the system allows developers to generate distinct caption cuts for various player shapes, such as 16:9 desktop players and 9:16 mobile social media formats, from one source.

Responsive Captioning and Formatting

Traditional caption formats like WebVTT or TTML store finished cues, which limits a player's ability to rewrap text effectively for different screen widths. Overtone addresses this by storing each spoken word with its associated timestamp and speaker identity.

This approach enables the system to apply specific line-cutting rules for different display profiles. These rules prioritize linguistic readability by avoiding line breaks on weak words (like "the" or "of"), keeping names and numbers together, and ensuring that pauses in speech are reflected as breaks in the text. The system includes a video.js plugin that allows for seamless integration of these responsive captions into web-based video players.

The Transcription and Processing Pipeline

Overtone utilizes a multi-engine pipeline to improve transcription accuracy and formatting:

  • Transcription: The system uses multiple engines to process audio. A text-based engine (gpt-transcribe) handles word identification, while a timing engine (Azure AI Speech) provides word-level timestamps and speaker labels.
  • Correction: An LLM-based editor reviews the transcript to correct misheard terms, identify speakers, and label key words. It uses provided context and glossaries to improve accuracy, flagging uncertain edits for manual review.
  • Recutting: Existing subtitles (SRT or WebVTT) can be imported into the system using the overtone recut command, which re-processes the cues into the Overtone format without requiring a full re-transcription.

Performance and Benchmarks

Overtone focuses on minimizing "silent failures" and improving the quality of line breaks compared to standard automated captioning methods. Benchmarks indicate that Overtone significantly reduces the frequency of awkward line breaks—where lines end on conjunctions or prepositions—compared to standard length-limit-based algorithms. The system is designed to be extensible, supporting various providers for transcription and offering a modular architecture for language-specific rule sets.

Community Feedback

Discussion surrounding the project highlights a growing frustration with the declining quality of automated subtitles in modern media. Users have noted that current industry standards often suffer from poor accuracy and a lack of human oversight. Regarding the potential for future features, commenters have expressed a strong desire for better handling of on-screen text translation:

"I am begging you to ensure there is some kind of system for translating foreign text on screen - most ideally, in a slightly differentiated font. So often someone is walking by a sign in some other language, and their comments on it are translated, but the sign is not!"

Technical Implementation

Overtone is built as a set of Node.js packages, including a CLI for processing, a video.js plugin for playback, and a format specification package. It supports offline processing via whisper.cpp and Ollama, or cloud-based processing through providers like OpenAI and Azure. The project is licensed under the MIT license and is available on GitHub.

Sources