TheStageAI/TheWhisper

📚 Quick Overview

TheWhisper is an open‑source speech‑to‑text system that builds on OpenAI’s Whisper model. The project provides:

Fine‑tuned Whisper checkpoints that accept short audio chunks (10‑30 s) instead of the original fixed 30 s window.
High‑performance inference engines for two major hardware families:
- NVIDIA GPUs (RTX 4090, L40s, H100, A100, Jetson‑Thor) via a custom TheStage AI “ElasticModels” engine and also via standard Hugging Face Transformers.
- Apple Silicon / macOS via CoreML (and MLX) engines that run at ~2 W power and ~2 GB RAM.
Streaming APIs that let you feed audio continuously from a microphone or file and receive word‑level timestamps.
Simple Python wrappers (ASRPipeline, StreamingPipeline) plus example front‑ends written in JavaScript/Electron.
Pre‑built desktop app TheNotes for macOS that demonstrates real‑time transcription.
Benchmarks (latency, memory, power, WER) compared to vanilla Whisper and posted in the benchmark/ folder.

The repository is geared toward developers who want to self‑host a low‑latency, low‑power ASR service on‑premise, in the cloud, or on‑device.

🛠️ Core Components

Component	What it does	Key tech
Model weights	Fine‑tuned Whisper‑large‑v3 (and a “turbo” variant) with flexible chunk sizes.	Hugging Face 🤗 hub
Inference engines	Optimized runtime for NVIDIA (TensorRT‑based ElasticModels) and Apple Silicon (CoreML/MLX).	TheStage AI, NVIDIA, Apple
Python API	`ASRPipeline` for single‑file transcription; `StreamingPipeline` for continuous audio.	PyTorch, Torch‑Audio, custom wrappers
Streaming utilities	`MicStream`, `FileStream`, `StdoutStream` to feed audio chunks and emit results.	Pure Python
Front‑end demos	Electron app and React example showing live captions.	Electron, ReactJS

🚀 Getting Started (Typical Flow)

Install – Choose the hardware target:

# Apple Silicon
pip install .[apple]

# NVIDIA GPU
pip install .[nvidia]
# (optional) TheStage AI optimized engine
pip install 'thestage-elastic-models[nvidia]==0.2.4.post0' ...

Configure – For TheStage AI engines, obtain an API token from the TheStage AI platform and run:
```
thestage config set -t <YOUR_API_TOKEN>
```

Run a quick transcription (single file):

from thestage_speechkit.nvidia import ASRPipeline
model = ASRPipeline(model='TheStageAI/thewhisper-large-v3-turbo',
                    chunk_length_s=10, device='cuda')
result = model('speech.wav')
print(result['text'])

Streaming example – Capture microphone audio and print live captions:

from thestage_speechkit.streaming import StreamingPipeline, MicStream, StdoutStream
pipe = StreamingPipeline(model='TheStageAI/thewhisper-large-v3-turbo',
                         chunk_length_s=10, platform='apple')
mic = MicStream(step_size_s=0.5)
out = StdoutStream()
while (chunk := mic.next_chunk()) is not None:
    txt, _ = pipe(chunk)
    out.write(txt, None)

Desktop app – Download the pre‑built TheNotes package for macOS or follow the React/Electron tutorial to build your own.

📊 Performance Highlights

NVIDIA L40s: ~220 tokens / second for the whisper-large-v3 model (≈ 0.5 s latency on a 10 s chunk).
Apple Silicon: ~2 W power draw, ~2 GB RAM, real‑time streaming on M2/M3 chips.
Accuracy: Evaluated on the Open ASR multilingual leaderboard; WER comparable to vanilla Whisper while offering lower latency.
Benchmarks for different chunk sizes (10 s, 15 s, 20 s, 30 s) are in benchmark/README.md.

📦 What’s Inside the Repo

thestage_speechkit/ – Python package with the Apple and NVIDIA pipelines, VAD, and streaming helpers.
electron_app/ – Source for the Electron front‑end and build instructions.
benchmark/ – Scripts and tables summarising latency, memory, power, and WER.
README.md – This documentation, quick‑start, and usage examples.
Model cards on Hugging Face for thewhisper-large-v3 and thewhisper-large-v3-turbo.

🛡️ License & Commercial Use

Code is released under the MIT License (see LICENSE).
Model weights are open‑source on Hugging Face.
For enterprises needing more than 4 GPUs per year of TheStage AI‑optimized engines, a commercial license can be requested via the TheStage AI contact page.

🎯 Who Should Use This

Developers building real‑time captioning, voice assistants, or meeting transcription services.
Edge/IoT teams deploying ASR on Jetson devices or Apple Silicon laptops.
Enterprises that want a self‑hosted alternative to cloud STT APIs with predictable latency and cost.

📚 Further Reading & Resources

Model hub: https://huggingface.co/TheStageAI/thewhisper-large-v3-turbo/
TheStage AI ElasticModels docs: https://docs.thestage.ai/elastic_models/docs/source/index.html
Tutorial (macOS note‑taker): https://app.thestage.ai/blog/Building-a-macOS-Note-Taker-app-on-Electron-with-TheWhisper?id=6
Open ASR Leaderboard: https://github.com/huggingface/open_asr_leaderboard

TheWhisper is a practical, production‑ready speech‑to‑text stack that bridges the gap between research‑grade Whisper models and the performance constraints of real‑world hardware.