TheStageAI/TheWhisper

Optimized Whisper models for streaming and on-device use

What it solves

TheWhisper provides a high-performance, low-latency speech-to-text (STT) solution optimized for self-hosting and on-device inference. It addresses the limitation of original Whisper models that require 30-second audio chunks, allowing for flexible chunk sizes (10s, 15s, 20s, and 30s) without needing to pad audio with silence.

How it works

The project provides fine-tuned versions of Whisper models with open weights and specialized inference engines for different hardware:

  • NVIDIA GPUs: Uses optimized engines (including TheStage AI ElasticModels) to achieve high throughput, such as 220 tok/s on L40s GPUs.
  • Apple Silicon: Uses CoreML and MLX engines to minimize power consumption (2W) and RAM usage (2GB).
  • Streaming: Implements streaming transcription for both NVIDIA and macOS platforms to support real-time applications.

Who it’s for

It is designed for developers building real-time captioning, live meeting tools, voice interfaces, and edge deployments where low power and low latency are are required.

Highlights

  • Flexible Chunking: Supports 10s, 15s, 20s, and 30s audio chunks for more efficient processing.
  • Hardware Optimization: Dedicated high-performance engines for NVIDIA GPUs (RTX 4090/5090, L40s, H100, A100) and Apple Silicon (M1-M4 series).
  • On-Device Deployment: Includes a Python API and examples for building macOS desktop applications using Electron and ReactJS.
  • Streaming Support: Built-in support for streaming transcription across supported platforms.

Related

  • Project
    argmaxinc/argmax-oss-swift

    A collection of on-device inference frameworks for Apple platforms providing speech-to-text, text-to-speech, and speaker diarization using Core ML.

  • Project
    TypeWhisper/typewhisper-mac

    A system-wide speech-to-text and AI text processing utility for macOS that supports both local on-device models and cloud APIs for transcription and transformation.

  • Project
    collabora/WhisperLive

    A real-time transcription application that uses OpenAI's Whisper model to convert live speech input from microphones or streams into text with minimal latency.

  • Project
    soniqo/speech-swift

    Speech Swift is an open‑source Swift package that provides dozens of on‑device speech AI models (ASR, TTS, translation, voice agents, enhancement, etc.) for Apple Silicon. It runs via Core ML and the MLX framework, supports hundreds of languages, offers multiple quantised sizes, and can be added to a project with a single Swift‑Package‑Manager dependency. The repo includes extensive docs, pre‑trained models, and example code for transcription, streaming dictation, and UI components, all without any cloud calls.

  • Project
    jim60105/docker-whisperX

    A collection of optimized Docker images for WhisperX, providing pre-bundled ASR models for fast and efficient speech-to-text transcription with word-level timestamps.