collabora/WhisperLive

A nearly-live implementation of OpenAI's Whisper.

What it solves

WhisperLive provides a nearly-live, real-time transcription system based on OpenAI's Whisper model. It solves the challenge of converting speech to text with minimal latency, allowing for live audio from microphones, RTSP/HLS streams, or pre-recorded files to be transcribed as they are spoken.

How it works

The project implements a client-server architecture. A server handles the heavy lifting of the AI model inference, while clients send audio data. It supports multiple high-performance backends for inference, including faster_whisper, tensorrt (NVIDIA), openvino (Intel), and AMD ROCm support via CTranslate2. The server can handle multiple concurrent clients and offers a batch inference mode to increase GPU throughput.

Who it’s for

This tool is designed for developers building real-time speech-to-text applications, such as live captioning, voice-controlled interfaces, or streaming transcription services. It also provides browser extensions and an iOS client for end-users.

Highlights

  • Multi-Backend Support: Optimized for NVIDIA GPUs (TensorRT), Intel CPUs/GPUs (OpenVINO), and AMD GPUs (ROCm).
  • Advanced Transcription Features: Includes word-level timestamps, custom vocabulary boosting (hotwords), and speaker diarization using pyannote.audio.
  • Flexible Input Sources: Supports live microphone input, audio files, and network streams (RTSP/HLS).
  • Streaming Client: A dedicated StreamingTranscriptionClient for pushing raw PCM audio bytes from any custom source.
  • Deployment Options: Available as Docker containers for various hardware configurations.

Related

  • Project
    OpenWhispr/openwhispr

    A privacy-first, open-source voice-to-text application providing local AI dictation, meeting transcription with speaker diarization, and AI-powered notes.

  • Project
    openai/whisper

    Whisper is OpenAI’s open‑source, transformer‑based speech‑to‑text system that handles transcription, language identification, and translation across many languages. It ships as a pip‑installable package with six model sizes (tiny → large, plus a fast “turbo” variant). The CLI (`whisper …`) and a simple Python API (`whisper.load_model(...).transcribe(...)`) let you turn audio files into text with just a few commands. The project runs on Python 3.8‑3.11, PyTorch, and requires ffmpeg (and optionally Rust for the tokenizer). All code and model weights are MIT‑licensed.

  • Project
    QuentinFuxa/WhisperLiveKit

    An ultra-low-latency, self-hosted speech-to-text pipeline that enables real-time transcription and translation using advanced simultaneous speech research and multiple model backends.

  • Project
    ggml-org/whisper.cpp

    A high-performance C/C++ port of OpenAI's Whisper model for efficient, offline, on-device speech-to-text transcription across diverse hardware.

  • Dispatch
    Hugging Face Inference Endpoints: Fast Whisper Transcriptions

    Hugging Face has introduced a new OpenAI Whisper deployment option on Inference Endpoints that delivers up to 8x performance improvements in transcription speed without sacrificing accuracy.