QuentinFuxa/WhisperLiveKit

Real-time, local speech-to-text with streaming ASR, speaker diarization, translation, and OpenAI/Deepgram-compatible APIs.

What it solves

WhisperLiveKit (WLK) provides an ultra-low-latency, self-hosted speech-to-text (STT) pipeline. It solves the problem of using models like Whisper for real-time transcription, as standard Whisper is designed for complete utterances rather than real-time chunks, which often leads to poor transcription and lost context when processed in small segments.

How it works

WLK implements state-of-the-art simultaneous speech research to handle intelligent buffering and incremental processing. It supports multiple backends including Whisper (via SimulStreaming and LocalAgreement policies), Voxtral, FunASR/SenseVoiceSmall, Qwen3-ASR, and NVIDIA Canary-1b-v2. The system uses Voice Activity Detection (VAD) to reduce overhead and can be deployed as a server with OpenAI-compatible REST APIs, Deepgram-compatible WebSockets, or native WebSockets for real-time streaming.

Who it’s for

It is designed for developers building real-time transcription services, applications requiring simultaneous translation across hundreds of languages, or those needing self-hosted, low-latency ASR (Automatic Speech Recognition) on various hardware including NVIDIA GPUs and Apple Silicon.

Highlights

  • Multi-Backend Support: Integrates Whisper, Voxtral, Qwen3-ASR, and NVIDIA Canary.
  • Ultra-Low Latency: Uses advanced policies like AlignAtt and LocalAgreement for real-time output.
  • Broad Language Support: Simultaneous translation to and from 200 languages via NLLW.
  • Hardware Optimized: Specific optimizations for Apple Silicon (MLX) and NVIDIA CUDA.
  • Flexible API: Offers OpenAI-compatible REST and WebSocket interfaces for easy integration.
  • Advanced Features: Supports real-time speaker diarization (via Sortformer) and causal streaming audio encoders.

Related

  • Project
    dimastatz/whisper-flow

    A real-time transcription service that enables streaming speech-to-text using OpenAI Whisper, providing low-latency results instead of batch processing.

  • Project
    KoljaB/RealtimeTTS

    A Python text-to-speech library designed for low-latency audio generation from strings, generators, and LLM token streams, supporting a wide variety of local and cloud TTS engines.

  • Project
    kadirnar/whisper-plus

    A comprehensive audio and video processing toolkit that extends OpenAI's Whisper for transcription, speaker diarization, summarization, and RAG-based video chatting.

  • Project
    ahmetoner/whisper-asr-webservice

    A general-purpose speech recognition toolkit that wraps Whisper models into a REST API for easy deployment and integration.

  • Project
    jasoncheng7115/jt-live-whisper

    A fully local AI toolkit for real-time speech transcription, translation, and meeting summarization that works across any system audio source without cloud APIs.