QuentinFuxa/WhisperLiveKit

Simultaneous speech-to-text models

What it solves

WhisperLiveKit provides an ultra-low-latency, self-hosted speech-to-text (STT) pipeline. It solves the problem of using Whisper models for real-time transcription, as standard Whisper is designed for complete utterances and often produces poor results or cuts off words when processing small, real-time audio chunks.

How it works

The project implements state-of-the-art simultaneous speech research, including AlignAtt SimulStreaming and LocalAgreement policies, to handle intelligent buffering and incremental processing. It supports multiple backends including Faster-Whisper, MLX (for Apple Silicon), Voxtral Mini, and Qwen3-ASR. To reduce overhead, it incorporates Voice Activity Detection (VAD) to ignore silence. The system can be deployed as a server with OpenAI-compatible REST APIs, Deepgram-compatible WebSockets, or a native WebSocket for streaming.

Who it’s for

It is designed for developers building real-time transcription services, applications requiring simultaneous translation across 200 languages, or those needing self-hosted, low-latency ASR (Automatic Speech Recognition) with speaker diarization.

Highlights

  • Ultra-low latency: Uses advanced streaming policies to provide real-time transcription.
  • Multiple Backends: Supports a wide range of models including Whisper, Voxtral Mini (4B parameters), and Qwen3-ASR.
  • Broad API Compatibility: Drop-in replacement for OpenAI and Deepgram APIs.
  • Multilingual Translation: Simultaneous translation to and from 200 languages via NLLW.
  • Speaker Diarization: Real-time speaker identification using Sortformer or Diart.
  • Hardware Optimized: Dedicated support for NVIDIA GPUs (CUDA) and Apple Silicon (MLX/Metal).