alesaccoia/VoiceStreamAI

Near-Realtime audio transcription using self-hosted Whisper and WebSocket in Python/JS

What it solves

VoiceStreamAI provides a way to perform near-real-time audio transcription by streaming audio from a web browser to a server. It addresses the challenge of processing continuous audio streams efficiently by using Voice Activity Detection (VAD) to ensure only actual speech is transcribed, reducing computational load and network bandwidth.

How it works

The system consists of a Python-based server and a JavaScript client that communicate via WebSockets. The client captures audio and sends it in chunks to the server. The server uses a modular pipeline to process this audio:

  1. Voice Activity Detection (VAD): Using models like Silero or Pyannote, the server identifies segments of the audio that contain speech.
  2. Transcription: The system uses Automatic Speech Recognition (ASR) models, defaulting to faster-whisper (a high-performance version of OpenAI's Whisper), to convert the detected speech into text.
  3. Buffering Strategy: To prevent words from being cut off at chunk boundaries, the system employs a "SilenceAtEndOfChunk" strategy, which waits for a pause in speech before triggering transcription.

Who it’s for

Developers looking to build applications that require real-time speech-to-text capabilities with a flexible, modular architecture that allows for swapping different VAD and ASR models.

Highlights

  • Modular Design: Uses factory and strategy patterns to easily integrate new VAD and ASR technologies.
  • Real-time Streaming: Leverages WebSockets for low-latency communication between the client and server.
  • Efficient Processing: VAD integration minimizes the processing of non-speech audio, saving GPU/CPU resources.
  • Customizable: Clients can dynamically configure language, chunk length, and processing strategies via JSON messages.

Related

  • Project
    dimastatz/whisper-flow

    A real-time transcription service that enables streaming speech-to-text using OpenAI Whisper, providing low-latency results instead of batch processing.

  • Project
    collabora/WhisperLive

    WhisperLive is an open‑source, near‑real‑time transcription server built around OpenAI’s Whisper model. It supports multiple inference back‑ends (faster‑whisper, NVIDIA TensorRT, Intel OpenVINO, AMD ROCm), offers word‑level timestamps, hot‑word boosting, speaker diarization and optional translation, and provides both simple command‑line clients and a Python streaming API. Docker images are provided for GPU and CPU deployments, and browser/iOS extensions let you transcribe audio directly from a web page or mobile device.

  • Project
    huggingface/speech-to-speech

    Speech‑to‑Speech is a Hugging Face open‑source pipeline that turns spoken input into spoken output. It chains VAD → STT → LLM → TTS, each component being swappable (Silero VAD, Parakeet/Faster‑Whisper/Whisper STT, any OpenAI‑compatible LLM or local Transformers/MLX model, Qwen3‑TTS or other TTS back‑ends). The whole system speaks the OpenAI Realtime protocol over WebSocket/WebRTC, so existing OpenAI Agents SDK code works. Install with a single `pip install speech-to-speech`, then run `speech-to-speech serve` (server) and `speech-to-speech talk` (client) or `speech-to-speech local` (both together). It supports local‑only operation, mixed local/hosted LLMs, Docker deployment, and optional extras for alternative TTS/STT models. Designed for voice‑assistant robots (e.g., Reachy Mini) and low‑latency spoken AI applications.

  • Project
    KoljaB/RealtimeSTT

    A Python speech-to-text library that simplifies the integration of voice activity detection, wake words, and fast transcription for voice assistants and dictation tools.

  • Project
    m-bain/whisperX

    A fast automatic speech recognition system that enhances OpenAI's Whisper with batched inference, word-level timestamps, and speaker diarization.