dimastatz/whisper-flow

Whisper Flow – Real‑time transcription built on OpenAI Whisper

What it does – Whisper Flow wraps the OpenAI Whisper speech‑to‑text model in a thin FastAPI service that accepts a live audio stream over a WebSocket and returns partial‑and‑final transcription results as JSON. It turns Whisper, which normally works on whole files, into a low‑latency, streaming ASR engine.

How it works

• Clients open ws://<host>:8181/ws and send raw PCM audio (16 kHz, mono, int16) in binary frames.
• The server buffers the incoming chunks, groups them into temporal windows (tumbling windows) and runs Whisper on each segment.
• After each inference pass a JSON payload is pushed back:

  {"is_partial": true|false, "data": {"text": "…"}, "time": <ms>}
  

  is_partial indicates whether the transcript may still change; a final result is emitted when speech pauses, the window limit is hit, or the client sends a flush/stop control frame.
• A small HTTP endpoint (GET /ready) reports the protocol version and which Whisper model files are available.

Key components

Module	Role
whisperflow.transcriber	Loads a Whisper model (tiny.en.pt by default) and provides synchronous (transcribe_pcm_chunks) and async (transcribe_pcm_chunks_async) helpers.
whisperflow.streaming	Implements TranscribeSession, which manages per‑client state, buffers audio chunks, calls the transcriber, and sends JSON results back to the WebSocket.
FastAPI server	Exposes the WebSocket endpoint, a health check (/health), and a batch‑transcription POST (/transcribe_pcm_chunk).

Getting started

1. Install – Python 3.8+, PortAudio (for PyAudio), then run ./run.sh -local which creates a virtualenv, installs requirements.txt, formats, lints and runs the test suite.
2. Run – ./run.sh -run-server starts the service on port 8181. Verify with http://localhost:8181/health.
3. Use – Either connect a WebSocket client that streams PCM audio, or install the package (pip install whisperflow) and call the library directly as shown in the README example.

Performance – Benchmarks on an Apple M1 (16 GB RAM) using LibriSpeech report average latency ≈ 275 ms per partial result and a word‑error‑rate around 7 %, which satisfies the sub‑second real‑time goal.

Deployment options

• Docker – ./run.sh -docker builds an image (whisperflow-image) and runs a container exposing port 8181.
• Python package – The library can be imported into other projects, enabling custom WebSocket servers or integration with existing pipelines.

Roadmap – v1.3 aims to expose the package through the broader py‑speech ecosystem.

---

All details are taken directly from the repository’s README.