jasoncheng7115/jt-live-whisper

jt‑live‑whisper – 本地端 AI 語音工具箱

What it does – A single‑machine (or machine + GPU‑server) application that captures system audio, runs speech‑to‑text, optional speaker diarisation, translates the text, and can summarise meetings – all with models that run on the user’s own hardware. No cloud APIs are required.

---

Core capabilities

Feature	How it works (as described in the README)
Real‑time transcription & translation	Captures system audio via ScreenCaptureKit (macOS), WASAPI Loopback (Windows) or PipeWire/PulseAudio monitor (Linux). The audio is fed to an on‑device ASR model (Whisper, Breeze‑ASR‑26, Moonshine, Qwen3‑ASR) and the resulting text is translated either by a local LLM (via Ollama, LM Studio, etc.) or by an offline NLLB/Argos model.
Batch processing of audio files	Supports mp3, wav, m4a, flac. Uses faster‑whisper (or other engines) for offline transcription and translation.
Speaker diarisation	Uses either resemblyzer + spectralcluster or NVIDIA’s Nemotron 3 Diarization model (default from v2.26.0).
Meeting summarisation	Sends the transcript to a locally hosted LLM (default qwen3.8:27b) which extracts decisions, action items, risks, timeline, speaker stats, etc.
Multi‑language support	ASR: Chinese, Japanese, Korean, English, Taiwanese (Breeze‑ASR‑26). Translation: Chinese↔Japanese↔Korean↔English via NLLB 600M; English↔Chinese via Argos as fallback.
Bidirectional subtitle mode	Simultaneously translates the remote speaker’s language to Chinese and the local speaker’s Chinese to the remote language (en‑zh, ja‑zh, ko‑zh).
Overlay & forwarding	Optional semi‑transparent subtitle overlay (PyQt6), keyword‑triggered alerts, and forwarding of live subtitles to Telegram, Slack, Discord, Teams, LINE, etc.
Web UI	A browser‑based interface (./start.sh --webui) that exposes all functions, shows progress, and lets you run the tool from phones or tablets.

---

Architecture & model stack

• ASR engines: whisper.cpp (ggml, macOS), faster‑whisper (CTranslate2, Windows/Linux, GPU), mlx‑whisper (Apple Silicon), Moonshine (ultra‑low‑latency English), vLLM / mlx‑audio / transformers for experimental Qwen3‑ASR.
• Translation: Local LLM server (default gemma4:26b for translation, qwen3.8:27b for summarisation) accessed through Ollama, LM Studio, vLLM, etc.; fallback to offline NLLB 600M or Argos.
• Diarisation: resemblyzer + spectralcluster (open source) or NVIDIA Nemotron 3 Diarization (OpenMDW‑1.1, no token required).
• Deployment modes:
  1. Single‑machine – everything runs on the same PC (CPU, Apple‑Silicon GPU, or NVIDIA GPU).
  2. Client + GPU‑server – audio capture on the laptop, heavy ASR/diarisation on a separate LAN GPU server.
  3. Headless server – Linux box exposing a Web UI; other devices upload audio for processing.

---

Getting started (summary of the README steps)

1. One‑click install – run the provided install.sh (macOS/Linux) or install.ps1 (Windows). The script creates a Python 3.12 virtual‑env, installs system dependencies, downloads the required models (≈ 3 GB minimal, ≈ 8 GB recommended).
2. Audio source setup – macOS uses ScreenCaptureKit (no driver needed) or optional BlackHole; Windows uses WASAPI Loopback; Linux uses the default monitor source.
3. Optional LLM server – install Ollama (or any compatible local LLM server) and pull gemma4:26b (translation) and qwen3.8:27b (summarisation).
4. Run – ./start.sh (or PowerShell equivalent) brings up an interactive menu, or launch the Web UI with --webui.
5. Choose mode – real‑time, batch, diarisation, summarisation, bidirectional subtitles, etc.

---

Who might use it?

• Privacy‑conscious professionals who need live subtitles or transcripts of meetings without sending data to cloud services.
• Researchers or developers experimenting with on‑device LLM‑driven translation and summarisation.
• Multilingual teams that want instant subtitles for Zoom/Teams/YouTube while keeping everything local.
• Anyone with a modern Mac (Apple Silicon), Windows PC with NVIDIA GPU, or Linux workstation who can allocate a few gigabytes of storage for models.

---

Limitations & notes

• Real‑time translation speed depends on the hardware and the chosen model; the fastest path on Apple Silicon is ~1‑3 s latency, on NVIDIA GPU ~0.3‑0.5 s.
• Summarisation requires a fairly large LLM (≥ 27 B parameters) and therefore a capable GPU or a dedicated server.
• Some features (e.g., Moonshine, Qwen3‑ASR) are marked experimental and may not be available on all platforms.
• The tool does not work on headless Linux for real‑time capture – you need a desktop session for that.

---

Bottom line

jt‑live‑whisper is a genuine, open‑source project that bundles state‑of‑the‑art speech‑to‑text, translation, diarisation, and meeting‑summary models into a fully offline workflow. It targets users who need accurate, low‑latency subtitles and transcripts while keeping data on‑premise.