jasoncheng7115/jt-live-whisper

What it solves

jt-live-whisper provides a completely local AI toolkit for real-time speech-to-text transcription and translation. It eliminates the need for cloud APIs, ensuring data privacy for sensitive meetings and removing monthly subscription costs. It solves the problem of language barriers during live audio events (like Zoom calls, YouTube videos, or Podcasts) and the tedious process of manually summarizing recorded meetings.

How it works

The tool captures system audio (using ScreenCaptureKit on macOS or WASAPI Loopback on Windows) and processes it through a pipeline of local AI models:

1. Speech Recognition (ASR): Uses OpenAI's Whisper (via whisper.cpp, faster-whisper, or mlx-whisper), Moonshine for ultra-low latency English, or Breeze-ASR-26 for Taiwanese Hokkien.
2. Translation: Employs local LLMs (via Ollama, LM Studio, etc.), Meta's NLLB 600M, or Argos Translate for offline translation between Chinese, English, and Japanese.
3. Speaker Diarization: Uses resemblyzer and spectralcluster to identify and distinguish different speakers by voice print.
4. Summarization: Uses local LLMs to generate meeting summaries and correct transcription errors.

It can run in a single-machine mode (using local CPU/GPU) or a hybrid mode where a local machine handles audio capture while a remote GPU server performs the heavy AI inference.

Who it’s for

• Professionals attending multilingual online meetings (Zoom, Teams, Meet) who need real-time subtitles.
• Users who want to transcribe and summarize recorded audio files privately without uploading data to the cloud.
• People learning languages or watching foreign content who need instant translation.
• Users with local GPU hardware (NVIDIA or Apple Silicon) who want to leverage their own compute for AI tasks.

Highlights

• 100% Local Execution: No cloud API keys or data uploads required.
• System-Wide Audio Capture: Works with any software outputting sound, not just specific apps.
• Multilingual Support: Specialized support for English, Chinese, Japanese, and Taiwanese Hokkien.
• Comprehensive Toolset: Includes real-time subtitles, offline batch processing, speaker identification, and AI-powered meeting summaries.
• Flexible Interfaces: Offers a terminal-based interactive menu, a full WebUI, and a semi-transparent floating subtitle overlay.
• Integration: Can forward real-time subtitles to platforms like Telegram, Slack, and Discord.