Sharrnah/whispering

What it solves

Whispering Tiger is a tool for real-time transcription and translation of audio streams or in-game text. It allows users to capture audio from a microphone or system audio, as well as text from screen images (OCR), and output the results to a web browser via Websockets or over OSC for use in streaming overlays or applications like VRChat.

How it works

The software runs entirely locally on the user's machine. It integrates multiple AI models for different tasks:

• Speech-to-Text & Translation: Uses models like OpenAI's Whisper, Meta's Seamless M4T, and Microsoft's Phi-4 Multimodal LLM to transcribe and translate spoken language.
• Text Translation: Employs NLLB-200, M2M-100, and others for high-accuracy text-to-text translation.
• OCR: Uses EasyOCR, GOT-OCR 2.0, and Phi-4 to capture and translate text from images/games.
• Text-to-Speech (TTS): Converts transcriptions or translations into spoken audio using Silero, Kokoro, or Zonos TTS.
• Voice Processing: Includes Voice Activity Detection (VAD) and Retrieval-based Voice Conversion (RVC).

Who it’s for

It is primarily designed for streamers, gamers (specifically VRChat users), and anyone needing live, local translation and transcription of system audio or screen content.

Highlights

• 100% Local Execution: No internet connection is required once models are downloaded.
• Multimodal Capabilities: Combines speech recognition, OCR, and LLM-based question answering in one tool.
• Extensive Language Support: Supports a vast array of languages across different models (e.g., NLLB-200 supports 200 languages).
• Flexible Output: Sends data to web browsers via Websockets or OSC for integration into overlays.