oboroge0/hayamimi

What it solves

hayamimi provides real-time, multilingual speech-to-text (STT) that runs entirely on a CPU with minimal memory usage (under 2GB RAM). It solves the problem of high latency and high hardware requirements typically associated with high-accuracy transcription, allowing for live subtitles and translation without needing a GPU or cloud APIs.

How it works

Instead of relying on a single general-purpose model, hayamimi uses a routing system that directs each utterance to a specialized, quantized (INT8) ONNX model best suited for that specific language. It utilizes sherpa-onnx for inference, avoiding PyTorch and CUDA. The pipeline includes:

• Language Identification (LID): Uses whisper-tiny to detect the language of the incoming audio.
• Specialized Routing: Routes audio to dedicated models like ReazonSpeech (Japanese), Paraformer (Chinese), SenseVoice (Korean/Cantonese), or Parakeet (English and 24 EU languages).
• Two-Pass Refinement: After a period of silence, recent utterances are re-decoded for higher accuracy.
• Post-Processing: Includes speaker labeling (via CAM++ and pyannote), CJK number normalization, and live translation (via FuguMT or M2M-100).
• Input Flexibility: Supports microphone, WAV files, WebSocket network audio, and Windows-specific system audio loopback.

Who it’s for

• Streamers: Those needing low-latency OBS overlays for live captions.
• Developers: People wanting to embed a lightweight, private STT engine into other applications via its EventHub and API.
• Meeting Transcribers: Users who need to transcribe both their own voice and system audio (calls/videos) simultaneously.

Highlights

• Ultra-low Latency: Finalized lines typically appear ~100ms after speech ends.
• CPU-Optimized: Runs 10-50x real-time on a 6-core desktop CPU without a GPU.
• High Accuracy: Achieves 3.8% CER for Japanese broadcast audio, significantly outperforming whisper-large-v3-turbo in that specific context.
• Memory Efficient: Uses an LRU cache to keep resident models under a configurable cap (default <2GB).
• Integrated Tooling: Comes with a built-in browser dashboard and OBS-ready overlay.