moonshine: a low-latency on-device voice toolkit for real-time conversational agents

What it solves

Moonshine Voice is designed for developers building real-time voice agents and applications. It addresses the high latency and resource inefficiency of traditional speech-to-text (STT) models like OpenAI's Whisper, which often require large input windows and lack caching for streaming audio. Moonshine provides a fast, private, on-device solution that allows for responsive, live voice interfaces without needing cloud APIs or external accounts.

How it works

Moonshine uses a portable C++ core library powered by OnnxRuntime for high performance across various platforms. It features a family of models trained from scratch with flexible input windows and caching for streaming, which allows the model to process audio incrementally as the user speaks. The toolkit integrates multiple stages of a voice interface—microphone capture, voice activity detection, speech-to-text, speaker identification, and intent recognition—into a single library. It also provides high-level APIs for text-to-speech (TTS) and conversational agents.

Who it’s for

Developers building voice-driven applications across a wide range of platforms, including Python, iOS, Android, MacOS, Linux, Windows, Raspberry Pi, and even microcontrollers or wearables.

Highlights

• Low Latency: Optimized for live streaming with caching and flexible input windows to ensure responsiveness below 200ms.
• On-Device Processing: Runs entirely locally for privacy and speed, removing the need for API keys or credit cards.
• High Accuracy: The Medium Streaming model for English outperforms Whisper Large v3 in word-error rate (WER) while using significantly fewer parameters (250M vs 1.5B).
• C++ Core: A single portable core allows the same library to run across mobile, desktop, and IoT devices.
• Multilingual Support: Offers specialized models for multiple languages including English, Spanish, Mandarin, Japanese, Korean, Vietnamese, Ukrainian, and Arabic.
• Comprehensive Toolkit: Includes built-in support for transcription, TTS, voice cloning, and intent recognition.