k2-fsa/sherpa-onnx

Speech-to-text, text-to-speech, speaker diarization, speech enhancement, source separation, and VAD using next-gen Kaldi with onnxruntime without Internet connection. Support embedded systems, Android, iOS, HarmonyOS, Raspberry Pi, RISC-V, RK NPU, Axera NPU, Ascend NPU, x86_64 servers, websocket server/client, support 12 programming languages

What it solves

Sherpa-onnx provides a highly portable and efficient way to run various speech and audio processing tasks locally. It eliminates the need for complex cloud dependencies by allowing developers to deploy speech-to-text, text-to-speech, and other audio analysis tools across a vast array of hardware platforms and programming languages.

How it works

The project leverages the ONNX Runtime to execute pre-trained models locally on the CPU, GPU, or specialized NPUs. It supports a wide range of audio functions including streaming and non-streaming ASR, TTS, speaker diarization, and voice activity detection (VAD). The system is designed for extreme cross-platform compatibility, providing APIs for over a dozen programming languages and supporting nearly every major operating system and processor architecture.

Who it’s for

Developers building audio-enabled applications who need local, offline execution on everything from high-end servers to mobile devices (Android, iOS), embedded systems (Raspberry Pi, Jetson), and web browsers via WebAssembly.

Highlights

  • Comprehensive Audio Suite: Supports ASR, TTS, speaker identification, diarization, audio tagging, and source separation.
  • Extreme Portability: Compatible with x86, ARM, RISC-V, and various NPUs (Rockchip, Qualcomm, Ascend, Axera).
  • Broad Language Support: Provides APIs for C++, C, Python, Go, C#, Java, Kotlin, JavaScript, Swift, Rust, Dart, and Pascal.
  • Local Execution: Designed for offline, local processing to ensure privacy and reduce latency.