Picovoice/speech-to-text-benchmark

What it solves

This project provides a standardized, extensible framework for evaluating the performance of various speech-to-text (STT) engines. It addresses the difficulty of comparing different models and cloud services by providing a consistent set of datasets and metrics to measure accuracy, computational efficiency, and latency.

How it works

The framework uses a collection of open-source speech datasets (such as LibriSpeech, Common Voice, and TED-LIUM) to test a wide range of STT engines, including cloud-based services (AWS, Azure, Google, IBM) and local models (Whisper, Vosk, Moonshine, Nemotron, and Picovoice). It calculates performance based on specific metrics and provides scripts to run these benchmarks locally.

Who it’s for

This tool is designed for developers, researchers, and engineers who need to select an STT engine for their applications and require objective, data-driven comparisons of accuracy, speed, and resource usage.

Highlights

• Comprehensive Metrics: Evaluates Word Error Rate (WER), Character Error Rate (CER), Punctuation Error Rate (PER), computational efficiency (Core-Hour), and streaming latency.
• Diverse Engine Support: Benchmarks both cloud-based APIs and local, self-hosted models.
• Extensible Framework: Supports multiple languages and a wide variety of standard speech datasets.
• Streaming & Batch Support: Includes specific testing protocols for both real-time streaming engines and batch processing.