Picovoice/speech-to-text-benchmark

speech to text benchmark framework

What it solves

It provides a standardized, minimalist, and extensible framework for comparing the performance of various speech-to-text (STT) engines. It allows developers to evaluate different models based on accuracy, speed, and resource consumption using consistent datasets and metrics.

How it works

The framework runs selected STT engines against a variety of public datasets (such as LibriSpeech, Common Voice, and TED-LIUM) and calculates specific performance indicators:

  • Accuracy: Measured via Word Error Rate (WER) and Punctuation Error Rate (PER).
  • Efficiency: Measured by "Core-Hour" (CPU hours needed to process one hour of audio) and total model size in MB.
  • Responsiveness: Measured by Word Emission Latency for streaming engines, tracking the delay between a word being spoken and its transcription.

It supports both batch and streaming modes across multiple languages including English, French, German, Spanish, Italian, and Portuguese.

Who it’s for

Developers and researchers who need to choose the most suitable speech-to-text engine for their specific application based on a trade-off between accuracy, latency, and computational cost.

Highlights

  • Broad Engine Support: Benchmarks a wide range of providers including OpenAI Whisper, Google, Amazon, Azure, IBM Watson, Vosk, Moonshine, and Picovoice.
  • Multi-Metric Evaluation: Goes beyond simple accuracy to include punctuation errors, CPU efficiency, and real-time latency.
  • Diverse Datasets: Integrates multiple industry-standard datasets for robust testing.
  • Extensible Design: Built as a framework that can be extended to include new engines or datasets.