Hugging Face Audio Datasets Guide – How to Load, Process, and Stream Audio Data

TL;DR

Hugging Face released a step‑by‑step guide demonstrating that the 🤗 Datasets library can download, preprocess, and stream any audio dataset from the Hub with a single load_dataset call, making large‑scale speech and audio research practical and reproducible.

The Hub is a One‑Stop Shop for Audio Data

The Hugging Face Hub hosts 77 speech‑recognition and 28 audio‑classification datasets (numbers as of Dec 2022). Each dataset card provides a preview with playable audio samples, licensing information, and links to models trained on the data. Users can filter datasets by task category directly on the Hub.

"The Dataset Preview is a brilliant way of experiencing audio datasets before committing to using them."

One‑Line Loading with load_dataset

The core API is datasets.load_dataset. Providing the Hub identifier (e.g., speechcolab/gigaspeech) and an optional configuration (e.g., "xs" for the 10‑hour split) returns a DatasetDict containing train, validation, and test splits.

from datasets import load_dataset

gigaspeech = load_dataset("speechcolab/gigaspeech", "xs")
print(gigaspeech)

The output shows the number of rows per split and the full feature list (e.g., segment_id, speaker, text, audio, …). Columns not needed for a task can be dropped with remove_columns.

Minimal Pre‑Processing Pipeline

Three generic steps cover most speech‑recognition workflows:

  1. Resampling – Use cast_column with datasets.Audio(sampling_rate=…) to change the sampling rate on‑the‑fly.
  2. Feature Extraction – Apply a map function that runs a transformers.AutoProcessor (e.g., Whisper) on each sample, converting raw waveforms to model inputs and tokenizing transcriptions.
  3. Filtering – Use filter to drop samples that exceed a duration threshold (e.g., 30 s) and avoid OOM errors.

All three steps together required only 13 lines of Python in the tutorial.

Streaming Mode Removes Disk‑Space Barriers

Large configurations (e.g., GigaSpeech xl with 10 000 h ≈ 1 TB) are impractical to store locally. Setting streaming=True loads samples lazily, downloading and processing each item only when iterated over.

gigaspeech = load_dataset("speechcolab/gigaspeech", "xs", streaming=True)

Advantages of streaming:

  • Zero disk footprint – data lives only in memory during iteration.
  • Immediate availability – start training as soon as the first sample is fetched.
  • Rapid prototyping – experiment on a handful of samples without a full download.

The trade‑off is that streamed data is not cached; repeated runs re‑download and re‑process each sample.

Quick Tour of Popular Audio Datasets

The guide lists the most widely used datasets for three task families, each accessible with the same load_dataset pattern.

English Speech Recognition

Dataset Domain Hours Casing Punctuation License
LibriSpeech Audiobook 960 ❌ ❌ CC‑BY‑4.0
Common Voice 11 Wikipedia 2 300 ✅ ✅ CC0‑1.0
VoxPopuli European Parliament 540 ❌ ✅ CC0
TED‑LIUM TED talks 450 ❌ ❌ CC‑BY‑NC‑ND 3.0
GigaSpeech Audiobook, podcast, YouTube 10 000 ❌ ✅ Apache‑2.0
SPGISpeech Financial earnings calls 5 000 ✅ ✅ User Agreement
Earnings‑22 Financial earnings calls 119 ✅ ✅ CC‑BY‑SA‑4.0
AMI Meetings 100 ✅ ✅ CC‑BY‑4.0

Multilingual Speech Recognition

  • Multilingual LibriSpeech – eight high‑resource languages.
  • Common Voice – >100 languages, crowd‑sourced.
  • VoxPopuli – 15 European languages.
  • FLEURS – 102 languages, ~10 h per language.

Speech Translation

  • CoVoST 2 – 2 900 h covering 21→EN and EN→15 language pairs.
  • FLEURS – parallel speech‑translation data for 101 language pairs.

Audio Classification

  • SpeechCommands – 1‑s keyword utterances for on‑device keyword spotting.
  • Multilingual Spoken Words – 23.4 h across 50 languages, 340 k keywords.
  • FLEURS – language‑identification labels for 102 languages.

Closing Remarks

The guide proves that 🤗 Datasets is the go‑to library for audio research: a single line loads any dataset, built‑in audio features simplify resampling, and streaming makes massive corpora tractable. The accompanying Colab notebook shows how to evaluate Whisper on eight English ASR datasets in one script, illustrating the practical benefits of the workflow.

Contributions to the blog post were made by Vaibhav Srivastav, Polina Kazakova, Patrick von Platen, Omar Sanseviero, and Quentin Lhoest.

Sources