Fine-Tuning Wav2Vec2 for English ASR with Hugging Face Transformers

Overview

Hugging Face has demonstrated that the Wav2Vec2 model can be fine-tuned for English Automatic Speech Recognition (ASR) using very small amounts of labeled data to achieve competitive results. By utilizing a pretrained checkpoint and fine-tuning it with Connectionist Temporal Classification (CTC), developers can build an end-to-end ASR system without the immediate need for a separate language model.

Wav2Vec2 Architecture and Pretraining

Wav2Vec2 is a pretrained model designed for ASR that learns speech representations from raw audio. It was released in September 2020 by Alexei Baevski, Michael Auli, and Alex Conneau.

Contrastive Pretraining

Wav2Vec2 uses a contrastive pretraining objective to learn from unlabeled speech (over 50,000 hours). Similar to BERT's masked language modeling, the model randomly masks feature vectors before passing them to a transformer network to learn contextualized speech representations.

Data Efficiency

Pretraining allows the model to reach high performance with minimal labeled data. For example, using as little as 10 minutes of labeled data, Wav2Vec2 can achieve a word error rate (WER) of less than 5% on the clean test set of LibriSpeech.

Fine-Tuning Process for English ASR

Fine-tuning Wav2Vec2 involves mapping pretrained context representations to a specific transcription vocabulary using a linear layer added on top of the transformer block.

Tokenization and Vocabulary

Because the output size of the final linear layer depends on the labeled dataset rather than the pretraining task, a custom vocabulary must be created. For the Timit dataset, this involves:

  • Normalizing text to lower-case.
  • Removing special characters (e.g., ,.?!;:) that do not correspond to distinct sound units.
  • Extracting all distinct letters from the training and test sets.
  • Adding a word delimiter token (|), an unknown token ([UNK]), and a padding token ([PAD]) which serves as the CTC "blank token."

Feature Extraction

Speech signals must be discretized via sampling. Wav2Vec2 expects input sampled at 16kHz, matching the sampling rate of LibriSpeech and LibriVox. The Wav2Vec2FeatureExtractor handles the raw speech signal, ensuring it is zero-mean-unit-variance normalized.

Data Preparation

Using the Wav2Vec2Processor, audio files are loaded and resampled to 16kHz. The processor handles both the feature extraction for the audio input and the tokenization for the target text labels.

Training with Connectionist Temporal Classification (CTC)

Wav2Vec2 is fine-tuned using CTC, an algorithm designed for sequence-to-sequence problems where the input length (audio frames) is significantly larger than the output length (text characters).

Training Configuration

To optimize training, the following technical configurations are employed:

  • Data Collator: A specialized DataCollatorCTCWithPadding is used to dynamically pad input_values and labels separately, as they belong to different modalities.
  • Frozen Feature Extractor: The CNN layers used for initial feature extraction are frozen (model.freeze_feature_extractor()) because they are sufficiently trained during pretraining.
  • Gradient Checkpointing: Enabled to save GPU memory.
  • Group by Length: Training samples of similar lengths are grouped together to reduce the number of padding tokens and increase efficiency.

Evaluation Metrics

The primary metric used is the Word Error Rate (WER). The model predicts logit vectors for each time step, and the argmax is taken to determine the most likely character. In CTC decoding, consecutive identical tokens are grouped, and blank tokens are used to separate identical characters in a word.

Results and Implications

In a demonstration fine-tuning the "base" Wav2Vec2 model on the Timit dataset (5 hours of training data), the model achieved a test WER of 22.1%.

Analysis of Errors

Predictions often show acoustic similarity to the target text but contain spelling or grammatical errors. This is expected when using a standalone acoustic model without a language model to constrain the output to valid linguistic patterns.

CTC Performance

The model demonstrates invariance to speaking rate. Because CTC allows the model to repeat the same token over multiple frames or insert blank tokens, the transcription remains consistent regardless of the length of the audio signal.

Sources