Fine-Tuning Wav2Vec2-BERT for Low-Resource ASR

Meta's Wav2Vec2-BERT is a versatile audio model that can be fine-tuned for Automatic Speech Recognition (ASR) in low-resource languages, offering a high-performance alternative to autoregressive models like Whisper. By utilizing Connectionist Temporal Classification (CTC), Wav2Vec2-BERT can achieve Word Error Rate (WER) performance similar to Whisper-large-v3 on resource-poor languages while operating 10x to 30x faster and being 2.5x more resource-efficient.

Wav2Vec2-BERT Architecture and Pre-training

Wav2Vec2-BERT is a 580M-parameter model developed as a building block for Meta's Seamless Communication family of AI translation models. It represents a significant evolution of the original Wav2Vec2 model, which first demonstrated low-resource transfer learning for ASR.

Key pre-training statistics include:

  • Data Volume: Pre-trained on 4.5 million hours of unlabeled audio data.
  • Language Coverage: Covers more than 143 languages.
  • Comparison: This scale is significantly larger than XLS-R (half a million hours in 128 languages) and MMS (half a million hours in over 1,400 languages).

Advantages Over Autoregressive Models

While Whisper is widely considered state-of-the-art for ASR, it faces specific limitations in "resource-poor" languages (such as Mongolian or Malayalam), where it has historically achieved over 100% WER. Additionally, Whisper's autoregressive nature makes it inherently slower, especially for languages infrequent in its training set, as it must generate more tokens per word.

In contrast, Wav2Vec2-BERT predicts ASR in a single pass. This non-autoregressive approach provides three primary benefits:

  1. Inference Speed: 10x to 30x faster than Whisper-large-v3.
  2. Resource Efficiency: 2.5x more resource-efficient.
  3. Adaptability: Easily adaptable to any alphabet and requires minimal data to achieve competitive performance.

Fine-Tuning Pipeline for Low-Resource ASR

To adapt the facebook/w2v-bert-2.0 checkpoint for a specific language, such as Mongolian using the Common Voice 16.0 dataset, the following pipeline is employed:

Data Pre-processing

  • Text Normalization: Transcriptions are converted to lowercase and stripped of special characters (e.g., punctuation) that do not correspond to acoustic sound units.
  • Vocabulary Construction: A vocabulary is built from distinct letters in the training and test sets. Redundant characters (such as Latin characters in a Mongolian dataset) are removed to reduce vocabulary size, which benefits the CTC algorithm.
  • Special Tokens: The vocabulary includes a word delimiter token (|), an unknown token ([UNK]), and a padding/blank token ([PAD]) essential for CTC alignment.
  • Audio Processing: Raw audio is resampled to 16 kHz to match the pre-training distribution of the model.

Technical Implementation

  • Feature Extraction: The SeamlessM4TFeatureExtractor maps raw audio amplitude values to log-mel spectrograms.
  • Tokenizer: The Wav2Vec2CTCTokenizer handles the mapping between character tokens and label IDs.
  • Training Objective: The model is fine-tuned using Connectionist Temporal Classification (CTC), which allows the network to map input sequences to output sequences of different lengths without requiring explicit alignment.

Training Results and Scaling Tips

In a demonstration fine-tuning on Mongolian ASR with approximately 14 hours of validated training data, Wav2Vec2-BERT achieved a Word Error Rate (WER) competitive with Whisper-large-v3 (which achieved 33.3% WER on the same task).

Expert Tips for Scaling Training

  • Vocabulary Pruning: Remove very low-frequency characters to prevent loss spikes caused by erroneous targets in the dataset annotations.
  • Temporal Resolution: The ideal duration seen per CTC token is between 10ms and 35ms. If the signal chunk is too long (e.g., 30-60ms), the loss curve may explode. Adding a convolutional adapter layer to sub-sample encoder hidden-states can resolve this.
  • Preventing Under-training:
    • Maintain a warm-up ratio of 5% to 15% and increase the number of epochs to ensure the language-model head weights align with the pre-trained model.
    • Adjust AdamW's $\beta_{2}$ (typically between 0.95 and 0.98) to smooth the loss curve.

Sources