Fine-Tuning Whisper for Multilingual ASR with Hugging Face Transformers

Hugging Face has released a technical guide for fine-tuning OpenAI's Whisper model for multilingual automatic speech recognition (ASR). By leveraging the 🤗 Transformers library, developers can adapt Whisper's pre-trained multilingual knowledge to low-resource languages, as demonstrated by a fine-tuning run on Hindi that reduced the Word Error Rate (WER) from 63.5% to 32.0% using only 8 hours of training data.

Whisper Model Architecture and Pre-training

Whisper is a Transformer-based encoder-decoder (sequence-to-sequence) model designed to map audio spectrogram features to text tokens. It differs from previous models like Wav2Vec 2.0 by being pre-trained on a massive volume of labelled audio-transcription data—680,000 hours in total, including 117,000 hours of multilingual ASR data.

Key Technical Characteristics

  • Supervised Pre-training: Because it is trained directly on speech-to-text mappings, Whisper requires significantly less fine-tuning than models pre-trained on unsupervised tasks (such as masked prediction).
  • Deep Fusion: The model incorporates the language model internally within the decoder, allowing the entire system to be trained end-to-end with a single loss function (cross-entropy).
  • Input Processing: Raw audio is converted into log-Mel spectrograms. The encoder processes these spectrograms into hidden states, which the decoder then uses to autoregressively predict text tokens.

Model Configurations

Whisper is available in several sizes to balance performance and computational cost:

Size Parameters English-only Multilingual
tiny 39 M Yes Yes
base 74 M Yes Yes
small 244 M Yes Yes
medium 769 M Yes Yes
large (v1, v2, v3) 1550 M No Yes

The Fine-Tuning Pipeline

Fine-tuning Whisper involves a three-part pipeline: a feature extractor for audio pre-processing, the sequence-to-sequence model, and a tokenizer for text post-processing.

Audio Pre-processing and Feature Extraction

Whisper requires audio inputs to be sampled at 16kHz. The WhisperFeatureExtractor performs two critical operations:

  1. Padding/Truncation: All audio samples are standardized to a length of 30 seconds. Shorter samples are padded with zeros; longer samples are truncated.
  2. Spectrogram Conversion: Padded audio arrays are converted into log-Mel spectrograms, which represent the frequencies of the signal over time.

Tokenization and Labeling

The WhisperTokenizer uses a pre-trained byte-pair encoding (BPE) vocabulary covering 96 languages. For fine-tuning, the tokenizer is configured with a specific target language and task (e.g., language="Hindi", task="transcribe"), which prepends the necessary special tokens to the label sequences.

Implementation Details for Low-Resource Languages

Using the Common Voice 11.0 dataset for Hindi, the guide outlines a specific workflow for adapting the whisper-small checkpoint.

Data Preparation

  • Resampling: Since Common Voice audio is often sampled at 48kHz, the datasets.Audio cast_column method is used to resample audio to 16kHz on the fly.
  • Mapping: The prepare_dataset function computes log-Mel input features and encodes target transcriptions into label IDs.

Training Configuration

  • Data Collator: A custom DataCollatorSpeechSeq2SeqWithPadding is used to handle input_features (fixed-dimension tensors) and labels (variable-length sequences padded with -100 to ignore them during loss calculation).
  • Evaluation Metric: Performance is measured using Word Error Rate (WER), the industry standard for ASR.
  • Hyperparameters: The demonstration used a learning rate of 1e-5, fp16 precision, and a maximum of 5,000 training steps.

Performance Results and Implications

Fine-tuning the whisper-small model on approximately 8 hours of Hindi data yielded a significant performance boost over the zero-shot pre-trained model.

  • Pre-trained WER: 63.5%
  • Fine-tuned WER: 32.0%
  • Absolute Improvement: 31.5%

This result demonstrates that Whisper's extensive pre-training allows it to generalize effectively to low-resource languages with minimal additional data. To further improve results, the guide suggests optimizing hyperparameters (learning rate, dropout) or utilizing larger checkpoints such as medium or large-v3.

Sources