Fine-Tune MMS Adapter Models for low-resource ASR

TL;DR

Hugging Face demonstrates that fine-tuning only the adapter layers of the MMS‑1B checkpoint yields strong Turkish speech recognition after just 10‑20 minutes of training on ~4 hours of Common Voice data, using far fewer trainable parameters than full‑model fine‑tuning.

Preserving the world’s language diversity

MMS supports transcription of over 1,100 languages, many of which are endangered, by using small adapter layers that act as linguistic bridges between languages.

Fine‑tuning MMS overview

The MMS‑1B checkpoint was pretrained on more than half a million hours of audio across over 1,400 languages and later fine‑tuned on 1,000+ languages with a joint vocabulary output layer, which was then replaced by language‑specific adapter layers containing roughly 2.5 M weights each. Three ASR‑oriented checkpoints are available: mms-1b-fl102, mms-1b-l1107, and mms-1b-all.

Training adaptive weights – why adapters help

Adapter layers insert small trainable modules between frozen transformer blocks, allowing the model to acquire language‑specific phonetic and grammatical traits without updating the bulk of the pretrained weights. This approach reduces memory usage, speeds up convergence, and avoids over‑fitting on low‑resource data.

Notebook setup – data and preprocessing

The notebook uses the Turkish split of Common Voice 6.1 (≈4 h of validated training audio). Audio is resampled to 16 kHz, special characters are removed, text is lower‑cased, and Turkish‑specific diacritics are normalized. A vocabulary of 37 tokens (including space, punctuation, [UNK], and [PAD]) is built from the transcriptions and stored under the ISO‑639‑3 code tur. A Wav2Vec2Processor combines a Wav2Vec2FeatureExtractor (feature_size=1, sampling_rate=16000, padding_value=0.0, do_normalize=True, return_attention_mask=True) with the custom tokenizer.

Training process and results

Training employs a custom DataCollatorCTCWithPadding that pads input values and labels separately, masks label padding with ‑100, and uses the word error rate (WER) as the evaluation metric. The model is loaded from facebook/mms-1b-all, dropout layers are disabled, the vocabulary size is set to 37, and mismatched sizes are ignored (the newly initialized LM head matches the new vocabulary). Adapter layers are (re)initialized, the base model is frozen, and only adapter weights receive gradients. Training arguments: group_by_length=True, per_device_train_batch_size=32, num_train_epochs=4, learning_rate=1e-3, gradient_checkpointing=True, fp16=True, save_steps=200, eval_steps=100, logging_steps=100, push_to_hub=True. After 100 training steps the validation WER was 0.280; after 200 steps it dropped to 0.232; after 300 steps to 0.229; and after 400 steps to 0.223. The training loss decreased from 4.905 to 0.2398 over the same span. These results show that adapter‑only fine‑tuning outperforms full‑model fine‑tuning of the smaller XLS‑R‑300M checkpoint on the same low‑resource Turkish data, while being more memory‑efficient.

Loading and using the fine‑tuned adapter

After training, the adapter weights are saved as adapter.tur.safetensors alongside the base model. To inference, load the checkpoint with target_lang="tur" and set the tokenizer’s target language accordingly:

model = Wav2Vec2ForCTC.from_pretrained("patrickvonplaten/wav2vec2-large-mms-1b-turkish-colab", target_lang="tur\)).to("cuda")
processor = Wav2Vec2Processor.from_pretrained("patrickvonplaten/wav2vec2-large-mms-1b-turkish-colab")
processor.tokenizer.set_target_lang("tur" )

A forward pass on a Turkish test sample yields a prediction that matches the reference transcription up to minor spacing differences. The same base model can load a Swedish adapter (adapter.swe.safetensors) by calling `model.load_adapter("swe\