Fine-Tune MMS Adapter Models for Low-Resource ASR

TL;DR

MMS adapter fine‑tuning delivers dramatically lower word error rates for low‑resource languages after just 10‑20 minutes of training, and it uses far less memory than full‑model fine‑tuning.

Why MMS adapters matter for low‑resource ASR

Adapter training on the Massive Multilingual Speech (MMS) models outperforms full‑model fine‑tuning on languages with limited data. The approach is more memory‑efficient, converges in minutes, and preserves the base model (≈99 % of parameters) while only training ~2.5 M adapter weights per language.

Background: From wav2vec 2.0 to MMS

  • wav2vec 2.0 (Sept 2020) introduced self‑supervised speech pre‑training.
  • XLS‑R extended wav2vec 2.0 to 128 languages.
  • MMS (Meta AI, 2023) scales this to >1,100 languages, with checkpoints of 300 M and 1 B parameters trained on >500 k h of audio across 1,400 languages.

MMS releases three ASR checkpoints with language‑specific adapters:

  • mms-1b-fl102 (102 languages)
  • mms-1b-l1107 (1,107 languages)
  • mms-1b-all (all 1,162 languages)

Each adapter contains ~2.5 M trainable weights (small linear projections per attention block plus a language‑specific vocab layer).

Adapter training vs. full fine‑tuning

Aspect Adapter fine‑tuning Full model fine‑tuning
Parameters updated ~2.5 M per language (≈0.2 % of a 1 B model) All model weights (≈1 B)
Memory usage Significantly lower; fits on a single GPU for many languages High; often requires multi‑GPU or gradient checkpointing
Training time 10‑20 min for 4 h of Common Voice data (≈4 epochs) Hours to days for comparable performance
Performance on low‑resource data Better WER, more robust Worse WER, prone to over‑fitting
Scalability Add new language by training a new adapter only Must retrain or fine‑tune entire model

For medium‑ to high‑resource languages, full fine‑tuning can still be advantageous, but adapters dominate for scarce data scenarios.

End‑to‑end adapter fine‑tuning workflow (Turkish example)

  1. Setup – Install datasets, transformers, torchaudio, jiwer, accelerate and log in to the Hugging Face Hub.
  2. Load data – Use the Common Voice Turkish split (train+validation ≈ 4 h) and the test split for evaluation.
  3. Pre‑process transcripts – Remove punctuation, normalize case, replace diacritics, and build a character‑level vocabulary (37 tokens including a word‑delimiter |, [UNK], and [PAD]).
  4. Create tokenizer & feature extractor – Build a Wav2Vec2CTCTokenizer from the custom vocab and a Wav2Vec2FeatureExtractor with sampling_rate=16000.
  5. Prepare dataset – Cast audio to 16 kHz, extract input_values with the processor, and encode labels to token IDs.
  6. Configure model – Load facebook/mms-1b-all with ignore_mismatched_sizes=True, re‑initialize adapter layers (model.init_adapter_layers()), freeze the base model (model.freeze_base_model()), and enable gradient checkpointing.
  7. Define data collator – Use a custom DataCollatorCTCWithPadding that pads inputs and labels separately and masks label padding with -100.
  8. Training arguments – Example settings: per_device_train_batch_size=32, learning_rate=1e-3, num_train_epochs=4, fp16=True, push_to_hub=True.
  9. Run Trainer – Training completes in < 30 min on a single GPU. Sample training log shows WER dropping from ~0.28 to ~0.22 after 400 steps.
  10. Save & share adapters – Adapter weights are saved as a safe‑tensor file (adapter.<lang>.safetensors) and uploaded to the Hub.
  11. Inference – Load the checkpoint with target_lang="tur", set the tokenizer language, and run a forward pass. Example transcription matches the reference with only minor spacing differences.

Multi‑language adapter management

  • Adapters are stored alongside the base model (e.g., adapter.fra.safetensors).
  • To add a new language, train a fresh adapter and push it to the same repository; the base model remains unchanged.
  • Loading a different language is as simple as:
    model.load_adapter("swe")
processor.tokenizer.set_target_lang("swe")
  • The same repository can host dozens of adapters, each identified by its ISO‑639‑3 code.

Empirical results

Model Training steps Validation WER
mms-1b-all adapter (Turkish) 100 0.280
200 0.232
300 0.229
400 0.223

These results surpass the XLS‑R 300 M baseline reported in the earlier "Fine‑tuning XLS‑R on Multi‑Lingual ASR" blog post, confirming that MMS adapters transfer knowledge more effectively to low‑resource languages.

Implications for language preservation

Around 40 % of the world’s 3,000 living languages are endangered (Ethnologue). MMS already supports transcription for languages such as Ari and Kaivi. By enabling rapid, low‑cost adaptation to any language with a few hours of audio, MMS adapters can help communities create written corpora and maintain digital presence for threatened tongues.

Resources

The notebook used for this tutorial is available at https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_Tune_MMS_on_Common_Voice.ipynb.

Sources