Training a Language Model from Scratch with Transformers and Tokenizers

Hugging Face has released a detailed technical guide demonstrating how to train a new language model from scratch using the transformers and tokenizers libraries. By creating a small RoBERTa-like model called "EsperBERTo" for the Esperanto language, the guide illustrates the end-to-end pipeline from dataset collection to downstream task fine-tuning.

Model Architecture and Specifications

EsperBERTo is designed as a "small" model to demonstrate the feasibility of training from scratch without massive compute resources. Its architecture consists of:

  • Parameters: 84 million
  • Layers: 6
  • Hidden Size: 768
  • Attention Heads: 12

This configuration mirrors the architecture of DistilBERT. The model is trained using Masked Language Modeling (MLM), where the model learns to predict randomly masked tokens within a dataset.

The Training Pipeline

1. Dataset Acquisition

The training corpus for EsperBERTo consists of approximately 3 GB of text. The data was sourced from two primary collections:

  • OSCAR corpus: The Esperanto portion of this multilingual corpus, which is derived from filtered Common Crawl web dumps.
  • Leipzig Corpora Collection: A sub-corpus containing diverse text from news, literature, and Wikipedia to supplement the OSCAR data.

2. Tokenizer Training

Hugging Face utilizes a byte-level Byte-pair encoding (BPE) tokenizer, similar to the one used in GPT-2, with a vocabulary size of 52,000.

Key technical advantages of the byte-level BPE approach include:

  • Elimination of <unk> tokens: Because the tokenizer builds its vocabulary from an alphabet of single bytes, all words can be decomposed into tokens.
  • Language Optimization: Training a native tokenizer for Esperanto allows diacritics (such as ĉ, ĝ, ĥ, ĵ, ŝ, and ŭ) to be encoded natively.
  • Efficiency: The guide notes that the average length of encoded sequences for the Esperanto corpus is approximately 30% smaller compared to using a pretrained GPT-2 tokenizer.

3. Language Model Pretraining

The model is trained using the run_language_modeling.py script from the transformers library. To train from scratch rather than from a checkpoint, the --model_name_or_path argument is set to None.

Training Hyperparameters:

  • Learning Rate: 1e-4
  • Epochs: 5
  • Batch Size: 16 per GPU
  • Seed: 42
  • Model Type: RoBERTa

4. Validation and Testing

To verify that the model has learned linguistic patterns, Hugging Face employs the FillMaskPipeline. This allows users to input a sequence with a <mask> token and observe the most probable completions. For example, the prompt "La suno ." resulted in the top prediction "brilis" (shone), confirming the model's grasp of basic syntax and grammar.

Downstream Task Fine-Tuning

Once pretrained, the model is fine-tuned for Part-of-speech (POS) tagging. Because Esperanto is a highly regular language where word endings typically indicate the grammatical part of speech, this task is treated as a token classification problem.

Using a dataset of annotated Esperanto POS tags in CoNLL-2003 format, the model was fine-tuned using the run_ner.py script. The process involved training for 3 epochs with a batch size of 64 per GPU, resulting in successful classification of pronouns, verbs, nouns, and adjectives.

Community Sharing and Distribution

Hugging Face encourages the sharing of trained models via the transformers-cli upload command. The recommended practice for model distribution includes providing a model card (README.md) that details:

  • Model description
  • Training parameters (dataset, preprocessing, and hyperparameters)
  • Evaluation results
  • Intended uses and limitations

Sources