VinAIResearch/PhoBERT

PhoBERT: Pre-trained language models for Vietnamese (EMNLP-2020 Findings)

What it solves

PhoBERT is designed to provide high-performance, pre-trained language models specifically for the Vietnamese language, overcoming the limitations of previous monolingual and multilingual models in understanding Vietnamese text.

How it works

It uses a RoBERTa-based architecture to optimize the BERT pre-training procedure. The models are trained on large-scale Vietnamese corpora, including Wikipedia and news texts. To ensure accuracy, the input text must be word-segmented (using tools like VnCoreNLP) before being processed by the model.

Who it’s for

It is intended for NLP researchers and developers building Vietnamese-specific applications, such as Part-of-speech tagging, Dependency parsing, Named-entity recognition, and Natural language inference.

Highlights

Available in "base" and "large" versions with parameter counts ranging from 135M to 370M.
Outperforms previous monolingual and multilingual approaches on several downstream Vietnamese NLP tasks.
Compatible with the Hugging Face transformers library and fairseq.
Pre-trained on up to 140GB of Vietnamese text (in the v2 base model).

Dispatch
mmBERT: ModernBERT goes Multilingual
Hugging Face introduces mmBERT, a massively multilingual encoder model trained on 3T+ tokens across 1,800+ languages that outperforms XLM-R in both performance and inference speed.
Dispatch
Training a Language Model from Scratch with Transformers and Tokenizers
Hugging Face provides a comprehensive guide and demonstration on training a new language model from scratch using the Transformers and Tokenizers libraries, featuring the creation of EsperBERTo for the Esperanto language.
Dispatch
Graphcore and Hugging Face Expand IPU-Ready Transformer Models
Graphcore and Hugging Face have expanded the Hugging Face Optimum library to include 10 optimized transformer models across NLP, speech, and computer vision for Graphcore IPUs.
Project
lightonai/pylate
A library for simplifying the fine-tuning, inference, and retrieval of ColBERT late interaction models, providing high-speed indexing and flexible training options.
Dispatch
Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models – Hugging Face Blog Summary
Hugging Face’s blog post explains how to warm-start encoder-decoder models using pre‑trained BERT, RoBERTa, or GPT2 checkpoints, showing that this approach matches the performance of large pre‑trained seq2seq models while cutting training cost.

VinAIResearch/PhoBERT

What it solves

How it works

Who it’s for

Highlights

Related