BERT 101: Understanding the Bidirectional Encoder Representations from Transformers
BERT (Bidirectional Encoder Representations from Transformers) is a machine learning model for natural language processing (NLP) that serves as a versatile solution for over 11 common language tasks. Developed by Google AI Language in 2018, BERT revolutionized the field by providing a single model capable of outperforming previous task-specific models across a wide array of applications.
Core Capabilities and Use Cases
BERT is designed to handle a variety of NLP tasks, ranging from sentiment analysis to complex reading comprehension. Key applications include:
- Sentiment Analysis: Determining the polarity (positive or negative) of text, such as movie reviews.
- Question Answering: Powering chatbots to provide accurate answers to user queries.
- Text Prediction and Generation: Predicting subsequent text in emails (e.g., Gmail) or generating articles from short inputs.
- Summarization: Condensing long documents, such as legal contracts, into shorter summaries.
- Polysemy Resolution: Differentiating between words with multiple meanings (e.g., "bank") based on the surrounding context.
Since November 2020, BERT has been integrated into Google Search to improve the understanding of English queries. For example, it allows the search engine to recognize that the phrase "for someone" in a prescription-related search refers to picking up a medication for another person, rather than simply filling a prescription.
Technical Architecture and Training
BERT's performance is driven by its use of the Transformer architecture and a two-step training process: unsupervised pre-training on massive unlabeled data, followed by fine-tuning on small amounts of human-annotated data.
Training Data and Infrastructure
BERT was trained on a dataset of 3.3 billion words, comprising Wikipedia (2.5 billion words) and Google's BooksCorpus (800 million words). To handle this volume of data, Google used Tensor Processing Units (TPUs). BERTbase was trained on 4 TPUs over 4 days, while BERTlarge was trained on 16 TPUs over 4 days.
Masked Language Model (MLM)
BERT employs a Masked Language Model (MLM) to enforce bidirectional learning. During training, a random 15% of tokenized words are hidden (masked), and the model must predict these words using the context from both the left and right sides of the mask. This differs from previous models that typically read text linearly.
Next Sentence Prediction (NSP)
To understand the relationships between sentences, BERT uses Next Sentence Prediction (NSP). The model is trained on a 50/50 mix of correct sentence pairs (where the second sentence follows the first) and random sentence pairs to predict whether a given sentence logically follows another.
The Transformer Encoder
BERT utilizes the encoder portion of the Transformer architecture, which uses an attention mechanism to assign differential weights to words, focusing on critical information while ignoring irrelevant data. Unlike some other Transformer models, BERT does not use a decoder.
Model Variants and Specifications
BERT is available in different sizes to balance performance and computational requirements.
| Model | Transformer Layers | Hidden Size | Attention Heads | Parameters | Processing | Training Length |
|---|---|---|---|---|---|---|
| BERTbase | 12 | 768 | 12 | 110M | 4 TPUs | 4 days |
| BERTlarge | 24 | 1024 | 16 | 340M | 16 TPUs | 4 days |
To address the need for smaller computational environments like mobile phones, 23 smaller BERT models were released in March 2020. Additionally, DistilBERT provides a lighter version that runs 60% faster while maintaining over 95% of BERT's performance.
Performance Benchmarks
BERT achieved state-of-the-art accuracy on 11 common NLP tasks and was the first model to outperform human-level performance on several benchmarks:
- SQuAD (Stanford Question Answering Dataset): A reading comprehension dataset of 108k questions. BERT outperformed both previous state-of-the-art models and humans.
- SWAG (Situations With Adversarial Generations): A commonsense inference dataset of 113k multiple-choice questions. BERT exceeded human-level performance here as well.
- GLUE (General Language Understanding Evaluation): A benchmark consisting of nine difficult tasks used to measure and analyze language models comparatively.
Open Source Impact and Ethical Considerations
BERT's source code is publicly accessible on GitHub, allowing developers to fine-tune the model for specific use cases without the massive cost of initial pre-training. This democratization has led to thousands of specialized open-source models, including those for Twitter sentiment analysis, clinical notes analysis, and toxic comment detection.
Environmental Impact
Training large models is computationally expensive and has a significant carbon footprint. The open-source sharing of pre-trained models is presented as a primary method for reducing the overall compute cost and environmental impact of the AI community.
Model Bias
BERT exhibits inherent biases present in its training data. For example, when predicting professions for the prompt "The man worked as a [MASK]," BERT suggested roles such as carpenter and mechanic. For the prompt "The woman worked as a [MASK]," it suggested roles such as nurse, waitress, and maid, demonstrating a clear gender bias in professional roles.