Contextual Word Representations with

BERT and Other Pre-trained Language

Models

Jacob Devlin
Google AI Language
History and Background
Pre-training in NLP
● Word embeddings are the basis of deep learning
for NLP
king queen

[-0.5, -0.9, 1.4, …] [-0.6, -0.8, -0.2, …]

● Word embeddings (word2vec, GloVe) are often

pre-trained on text corpus from co-occurrence
statistics
Inner Product Inner Product

the king wore a crown the queen wore a crown

Contextual Representations
● Problem: Word embeddings are applied in a
context free manner
open a bank account on the river bank

[0.3, 0.2, -0.8, …]

● Solution: Train contextual representations on text

corpus
[0.9, -0.2, 1.6, …] [-1.9, -0.4, 0.1, …]

open a bank account on the river bank

History of Contextual Representations
● Semi-Supervised Sequence Learning, Google, 2015

Train LSTM Fine-tune on

Language Model Classification Task
open a bank POSITIVE

LSTM LSTM LSTM ... LSTM LSTM LSTM

<s> open a very funny movie

History of Contextual Representations
● ELMo: Deep Contextual Word Embeddings, AI2 &
University of Washington, 2017
Train Separate Left-to-Right and Apply as “Pre-trained
Right-to-Left LMs Embeddings”
open a bank <s> open a
Existing Model Architecture

LSTM LSTM LSTM LSTM LSTM LSTM

<s> open a open a bank

open a bank
History of Contextual Representations
● Improving Language Understanding by Generative
Pre-Training, OpenAI, 2018

Train Deep (12-layer) Fine-tune on

Transformer LM Classification Task
POSITIVE
open a bank

Transformer Transformer Transformer

Transformer Transformer Transformer

<s> open a
<s> open a
Model Architecture
Transformer encoder
● Multi-headed self attention
○ Models context
● Feed-forward layers
○ Computes non-linear hierarchical features
● Layer norm and residuals
○ Makes training deep networks healthy
● Positional embeddings
○ Allows model to learn relative positioning
Model Architecture
● Empirical advantages of Transformer vs. LSTM:
1. Self-attention == no locality bias
● Long-distance context has “equal opportunity”
2. Single multiplication per layer == efficiency on TPU
● Effective batch size is number of words, not sequences
Transformer LSTM

X_0_0 X_0_1 X_0_2 X_0_3 X_0_0 X_0_1 X_0_2 X_0_3

✕ W ✕ W
X_1_0 X_1_1 X_1_2 X_1_3 X_1_0 X_1_1 X_1_2 X_1_3
BERT
Problem with Previous Methods
● Problem: Language models only use left context or
right context, but language understanding is
bidirectional.
● Why are LMs unidirectional?
● Reason 1: Directionality is needed to generate a
well-formed probability distribution.
○ We don’t care about this.
● Reason 2: Words can “see themselves” in a
bidirectional encoder.
Unidirectional vs. Bidirectional Models

Unidirectional context Bidirectional context

Build representation incrementally Words can “see themselves”

open a bank open a bank

Layer 2 Layer 2 Layer 2 Layer 2 Layer 2 Layer 2

Layer 2 Layer 2 Layer 2 Layer 2 Layer 2 Layer 2

<s> open a <s> open a

Masked LM
● Solution: Mask out k% of the input words, and
then predict the masked words
○ We always use k = 15%

store gallon

the man went to the [MASK] to buy a [MASK] of milk

● Too little masking: Too expensive to train

● Too much masking: Not enough context
Masked LM
● Problem: Mask token never seen at fine-tuning
● Solution: 15% of the words to predict, but don’t
replace with [MASK] 100% of the time. Instead:
● 80% of the time, replace with [MASK]
went to the store → went to the [MASK]
● 10% of the time, replace random word
went to the store → went to the running
● 10% of the time, keep same
went to the store → went to the store
Next Sentence Prediction
● To learn relationships between sentences, predict
whether Sentence B is actual sentence that
proceeds Sentence A, or a random sentence
Input Representation

● Use 30,000 WordPiece vocabulary on input.

● Each token is sum of three embeddings
● Single sequence is much more efficient.
Model Details
● Data: Wikipedia (2.5B words) + BookCorpus (800M
words)
● Batch Size: 131,072 words (1024 sequences * 128
length or 256 sequences * 512 length)
● Training Time: 1M steps (~40 epochs)
● Optimizer: AdamW, 1e-4 learning rate, linear decay
● BERT-Base: 12-layer, 768-hidden, 12-head
● BERT-Large: 24-layer, 1024-hidden, 16-head
● Trained on 4x4 or 8x8 TPU slice for 4 days
Fine-Tuning Procedure
Fine-Tuning Procedure
 GLUE Results

MultiNLI CoLa
Premise: Hills and mountains are especially Sentence: The wagon rumbled down the road.
sanctified in Jainism. Label: Acceptable
Hypothesis: Jainism hates nature.
Label: Contradiction Sentence: The car honked down the road.
Label: Unacceptable
SQuAD 2.0

● Use token 0 ([CLS]) to emit

logit for “no answer”.
● “No answer” directly
competes with answer span.
● Threshold is optimized on dev
set.
Effect of Pre-training Task

● Masked LM (compared to left-to-right LM) is very important on

some tasks, Next Sentence Prediction is important on other tasks.
● Left-to-right model does very poorly on word-level task (SQuAD),
although this is mitigated by BiLSTM
Effect of Directionality and Training Time

● Masked LM takes slightly longer to converge because we

only predict 15% instead of 100%
● But absolute results are much better almost immediately
Effect of Model Size

● Big models help a lot

● Going from 110M -> 340M params helps even on
datasets with 3,600 labeled examples
● Improvements have not asymptoted
Open Source Release
● One reason for BERT’s success was the open
source release
○ Minimal release (not part of a larger codebase)
○ No dependencies but TensorFlow (or PyTorch)
○ Abstracted so people could including a single file to use model
○ End-to-end push-button examples to train SOTA models
○ Thorough README
○ Idiomatic code
○ Well-documented code
○ Good support (for the first few months)
Post-BERT Pre-training Advancements
RoBERTA
● RoBERTa: A Robustly Optimized BERT Pretraining
Approach (Liu et al, University of Washington and
Facebook, 2019)
● Trained BERT for more epochs and/or on more data
○ Showed that more epochs alone helps, even on same data
○ More data also helps
● Improved masking and pre-training data slightly
XLNet
● XLNet: Generalized Autoregressive Pretraining for
Language Understanding (Yang et al, CMU and
Google, 2019)
● Innovation #1: Relative position embeddings
○ Sentence: John ate a hot dog
○ Absolute attention: “How much should dog attend to hot(in any
position), and how much should dog in position 4 attend to the word
in position 3? (Or 508 attend to 507, …)”
○ Relative attention: “How much should dog attend to hot (in any
position) and how much should dog attend to the previous word?”
XLNet
● Innovation #2: Permutation Language Modeling
○ In a left-to-right language model, every word is predicted based on
all of the words to its left
○ Instead: Randomly permute the order for every training sentence
○ Equivalent to masking, but many more predictions per sentence
○ Can be done efficiently with Transformers
XLNet
● Also used more data and bigger models, but
showed that innovations improved on BERT even
with same data and model size
● XLNet results:
ALBERT
● ALBERT: A Lite BERT for Self-supervised Learning
of Language Representations (Lan et al, Google
and TTI Chicago, 2019)
● Innovation #1: Factorized embedding
parameterization
○ Use small embedding size (e.g., 128) and then project it to
Transformer hidden size (e.g., 1024) with parameter matrix

1024 128 1024

x vs. x ⨉ x
100k 100k 128
ALBERT
● Innovation #2: Cross-layer parameter sharing
○ Share all parameters between Transformer layers
● Results:

● ALBERT is light in terms of parameters, not speed

T5
● Exploring the Limits of Transfer Learning with a
Unified Text-to-Text Transformer (Raffel et al,
Google, 2019)
● Ablated many aspects of pre-training:
○ Model size
○ Amount of training data
○ Domain/cleanness of training data
○ Pre-training objective details (e.g., span length of masked text)
○ Ensembling
○ Finetuning recipe (e.g., only allowing certain layers to finetune)
○ Multi-task training
T5
● Conclusions:
○ Scaling up model size and amount of training data helps a lot
○ Best model is 11B parameters (BERT-Large is 330M), trained on 120B
words of cleaned common crawl text
○ Exact masking/corruptions strategy doesn’t matter that much
○ Mostly negative results for better finetuning and multi-task strategies
● T5 results:
ELECTRA
● ELECTRA: Pre-training Text Encoders as
Discriminators Rather Than Generators (Clark et al,
2020)
● Train model to discriminate locally plausible text
from real text
ELECTRA
● Difficult to match SOTA results with less compute
Distillation
Applying Models to Production Services
● BERT and other pre-trained language models are
extremely large and expensive
● How are companies applying them to low-latency
production services?
Distillation
● Answer: Distillation (a.k.a., model compression)
● Idea has been around for a long time:
○ Model Compression (Bucila et al, 2006)
○ Distilling the Knowledge in a Neural Network (Hinton et al, 2015)
● Simple technique:
○ Train “Teacher”: Use SOTA pre-training + fine-tuning technique to
train model with maximum accuracy
○ Label a large amount of unlabeled input examples with Teacher
○ Train “Student”: Much smaller model (e.g., 50x smaller) which is
trained to mimic Teacher output
○ Student objective is typically Mean Square Error or Cross Entropy
Distillation
● Example distillation results
○ 50k labeled examples, 8M unlabeled examples

Well-Read Students Learn

Better: On the Importance of
Pre-training Compact Models
(Turc et al, 2020)

● Distillation works much better than pre-training +

fine-tuning with smaller model
Distillation
● Why does distillation work so well? A hypothesis:
○ Language modeling is the “ultimate” NLP task in many ways
■ I.e., a perfect language model is also a perfect question
answering/entailment/sentiment analysis model
○ Training a massive language model learns millions of latent features
which are useful for these other NLP tasks
○ Finetuning mostly just picks up and tweaks these existing latent
features
○ This requires an oversized model, because only a subset of the
features are useful for any given task
○ Distillation allows the model to only focus on those features
○ Supporting evidence: Simple self-distillation (distilling a smaller BERT
model) doesn’t work
Conclusions
Conclusions
● Pre-trained bidirectional language models work
incredibly well
● However, the models are extremely expensive
● Improvements (unfortunately) seem to mostly
come from even more expensive models and more
data
● The inference/serving problem is mostly “solved”
through distillation