![Next Sentence Prediction
BERT
[SEP]
Yes/No
[CLS]
Linear
Robustly optimized BERT approach
(RoBERTa)
w1 w2
Sentence 1
w3 w4 w5
Sentence 2
• This approach is not helpful.
https://arxiv.org/abs/1907.11692
• SOP: Sentence order prediction
Used in ALBERT
https://arxiv.org/abs/1909.11942](https://image.slidesharecdn.com/bertv8-250118012622-9739cdfe/85/self-supervised-learning-and-Bert-from-a-13-320.jpg)
![How to use BERT – Case 1
BERT
[CLS] w1 w2 w3
Linear
class Input: sequence
output: class
sentence
Example:
Sentiment analysis
Random
initialization
Init by pre-train
This is the model
to be learned.
this is good
positive
Better than random](https://image.slidesharecdn.com/bertv8-250118012622-9739cdfe/85/self-supervised-learning-and-Bert-from-a-17-320.jpg)
![How to use BERT – Case 2
BERT
[CLS] w1 w2 w3
Linear
class
Input: sequence
output: same as input
sentence
Linear
class
Linear
class
I saw a saw
N V DET N
Example:
POS tagging](https://image.slidesharecdn.com/bertv8-250118012622-9739cdfe/85/self-supervised-learning-and-Bert-from-a-20-320.jpg)
![Linear
w1 w2
How to use BERT – Case 3
BERT
[CLS] [SEP]
Class
Sentence 1 Sentence 2
w3 w4 w5
Input: two sequences
Output: a class](https://image.slidesharecdn.com/bertv8-250118012622-9739cdfe/85/self-supervised-learning-and-Bert-from-a-22-320.jpg)
![q1 q2
How to use BERT – Case 4
BERT
[CLS] [SEP]
question document
d1 d2 d3
inner product
Softmax
0.5
0.3 0.2
s = 2
Random
Initialized](https://image.slidesharecdn.com/bertv8-250118012622-9739cdfe/85/self-supervised-learning-and-Bert-from-a-24-320.jpg)
![q1 q2
How to use BERT – Case 4
BERT
[CLS] [SEP]
question document
d1 d2 d3
inner product
Softmax
0.2
0.1 0.7
The answer is “d2 d3”.
s = 2 e = 3
Random
Initialized](https://image.slidesharecdn.com/bertv8-250118012622-9739cdfe/85/self-supervised-learning-and-Bert-from-a-25-320.jpg)
![MASS / BART
BART
A B [SEP] C D E
A B [SEP] C D E
A B [SEP] C E
C D E [SEP] A B
D E A B [SEP] C
A B [SEP] E
MASS
(Delete
“D”)
Text Infilling
(permutation)
(rotation)
https://arxiv.org/abs/1905.02450
https://arxiv.org/abs/1910.13461](https://image.slidesharecdn.com/bertv8-250118012622-9739cdfe/85/self-supervised-learning-and-Bert-from-a-30-320.jpg)
![A we
T you
C he
G she
This work is done by 高瑋聰
https://arxiv.org/abs/2103.07162
BERT
[CLS]
Linear
class
DNA sequence
Random
initialization
Init by pre-train
pre-train on English
Why does BERT work?
A G A C
we we
she he](https://image.slidesharecdn.com/bertv8-250118012622-9739cdfe/85/self-supervised-learning-and-Bert-from-a-37-320.jpg)
Ad
More Related Content
Similar to self-supervised learning and Bert from a (20)
Recently uploaded (20)
Ad
self-supervised learning and Bert from a
- 2. 死臭酸宅本人 芝麻街
- 3. ELMo (Embeddings from Language Models) BERT (Bidirectional Encoder Representations from Transformers) ERNIE (Enhanced Representation through Knowledge Integration) Big Bird: Transformers for Longer Sequences
- 4. Source of image: https://leemeng.tw/attack_on_bert_transfer_learni ng_in_nlp.html BERT Bertolt Hoover 340M parameters
- 6. Source of image: https://huaban.com/pins/1714071707/ ELMO (94M) BERT (340M) GPT-2 (1542M) The models become larger and larger …
- 7. Megatron (8B) GPT-2 T5 (11B) Turing NLG (17B) The models become larger and larger … GPT-3 is 10 times larger than Turing NLG.
- 9. Outline BERT series GPT series
- 11. Masking Input BERT 台 MASK Random (special token) https://arxiv.org/abs/1810.04805 灣 大 學 Transformer Encoder Linear 學 0.1 灣 0.7 台 0.1 大 0.1 …… …… (all characters) = = or Randomly masking some tokens 一、天、大、小 … softmax
- 12. Masking Input BERT 台 MASK Random (special token) https://arxiv.org/abs/1810.04805 灣 大 學 Transformer Encoder Linear = = or Randomly masking some tokens 一、天、大、小 … softmax 灣 Ground truth minimize cross entropy
- 13. Next Sentence Prediction BERT [SEP] Yes/No [CLS] Linear Robustly optimized BERT approach (RoBERTa) w1 w2 Sentence 1 w3 w4 w5 Sentence 2 • This approach is not helpful. https://arxiv.org/abs/1907.11692 • SOP: Sentence order prediction Used in ALBERT https://arxiv.org/abs/1909.11942
- 14. • Masked token prediction • Next sentence prediction BERT Self-supervised Learning Model for Task 1 Downstream Tasks Model for Task 2 Model for Task 3 • The tasks we care • We have a little bit labeled data. Fine-tune Pre-train
- 15. GLUE • Corpus of Linguistic Acceptability (CoLA) • Stanford Sentiment Treebank (SST-2) • Microsoft Research Paraphrase Corpus (MRPC) • Quora Question Pairs (QQP) • Semantic Textual Similarity Benchmark (STS-B) • Multi-Genre Natural Language Inference (MNLI) • Question-answering NLI (QNLI) • Recognizing Textual Entailment (RTE) • Winograd NLI (WNLI) General Language Understanding Evaluation (GLUE) https://gluebenchmark.com/ GLUE also has Chinese version (https://www.cluebenchmarks.com/)
- 16. BERT and its Family • GLUE scores Source of image: https://arxiv.org/abs/1905.00537
- 17. How to use BERT – Case 1 BERT [CLS] w1 w2 w3 Linear class Input: sequence output: class sentence Example: Sentiment analysis Random initialization Init by pre-train This is the model to be learned. this is good positive Better than random
- 18. Pre-train v.s. Random Initialization Source of image: https://arxiv.org/abs/1908.05620 (fine- tune) (scratch)
- 19. 19
- 20. How to use BERT – Case 2 BERT [CLS] w1 w2 w3 Linear class Input: sequence output: same as input sentence Linear class Linear class I saw a saw N V DET N Example: POS tagging
- 21. How to use BERT – Case 3 Input: two sequences Output: a class premise: A person on a horse jumps over a broken down airplane hypothesis: A person is at a diner. contradiction Model contradiction entailment neutral Example: Natural Language Inferencee (NLI)
- 22. Linear w1 w2 How to use BERT – Case 3 BERT [CLS] [SEP] Class Sentence 1 Sentence 2 w3 w4 w5 Input: two sequences Output: a class
- 23. How to use BERT – Case 4 • Extraction-based Question Answering (QA) 𝐷={𝑑1,𝑑2 ,⋯ ,𝑑𝑁 } 𝑄={𝑞1 , 𝑞2 , ⋯ , 𝑞𝑀 } QA Model output: two integers (, ) 𝐴={𝑑𝑠 , ⋯ ,𝑑𝑒 } Document: Query: Answer: 𝐷 𝑄 𝑠 𝑒 17 77 79 𝑠=17 , 𝑒=17 𝑠=77 , 𝑒=79
- 24. q1 q2 How to use BERT – Case 4 BERT [CLS] [SEP] question document d1 d2 d3 inner product Softmax 0.5 0.3 0.2 s = 2 Random Initialized
- 25. q1 q2 How to use BERT – Case 4 BERT [CLS] [SEP] question document d1 d2 d3 inner product Softmax 0.2 0.1 0.7 The answer is “d2 d3”. s = 2 e = 3 Random Initialized
- 26. That is all about BERT!
- 27. Training BERT is challenging! GLUE scores This work is done by 姜成翰 台達電產學合作計畫研究成果 Our ALBERT-base Google’s ALBERT-base https://arxiv.org/abs/2010.02480 Google’s BERT-base Training data has more than 3 billions of words. 3000 times of Harry Potter series 8 days with TPU v3
- 28. BERT Embryology ( 胚胎學 ) When does BERT know POS tagging, syntactic parsing, semantics? The answer is counterintuitive! https://arxiv.org/abs/2010.02480
- 29. Pre-training a seq2seq model w1 w2 w3 w5 w6 w7 w4 Cross Attention w8 Decoder Encoder w1 w2 w3 w4 Reconstruct the input Corrupted
- 30. MASS / BART BART A B [SEP] C D E A B [SEP] C D E A B [SEP] C E C D E [SEP] A B D E A B [SEP] C A B [SEP] E MASS (Delete “D”) Text Infilling (permutation) (rotation) https://arxiv.org/abs/1905.02450 https://arxiv.org/abs/1910.13461
- 31. T5 – Comparison • Transfer Text-to-Text Transformer (T5) • Colossal Clean Crawled Corpus (C4)
- 32. Why does BERT work? BERT 台 灣 大 學 Represent the meaning of “ 大” 魚 鳥 草 電 吃蘋果 蘋果手機 embedding The tokens with similar meaning have similar embedding. Context is considered.
- 33. Why does BERT work? BERT 喝 蘋 果 汁 BERT 蘋 果 電 腦 compute cosine similarity
- 35. Why does BERT work? John Rupert Firth You shall know a word by the company it keeps BERT w1 w2 w3 w4 w2 word embedding Contextualized word embedding
- 36. Why does BERT work? • Applying BERT to protein, DNA, music classification This work is done by 高瑋聰 https://arxiv.org/abs/2103.07162 EI CCAGCTGCATCACAGGAGGCCAGCG EI AGACCCGCCGGGAGGCGGAGGACC IE AACGTGGCCTCCTTGTGCCCTTCCCC IE CCACTCAGCCAGGCCCTTCTTCTCCT IE CCTGATCTGGGTCTCCCCTCCCACCCT IE AGCCCTCAACCCTTCTGTCTCACCCTC IE CCACTCAGCCAGGCCCTTCTTCTCCT N CTGTGTTCACCACATCAAGCGCCGGG N GTGTTACCGAGGGCATTTCTAACAGT N TCTGAGCTCTGCATTTGTCTATTCTCC class DNA sequence
- 37. A we T you C he G she This work is done by 高瑋聰 https://arxiv.org/abs/2103.07162 BERT [CLS] Linear class DNA sequence Random initialization Init by pre-train pre-train on English Why does BERT work? A G A C we we she he
- 38. Why does BERT work? • Applying BERT to protein, DNA, music classification This work is done by 高瑋聰 https://arxiv.org/abs/2103.07162
- 39. To Learn More …… BERT (Part 1) BERT (Part 2) https://youtu.be/1_gRK9EIQpc https://youtu.be/Bywo7m6ySlk
- 40. Multi-lingual BERT Multi-BERT 深 度 學 習 Training a BERT model by many different languages. Multi-BERT high est moun tain Mask Mask
- 41. Zero-shot Reading Comprehension Training on the sentences of 104 languages Multi-BERT Doc1 Query1 Ans1 Doc2 Query2 Ans2 Doc3 Query3 Ans3 Doc4 Query4 Ans4 Doc5 Query5 Ans5 Doc1 Query1 ? Doc3 Query3 ? Doc2 Query2 ? Train on English QA training examples Test on Chinese QA test
- 42. Zero-shot Reading Comprehension • English: SQuAD, Chinese: DRCD F1 score of Human performance is 93.30% Model Pre-train Fine-tune Test EM F1 QANet none Chinese Chinese 66.1 78.1 BERT Chinese Chinese 82.0 89.1 104 languages Chinese 81.2 88.7 English 63.3 78.8 Chinese + English 82.6 90.1 This work is done by 劉記良、許宗嫄 https://arxiv.org/abs/1909.09587
- 43. Cross-lingual Alignment? Multi-BERT 深 度 學 習 high est moun tain 魚 兔 跳 游 swim jump rabbit fish
- 44. 投影片來源 : 許宗嫄同學碩士口試投影 片 https://arxiv.org/abs/2010.10938 Mean Reciprocal Rank (MRR): Higher MRR, better alignment Google’s Multi-BERT Our Multi-BERT 200k sentences for each lang How about 1000k?
- 45. The training is also challenging … Two days …… (the whole training took one week)
- 46. 投影片來源 : 許宗嫄同學碩士口試投影 片 https://arxiv.org/abs/2010.10938 Mean Reciprocal Rank (MRR): Higher MRR, better alignment Google’s Multi- BERT Our Multi-BERT 200k sentences for each lang Our Multi-BERT 1000k sentences The amount of training data is critical for alignment.
- 47. 魚 兔 跳 游 swim jump rabbit fish Multi-BERT 深 度 學 習 high est moun tain Reconstruction 深 度 學 習 high est moun tain Weird??? If the embedding is language independent … How to correctly reconstruct? There must be language information. https://arxiv.org/abs/2010.10041
- 48. Multi-BERT Reconstruction 那 有 一 貓 Where is Language? Average of Chinese Average of English This work is done by 劉記良、許宗嫄、莊永松 there is a cat + + + + 魚 兔 跳 游 swim jump rabbit fish
- 49. If this is true … Average of Chinese Average of English This work is done by 劉記良、許宗嫄、莊永松 魚 兔 跳 游 swim jump rabbit fish x Unsupervised token-level translation https://arxiv.org/abs/2010.10041
- 50. Outline BERT series GPT series
- 51. Predict Next Token <BOS> 台 灣 台 灣 大 h1 h2 h3 h4 Model ? ? ? ? 大 學 Linear Transform softmax Cross entropy wt+1 from wt h𝑡 Training data: “ 台灣 大學”
- 52. Predict Next Token They can do generation. https://talktotransformer.com/
- 53. How to use GPT? Description A few example
- 55. Average of 42 tasks
- 56. To learn more …… https://youtu.be/DOG1L9lvsDY
- 57. Beyond Text Data Centric Prediction Position, 2015 Jigsaw, 2017 Rotation, 2018 Cutout, 2015 RNNLM, 1997 word2v, 2013 audio2v, 2019 BERT, 2018 Mock, 2020 TERA, 2020 APC, 2019 NLP Speech CV Contrastive InfoNCE, 2017 CPC, 2019 MoCo, 2019 SimCLR, 2020 MoCov2, 2020 BYOL, 2020 SimSiam, 2020 本投影片由劉廷緯同學提供
- 59. Image- BYOL Bootstrap your own latent: A new approach to self-supervised Learning https://arxiv.org/abs/2006.07733
- 60. Speech Audio version BERT 深 度 學 習
- 61. Speech GLUE- SUPERB • Speech processing Universal PERformance Benchmark • Will be available soon • Downstream: Benchmark with 10+ tasks • The models need to know how to process content, speaker, emotion, and even semantics. • Toolkit: A flexible and modularized framework for self-supervised speech models. • https://github.com/s3prl/s3prl
- 64. Predict Next Token They can do generation.
- 65. 律師
- 66. 混亂
- 67. I forced a bot to watch over 1,000 hours of XXX 是一個梗 ! 人在模仿機器模仿人 !!!
Editor's Notes
- #3: BERT & PALs (Projected Attention Layers)
- #4: 超巨大巨人 諫ㄐㄧㄢˋ山創 貝爾托特·胡佛 Bertholdt Hoover 「進擊的巨人可以窺視未來繼承 者的記憶，也就是說它能看到未來。」
- #6: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf 93.6M 340M 1542M 40 GB
- #8: Switch! 比玉山高
- #13: [CLS] stands for classification.
- #14: Downstream Tasks : the task you really want to solve Better than directly using labeld data
- #15: COLA: Each example is a sequence of words annotated with whether it is a grammatical English sentence MRPC: with human annotations for whether the sentences in the pair are semantically equivalent QQP:o determine whether a pair of questions are semantically equivalent. STS: Each pair is human-annotated with a similarity score from 1 to 5; the task is to predict these scores
- #17: Do I have to mention adaptor??? (https://arxiv.org/abs/1805.12471)
- #20: (https://arxiv.org/abs/1805.12471)
- #21: premise前提
- #22: determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”.
- #23: (E.g. SQuAD)
- #24: determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”.
- #25: determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”.
- #27: (哈利波特全套約 100 萬個詞) BERT (一個巨大的 LM) 用了 30 億個以上的詞 ===== 12500 x 60 x 130 = 97,500,000 (將近 一億) English Gigaword corpus (1200M words the Harry Potter books contain 1,084,170 words. At a typical speaking pace of 130 words per minute, a 1 minute speech will be about 130 words. https://wordcounter.net/blog/2015/11/23/10922_how-many-words-harry-potter.html
- #28: some spoil here
- #29: [Song, et al., ICML’19] [Lewis, et al., arXiv’19]
- #30: Permutation / Rotation do not perform well. Text Infilling is consistently good.
- #31: The C4 dataset we created for unsupervised pre-training is available in TensorFlow Datasets, but it requires a significant amount of bandwidth for downloading the raw Common Crawl scrapes (~7 TB) and compute for its preparation (~335 CPU-days). We suggest you take advantage of the Apache Beam support in TFDS, which enables distributed preprocessing of the dataset and can be run on Google Cloud Dataflow. With 500 workers, the job should complete in ~16 hours. Colossal龐大柯羅索巨獸 吐槽谷歌T5 Level（高級軟件工程師）。 Interesting demo: https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html 67
- #34: https://colab.research.google.com/drive/1mWOWFdmI_f18Vo2HpxaVrauzlnAAWw2C
- #38: https://ppfocus.com/hk/0/di75f00e4.html
- #40: You have an universal token set for all languages.
- #41: Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT https://arxiv.org/abs/1904.09077
- #43: https://www.youtube.com/watch?v=8rDN1jUI82g
- #44: https://en.wikipedia.org/wiki/Language_code TH態文 EL希臘文
- #46: https://en.wikipedia.org/wiki/Language_code TH態文 EL希臘文
- #51: https://arxiv.org/abs/1802.05365
- #52: https://app.inferkit.com/generate
- #53: Two reason: BERT has done It is too large ……
- #56: Self-supervised Learning: (or semi-supervied learning ) Good ref: https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html Scenario Application: text, image, audio Pretext: Contrastive SimCLR, MoCo Comparison of different approaches: https://arxiv.org/pdf/2011.10566v1.pdf No negative - BOYL Why it work: https://www.untitled-ai.com/understanding-self-supervised-contrastive-learning.html Prediction Reconstruction Teacher / Student Self-training with Noisy Student improves ImageNet classification Billion-scale semi-supervised learning for image classification Self-training v.s. Self-supervied learning https://medium.com/%E8%BB%9F%E9%AB%94%E4%B9%8B%E5%BF%83/deep-learning-self-training%E7%95%B6%E9%81%93-%E5%B0%8D%E6%AF%94pre-training%E7%9A%84%E5%84%AA%E7%BC%BA%E9%BB%9E-4f1b5a937c5d Other task: Unsupervised Representation Learning by Predicting Image Rotations https://arxiv.org/pdf/1803.07728.pdf Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles https://arxiv.org/abs/1603.09246 Tracking Emerges by Colorizing Videos Weird thing: https://research.aimultiple.com/self-supervised-learning/
- #60: http://superbbenchmark.org/
- #62: self-supervised speech
- #64: https://talktotransformer.com/ https://disp.cc/b/115-bC2b
- #65: əˈtərnē
- #66: ˈanərkē