Scribd
Upload
50 views

BERT Diagrams Public

The document contains illustrations from Devlin et al. (2019) showing how BERT represents input text for various tasks. It shows how tokens are embedded and combined with segment, position, and classification embeddings to capture relationships between words and tasks.

Uploaded by

Stelios Iordanis
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
50 views

BERT Diagrams Public

The document contains illustrations from Devlin et al. (2019) showing how BERT represents input text for various tasks. It shows how tokens are embedded and combined with segment, position, and classification embeddings to capture relationships between words and tasks.

Uploaded by

Stelios Iordanis
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 10

Illustration of BERT, showing composition of input embeddings. Redrawn from Devlin et al.

(NAACL 2019)

T[CLS] T1 T2 T3 T4 T5 T6 T7 T[SEP]

… … … … … … … … …

E[CLS] E1 E2 E3 E4 E5 E6 E7 E[SEP]

Token
[CLS] At11 At22 At33 t4
[SEP] Bt51 Bt62 t7 [SEP]
Embeddings

Segment
+ + + + + + + + +
EA EA EA EA EA EA EA EA EA
Embeddings

Position
+ + + + + + + + +
P0 P1 P2 P3 P4 P5 P6 P7 P8
Embeddings

By Jimmy Lin ([email protected]), released under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/
Illustration of BERT, showing composition of input embeddings. Redrawn from Devlin et al. (NAACL 2019)

T[CLS] T1 T2 T3 T4 T5 T6 T7 T[SEP]

E[CLS] E1 E2 E3 E4 E5 E6 E7 E[SEP]

Token
[CLS] At11 At22 At33 t4
[SEP] Bt51 Bt62 t7 [SEP]
Embeddings

Segment
+ + + + + + + + +
EA EA EA EA EA EA EA EA EA
Embeddings

Position
+ + + + + + + + +
P0 P1 P2 P3 P4 P5 P6 P7 P8
Embeddings

By Jimmy Lin ([email protected]), released under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/
Illustration of BERT for single-sentence classification tasks. Redrawn from Devlin et al. (NAACL 2019)

Class Label

T[CLS] U1 U2 U3 … Un-2 Un-1 Un T[SEP]

… … … … … … … … …

E[CLS] E1 E2 E3 … En-2 En-1 En E[SEP]

[CLS] A1 A2 A3 … An-2 An-1 An [SEP]

Sentence

By Jimmy Lin ([email protected]), released under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/
Illustration of BERT for single-sentence classification tasks. Redrawn from Devlin et al. (NAACL 2019)

Class Label

T[CLS] U1 U2 U3 … Un-2 Un-1 Un T[SEP]

E[CLS] E1 E2 E3 … En-2 En-1 En E[SEP]

[CLS] A1 A2 A3 … An-2 An-1 An [SEP]

Sentence

By Jimmy Lin ([email protected]), released under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/
Illustration of BERT for single-sentence sequence labeling tasks. Redrawn from Devlin et al. (NAACL 2019)

O O B-PER O O O

T[CLS] U1 U2 U3 … Un-2 Un-1 Un T[SEP]

… … … … … … … … …

E[CLS] E1 E2 E3 … Fn-2 Fn-1 Fn E[SEP]

[CLS] A1 A2 A3 … An-2 An-1 An [SEP]

Sentence

By Jimmy Lin ([email protected]), released under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/
Illustration of BERT for single-sentence sequence labeling tasks. Redrawn from Devlin et al. (NAACL 2019)

O O B-PER O O O

T[CLS] U1 U2 U3 … Un-2 Un-1 Un T[SEP]

E[CLS] E1 E2 E3 … Fn-2 Fn-1 Fn E[SEP]

[CLS] A1 A2 A3 … An-2 An-1 An [SEP]

Sentence

By Jimmy Lin ([email protected]), released under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/
Illustration of BERT for two-sentence classification tasks. Redrawn from Devlin et al. (NAACL 2019)

Class Label

T[CLS] U1 … U3 T[SEP1] V1 … Vm T[SEP2]

… … … … … … … … …

E[CLS] E1 … E3 E[SEP1] F1 … Fm E[SEP2]

[CLS] A1 … A3 [SEP1] B1 … Bm [SEP2]

Sentence 1 Sentence 2

By Jimmy Lin ([email protected]), released under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/
Illustration of BERT for two-sentence classification tasks. Redrawn from Devlin et al. (NAACL 2019)

Class Label

T[CLS] U1 … U3 T[SEP1] V1 … Vm T[SEP2]

E[CLS] E1 … E3 E[SEP1] F1 … Fm E[SEP2]

[CLS] A1 … A3 [SEP1] B1 … Bm [SEP2]

Sentence 1 Sentence 2

By Jimmy Lin ([email protected]), released under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/
Illustration of BERT for two-sentence sequence labeling tasks. Redrawn from Devlin et al. (NAACL 2019)

Start/End Span

T[CLS] U1 … U3 T[SEP1] V1 … Vm T[SEP2]

… … … … … … … … …

E[CLS] E1 … E3 E[SEP1] F1 … Fm E[SEP2]

[CLS] A1 … A3 [SEP1] B1 … Bm [SEP2]

Question Candidate

By Jimmy Lin ([email protected]), released under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/
Illustration of BERT for two-sentence sequence labeling tasks. Redrawn from Devlin et al. (NAACL 2019)

Start/End Span

T[CLS] U1 … U3 T[SEP1] V1 … Vm T[SEP2]

E[CLS] E1 … E3 E[SEP1] F1 … Fm E[SEP2]

[CLS] A1 … A3 [SEP1] B1 … Bm [SEP2]

Question Candidate

By Jimmy Lin ([email protected]), released under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/

You might also like