Last Time: Recurrent Neural Networks

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 3 May 02, 2023
 Last Time: Variable length computation L
graph with shared weights
y1 L1 y2 L2 y3 L3 yT LT

h0 fW h1 fW h2 fW h3
… hT

x1 x2 x3
W

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 4 May 02, 2023
 Image Captioning using spatial features
Input: Image I
Output: Sequence y = y1, y2,..., yT

z0,0 z0,1 z0,2

z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 26 May 02, 2023
 Image Captioning using spatial features
Input: Image I
Output: Sequence y = y1, y2,..., yT

Encoder: h0 = fW(z)
where z is spatial CNN features
fW(.) is an MLP

z0,0 z0,1 z0,2

h0
z1,0 z1,1 z1,2
CNN MLP
z2,0 z2,1 z2,2

Extract spatial Features:

features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 27 May 02, 2023
 Image Captioning using spatial features
Input: Image I Decoder: yt = gV(yt-1, ht-1, c)
Output: Sequence y = y1, y2,..., yT where context vector c is often c = h0

Encoder: h0 = fW(z) person

where z is spatial CNN features
fW(.) is an MLP y1

z0,0 z0,1 z0,2

h0 h1
z1,0 z1,1 z1,2
CNN MLP
z2,0 z2,1 z2,2

Extract spatial Features:

c y0
features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START]

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 28 May 02, 2023
 Image Captioning using spatial features
Input: Image I Decoder: yt = gV(yt-1, ht-1, c)
Output: Sequence y = y1, y2,..., yT where context vector c is often c = h0

Encoder: h0 = fW(z) person wearing

where z is spatial CNN features
fW(.) is an MLP y1 y2

z0,0 z0,1 z0,2

h0 h1 h2
z1,0 z1,1 z1,2
CNN MLP
z2,0 z2,1 z2,2

Extract spatial Features:

c y0 y1
features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START] person

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 29 May 02, 2023
 Image Captioning using spatial features
Input: Image I Decoder: yt = gV(yt-1, ht-1, c)
Output: Sequence y = y1, y2,..., yT where context vector c is often c = h0

Encoder: h0 = fW(z) person wearing hat

where z is spatial CNN features
fW(.) is an MLP y1 y2 y3

z0,0 z0,1 z0,2

h0 h1 h2 h3
z1,0 z1,1 z1,2
CNN MLP
z2,0 z2,1 z2,2

Extract spatial Features:

c y0 y1 y2
features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START] person wearing

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 30 May 02, 2023
 Image Captioning using spatial features
Input: Image I Decoder: yt = gV(yt-1, ht-1, c)
Output: Sequence y = y1, y2,..., yT where context vector c is often c = h0

Encoder: h0 = fW(z) person wearing hat [END]

where z is spatial CNN features
fW(.) is an MLP y1 y2 y3 y4

z0,0 z0,1 z0,2

h0 h1 h2 h3 h4
z1,0 z1,1 z1,2
CNN MLP
z2,0 z2,1 z2,2

Extract spatial Features:

c y0 y1 y2 y3
features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START] person wearing hat

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 31 May 02, 2023
 Image Captioning using spatial features
Problem: Input is "bottlenecked" through c
- Model needs to encode everything it
wants to say within c
person wearing hat [END]
This is a problem if we want to generate
really long descriptions? 100s of words long
y1 y2 y3 y4

z0,0 z0,1 z0,2

h0 h1 h2 h3 h4
z1,0 z1,1 z1,2
CNN MLP
z2,0 z2,1 z2,2

Extract spatial Features:

c y0 y1 y2 y3
features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START] person wearing hat

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 32 May 02, 2023
 Image Captioning with RNNs and Attention
gif source

Attention idea: New context vector at every time step.

Each context vector will attend to different image regions

Attention Saccades in humans

z0,0 z0,1 z0,2
h0
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 33 May 02, 2023
 Image Captioning with RNNs and Attention
Alignment scores:
Compute alignments HxW
scores (scalars):
e1,0,0 e1,0,1 e1,0,2

e1,1,0 e1,1,1 e1,1,2

fatt(.) is an MLP
e1,2,0 e1,2,1 e
1,2,2

z0,0 z0,1 z0,2

h0
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 34 May 02, 2023
 Image Captioning with RNNs and Attention
Alignment scores: Attention:
Compute alignments Normalize to get
HxW HxW
scores (scalars): attention weights:
e1,0,0 e1,0,1 e1,0,2 a1,0,0 a1,0,1 a1,0,2

e1,1,0 e1,1,1 e1,1,2 a1,1,0 a1,1,1 a1,1,2

fatt(.) is an MLP 0 < at, i, j < 1,
e1,2,0 e1,2,1 e1,2,2 a1,2,0 a1,2,1 a1,2,2 attention values sum to 1

z0,0 z0,1 z0,2

h0
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 35 May 02, 2023
 Image Captioning with RNNs and Attention
Alignment scores: Attention:
Compute alignments Normalize to get Compute context vector:
HxW HxW
scores (scalars): attention weights:
e1,0,0 e1,0,1 e1,0,2 a1,0,0 a1,0,1 a1,0,2

e1,1,0 e1,1,1 e a1,1,0 a1,1,1 a1,1,2

fatt(.) is an MLP 1,1,2 0 < at, i, j < 1,
e1,2,0 e1,2,1 e1,2,2 a1,2,0 a1,2,1 a1,2,2 attention values sum to 1

z0,0 z0,1 z0,2

h0
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

c1
features from a HxWxD
pretrained CNN
X
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 36 May 02, 2023
 Image Captioning with RNNs and Attention
Decoder: yt = gV(yt-1, ht-1, ct)
Each timestep of decoder uses a
New context vector at every time step
different context vector that looks at
different parts of the input image

person

y1

z0,0 z0,1 z0,2

h0 h1
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

c1 y0
features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START]

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 37 May 02, 2023
 Image Captioning with RNNs and Attention
Alignment scores: Attention: Decoder: yt = gV(yt-1, ht-1, ct)
HxW HxW New context vector at every time step
e1,0,0 e1,0,1 e1,0,2 a1,0,0 a1,0,1 a1,0,2

e1,1,0 e1,1,1 e1,1,2 a1,1,0 a1,1,1 a1,1,2

person
e1,2,0 e1,2,1 e1,2,2 a1,2,0 a1,2,1 a1,2,2
y1

z0,0 z0,1 z0,2

h0 h1
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

c1 y0 c2
features from a HxWxD
pretrained CNN
X
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START]

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 38 May 02, 2023
 Image Captioning with RNNs and Attention
Decoder: yt = gV(yt-1, ht-1, ct)
Each timestep of decoder uses a
New context vector at every time step
different context vector that looks at
different parts of the input image

person wearing

y1 y2

z0,0 z0,1 z0,2

h0 h1 h2
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

c1 y0 c2 y1
features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START] person

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 39 May 02, 2023
 Image Captioning with RNNs and Attention
Decoder: yt = gV(yt-1, ht-1, ct)
Each timestep of decoder uses a
New context vector at every time step
different context vector that looks at
different parts of the input image

person wearing hat

y1 y2 y3

z0,0 z0,1 z0,2

h0 h1 h2 h3
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

c1 y0 c2 y1 c3 y2
features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START] person wearing

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 40 May 02, 2023
 Image Captioning with RNNs and Attention
Decoder: yt = gV(yt-1, ht-1, ct)
Each timestep of decoder uses a
New context vector at every time step
different context vector that looks at
different parts of the input image

person wearing hat [END]

y1 y2 y3 y4

z0,0 z0,1 z0,2

h0 h1 h2 h3 h4
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

c1 y0 c2 y1 c3 y2 c4 y3
features from a HxWxD
pretrained CNN
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START] person wearing hat

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 41 May 02, 2023
 Image Captioning with RNNs and Attention
Alignment scores: Attention: This entire process is differentiable.
HxW HxW - model chooses its own
e1,0,0 e1,0,1 e1,0,2
attention weights. No attention
a1,0,0 a1,0,1 a1,0,2
supervision is required
e1,1,0 e1,1,1 e1,1,2 a1,1,0 a1,1,1 a1,1,2
person wearing hat [END]
e1,2,0 e1,2,1 e1,2,2 a1,2,0 a1,2,1 a1,2,2
y1 y2 y3 y4

z0,0 z0,1 z0,2

h0 h1 h2 h3 h4
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

c1 y0 c2 y1 c3 y2 c4 y3
features from a HxWxD
pretrained CNN
X
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START] person wearing hat

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 42 May 02, 2023
 Image Captioning with Attention

Soft attention

Hard attention
(requires
reinforcement
learning)

Xu et al, “Show, Attend, and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015
Figure copyright Kelvin Xu, Jimmy Lei Ba, Jamie Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Benchio, 2015. Reproduced with permission.

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 43 May 02, 2023
 Image Captioning with Attention

Xu et al, “Show, Attend, and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015
Figure copyright Kelvin Xu, Jimmy Lei Ba, Jamie Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Benchio, 2015. Reproduced with permission.

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 44 May 02, 2023
 Image Captioning with RNNs and Attention
Alignment scores: Attention: This entire process is differentiable.
HxW HxW - model chooses its own
e1,0,0 e1,0,1 e1,0,2
attention weights. No attention
a1,0,0 a1,0,1 a1,0,2
supervision is required
e1,1,0 e1,1,1 e1,1,2 a1,1,0 a1,1,1 a1,1,2
person wearing hat [END]
e1,2,0 e1,2,1 e1,2,2 a1,2,0 a1,2,1 a1,2,2
y1 y2 y3 y4

z0,0 z0,1 z0,2

h0 h1 h2 h3 h4
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

c1 y0 c2 y1 c3 y2 c4 y3
features from a HxWxD
pretrained CNN
X
Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 [START] person wearing hat

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 45 May 02, 2023
 Attention we just saw in image captioning

z0,0 z0,1 z0,2

Features

z1,0 z1,1 z1,2

z2,0 z2,1 z2,2

Inputs:
h Features: z (shape: H x W x D)
Query: h (shape: D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 46 May 02, 2023
 Attention we just saw in image captioning

Operations:
Alignment: ei,j = fatt(h, zi,j)

z0,0 z0,1 z0,2 e0,0 e0,1 e0,2

Features

Alignment

z1,0 z1,1 z1,2 e1,0 e1,1 e1,2

z2,0 z2,1 z2,2 e2,0 e2,1 e2,2

Inputs:
h Features: z (shape: H x W x D)
Query: h (shape: D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 47 May 02, 2023
 Attention we just saw in image captioning

a0,0 a0,1 a0,2

Attention
a1,0 a1,1 a1,2
Operations:
a2,0 a2,1 a2,2 Alignment: ei,j = fatt(h, zi,j)
Attention: a = softmax(e)

softmax

z0,0 z0,1 z0,2 e0,0 e0,1 e0,2

Features

Alignment

z1,0 z1,1 z1,2 e1,0 e1,1 e1,2

z2,0 z2,1 z2,2 e2,0 e2,1 e2,2

Inputs:
h Features: z (shape: H x W x D)
Query: h (shape: D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 48 May 02, 2023
 Attention we just saw in image captioning
c
Outputs:
context vector: c (shape: D)
mul + add

a0,0 a0,1 a0,2

Attention
a1,0 a1,1 a1,2
Operations:
a2,0 a2,1 a2,2 Alignment: ei,j = fatt(h, zi,j)
Attention: a = softmax(e)
Output: c = ∑i,j ai,jzi,j
softmax

z0,0 z0,1 z0,2 e0,0 e0,1 e0,2

Features

Alignment

z1,0 z1,1 z1,2 e1,0 e1,1 e1,2

z2,0 z2,1 z2,2 e2,0 e2,1 e2,2

Inputs:
h Features: z (shape: H x W x D)
Query: h (shape: D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 49 May 02, 2023
General attention layer
c
Outputs:
context vector: c (shape: D)
mul + add

a0

Attention
a1 Operations:
Alignment: ei = fatt(h, xi)
a2
Attention: a = softmax(e)
Output: c = ∑i ai xi
softmax
Input vectors

x0 e0
Alignment

x1 e1
Attention operation is permutation invariant.
x2 e2 - Doesn't care about ordering of the features
- Stretch H x W = N into N vectors
Inputs:
h Input vectors: x (shape: N x D)
Query: h (shape: D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 50 May 02, 2023
General attention layer
c
Outputs:
context vector: c (shape: D)
mul + add

a0

Attention
Change fatt(.) to a simple dot product
a1 Operations: - only works well with key & value
Alignment: ei = h ᐧ xi transformation trick (will mention in a
a2
Attention: a = softmax(e) few slides)
Output: c = ∑i ai xi
softmax
Input vectors

x0 e0
Alignment

x1 e1

x2 e2

Inputs:
h Input vectors: x (shape: N x D)
Query: h (shape: D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 51 May 02, 2023
General attention layer
c
Outputs:
context vector: c (shape: D)
mul + add
Change fatt(.) to a scaled simple dot product
a0 - Larger dimensions means more terms in

Attention
the dot product sum.
a1 Operations: - So, the variance of the logits is higher.
Alignment: ei = h ᐧ xi / √D Large magnitude vectors will produce
a2
Attention: a = softmax(e) much higher logits.
Output: c = ∑i ai xi - So, the post-softmax distribution has
softmax lower-entropy, assuming logits are IID.
Input vectors

- Ultimately, these large magnitude

x0 e0 vectors will cause softmax to peak and
Alignment

assign very little weight to all others

x1 e1 - Divide by √D to reduce effect of large
magnitude vectors
x2 e2

Inputs:
h Input vectors: x (shape: N x D)
Query: h (shape: D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 52 May 02, 2023
General attention layer
y0 y1 y2
Outputs:
context vectors: y (shape: D)
mul(→) + add (↑)
Multiple query vectors
- each query creates a new output
a0,0 a0,1 a0,2
context vector

Attention
a1,0 a1,1 a1,2
Operations:
a2,0 a2,1 a2,2
Alignment: ei,j = qj ᐧ xi / √D
Attention: a = softmax(e)
Output: yj = ∑i ai,j xi
softmax (↑)
Input vectors

x0 e0,0 e0,1 e0,2

Alignment

x1 e1,0 e1,1 e1,2

x2 e2,0 e2,1 e2,2

Multiple query vectors

Inputs:
q0 q1 q2 Input vectors: x (shape: N x D)
Queries: q (shape: M x D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 53 May 02, 2023
General attention layer
y0 y1 y2
Outputs:
context vectors: y (shape: D)
mul(→) + add (↑)

a0,0 a0,1 a0,2

Notice that the input vectors are used for

Attention
a1,0 a1,1 a1,2
Operations: both the alignment as well as the
Alignment: ei,j = qj ᐧ xi / √D attention calculations.
a2,0 a2,1 a2,2
Attention: a = softmax(e) - We can add more expressivity to
Output: yj = ∑i ai,j xi the layer by adding a different FC
softmax (↑) layer before each of the two steps.
Input vectors

x0 e0,0 e0,1 e0,2

Alignment

x1 e1,0 e1,1 e1,2

x2 e2,0 e2,1 e2,2

Inputs:
q0 q1 q2 Input vectors: x (shape: N x D)
Queries: q (shape: M x D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 54 May 02, 2023
 General attention layer

v0
Notice that the input vectors are used for
v1 Operations: both the alignment as well as the
Key vectors: k = xWk attention calculations.
v2 - We can add more expressivity to
Value vectors: v = xWv
the layer by adding a different FC
layer before each of the two steps.
Input vectors

x0 k0

x1 k1

x2 k2
Inputs:
q0 q1 q2 Input vectors: x (shape: N x D)
Queries: q (shape: M x Dk)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 55 May 02, 2023
 General attention layer
y0 y1 y2
Outputs:
The input and output dimensions can
context vectors: y (shape: Dv)
mul(→) + add (↑) now change depending on the key and
value FC layers
v0 a0,0 a0,1 a0,2
Notice that the input vectors are used for

Attention
v1 a1,0 a1,1 a1,2
Operations: both the alignment as well as the
Key vectors: k = xWk attention calculations.
v2 a2,0 a2,1 a2,2
- We can add more expressivity to
Value vectors: v = xWv
Alignment: ei,j = qj ᐧ ki / √D the layer by adding a different FC
softmax (↑) Attention: a = softmax(e) layer before each of the two steps.
Input vectors

Output: yj = ∑i ai,j vi
x0 k0 e0,0 e0,1 e0,2
Alignment

x1 k1 e1,0 e1,1 e1,2

x2 k2 e2,0 e2,1 e2,2

Inputs:
q0 q1 q2 Input vectors: x (shape: N x D)
Queries: q (shape: M x Dk)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 56 May 02, 2023
 General attention layer
y0 y1 y2
Outputs:
context vectors: y (shape: Dv)
mul(→) + add (↑)

Recall that the query vector was a

v0 a0,0 a0,1 a0,2
function of the input vectors

Attention
v1 a1,0 a1,1 a1,2
Operations: Encoder: h0 = fW(z)
Key vectors: k = xWk
v2 a2,0 a2,1 a2,2
Value vectors: v = xWv
where z is spatial CNN features
Alignment: ei,j = qj ᐧ ki / √D fW(.) is an MLP
softmax (↑) Attention: a = softmax(e)
z0,0 z0,1 z0,2
Input vectors

Output: yj = ∑i ai,j vi h0
x0 k0 e0,0 e0,1 e0,2
CNN z1,0 z1,1 z1,2 MLP
Alignment

x1 k1 e1,0 e1,1 e1,2 z2,0 z2,1 z2,2

x2 k2 e2,0 e2,1 e2,2

Inputs:
q0 q1 q2 Input vectors: x (shape: N x D)
Queries: q (shape: M x Dk)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 57 May 02, 2023
 Self attention layer

We can calculate the query vectors

from the input vectors, therefore,
defining a "self-attention" layer.
Operations:
Key vectors: k = xWk
Value vectors: v = xWv
Query vectors: q = xWq Instead, query vectors are
Alignment: ei,j = qj ᐧ ki / √D calculated using a FC layer.
Input vectors

Attention: a = softmax(e)
x0 Output: yj = ∑i ai,j vi

x1

x2
No input query vectors anymore
Inputs:
q0 q1 q2 Input vectors: x (shape: N x D)
Queries: q (shape: M x Dk)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 58 May 02, 2023
 Self attention layer
y0 y1 y2
Outputs:
context vectors: y (shape: Dv)
mul(→) + add (↑)

v0 a0,0 a0,1 a0,2

Attention
v1 a1,0 a1,1 a1,2
Operations:
Key vectors: k = xWk
v2 a2,0 a2,1 a2,2
Value vectors: v = xWv
Query vectors: q = xWq
softmax (↑) Alignment: ei,j = qj ᐧ ki / √D
Input vectors

Attention: a = softmax(e)
x0 k0 e0,0 e0,1 e0,2 Output: yj = ∑i ai,j vi
Alignment

x1 k1 e1,0 e1,1 e1,2

x2 k2 e2,0 e2,1 e2,2

Inputs:
q0 q1 q2 Input vectors: x (shape: N x D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 59 May 02, 2023
 Self attention layer - attends over sets of inputs
y0 y1 y2
Outputs:
context vectors: y (shape: Dv)
mul(→) + add (↑)

v0 a0,0 a0,1 a0,2

Attention
y0 y1 y2
v1 a1,0 a1,1 a1,2
Operations:
Key vectors: k = xWk
v2 a2,0 a2,1 a2,2
Value vectors: v = xWv
self-attention
Query vectors: q = xWq
softmax (↑) Alignment: ei,j = qj ᐧ ki / √D x0 x1 x2
Input vectors

Attention: a = softmax(e)
x0 k0 e0,0 e0,1 e0,2 Output: yj = ∑i ai,j vi
Alignment

x1 k1 e1,0 e1,1 e1,2

x2 k2 e2,0 e2,1 e2,2

Inputs:
q0 q1 q2 Input vectors: x (shape: N x D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 60 May 02, 2023
Self attention layer - attends over sets of inputs

y1 y0 y2 y2 y1 y0 y0 y1 y2

self-attention self-attention self-attention

x1 x0 x2 x2 x1 x0 x0 x1 x2

Permutation equivariant

Self-attention layer doesn’t care about the orders of the inputs!

Problem: How can we encode ordered sequences like language or spatially ordered image features?

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 61 May 02, 2023
Positional encoding
y0 y1 y2

self-attention

x0 x1 x2
p0 p1 p2

position encoding
Desiderata of pos(.) :
x0 x1 x2 1. It should output a unique encoding for each
time-step (word’s position in a sentence)
Concatenate/add special positional 2. Distance between any two time-steps should be
encoding pj to each input vector xj consistent across sentences with different lengths.
3. Our model should generalize to longer sentences
We use a function pos: N →Rd without any efforts. Its values should be bounded.
to process the position j of the vector 4. It must be deterministic.
into a d-dimensional vector

So, pj = pos(j)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 62 May 02, 2023
Positional encoding
Options for pos(.)
y0 y1 y2
1. Learn a lookup table:
○ Learn parameters to use for pos(t) for t ε [0, T)
self-attention ○ Lookup table contains T x d parameters.

x0 x1 x2
p0 p1 p2

position encoding
Desiderata of pos(.) :
x0 x1 x2 1. It should output a unique encoding for each
time-step (word’s position in a sentence)
Concatenate special positional 2. Distance between any two time-steps should be
encoding pj to each input vector xj consistent across sentences with different lengths.
3. Our model should generalize to longer sentences
We use a function pos: N →Rd without any efforts. Its values should be bounded.
to process the position j of the 4. It must be deterministic.
vector into a d-dimensional vector

So, pj = pos(j) Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 63 May 02, 2023
Positional encoding
Options for pos(.)
y0 y1 y2
1. Learn a lookup table:
○ Learn parameters to use for pos(t) for t ε [0, T)
self-attention ○ Lookup table contains T x d parameters.

x0 x1 x2 2. Design a fixed function with the desiderata

p0 p1 p2

position encoding

x1 x0 x2

Concatenate special positional

encoding pj to each input vector xj
p(t) =
We use a function pos: N →Rd
to process the position j of the
vector into a d-dimensional vector where
So, pj = pos(j) Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 64 May 02, 2023
Positional encoding
Options for pos(.)
y0 y1 y2
1. Learn a lookup table:
○ Learn parameters to use for pos(t) for t ε [0, T)
self-attention ○ Lookup table contains T x d parameters.

x0 x1 x2 2. Design a fixed function with the desiderata

p0 p1 p2

position encoding Intuition:

x0 x1 x2

Concatenate special positional

encoding pj to each input vector xj
p(t) =
We use a function pos: N →Rd
to process the position j of the
vector into a d-dimensional vector where
image source
So, pj = pos(j) Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 65 May 02, 2023
 Masked self-attention layer
y0 y1 y2
Outputs:
context vectors: y (shape: Dv)
mul(→) + add (↑)

v0 a0,0 a0,1 a0,2

Attention
v1 0 a1,1 a1,2
Operations: - Prevent vectors from
v2 0 0 a2,2
Key vectors: k = xWk looking at future vectors.
Value vectors: v = xWv
- Manually set alignment
Query vectors: q = xWq
softmax (↑) Alignment: ei,j = qj ᐧ ki / √D scores to -infinity
Input vectors

Attention: a = softmax(e)
x0 k0 e0,0 e0,1 e0,2 Output: yj = ∑i ai,j vi
Alignment

x1 k1 -∞ e1,1 e1,2

x2 k2 -∞ -∞ e2,2

Inputs:
q0 q1 q2 Input vectors: x (shape: N x D)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 66 May 02, 2023
Multi-head self-attention layer
- Multiple self-attention heads in parallel
y0 y1 y2

Concatenate

head0 head1 headH-1

y0 y1 y2 y0 y1 y2 y0 y1 y2

Self-attention Self-attention ... Self-attention

x0 x1 x2 x0 x1 x2 x0 x1 x2

Split
x0 x1 x2

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 67 May 02, 2023
General attention versus self-attention

y0 y1 y2 y0 y1 y2

attention self-attention

k0 k1 k2 v0 v1 v2 q0 q1 q2 x0 x1 x2

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 68 May 02, 2023
 Example: CNN with Self-Attention

Input Image

CNN
Features:
CxHxW
Cat image is free to use under the Pixabay License

Zhang et al, “Self-Attention Generative Adversarial Networks”, ICML 2018 Slide credit: Justin Johnson

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 69 May 02, 2023
 Example: CNN with Self-Attention

Queries:
C’ x H x W

Input Image 1x1 Conv

Keys:
CNN C’ x H x W

1x1 Conv
Features:
CxHxW
Cat image is free to use under the Pixabay License

Values:
C’ x H x W

1x1 Conv

Zhang et al, “Self-Attention Generative Adversarial Networks”, ICML 2018 Slide credit: Justin Johnson

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 70 May 02, 2023
 Example: CNN with Self-Attention

Attention Weights
Queries:
Transpose (H x W) x (H x W)
C’ x H x W

Input Image 1x1 Conv

softmax
x
Keys:
CNN C’ x H x W

1x1 Conv
Features:
CxHxW
Cat image is free to use under the Pixabay License

Values:
C’ x H x W

1x1 Conv

Zhang et al, “Self-Attention Generative Adversarial Networks”, ICML 2018 Slide credit: Justin Johnson

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 71 May 02, 2023
 Example: CNN with Self-Attention

Attention Weights
Queries:
Transpose (H x W) x (H x W)
C’ x H x W

Input Image 1x1 Conv

softmax
x
Keys:
CNN C’ x H x W

1x1 Conv
Features:
CxHxW C’ x H x W
Cat image is free to use under the Pixabay License

Values:
C’ x H x W
x
1x1 Conv

Zhang et al, “Self-Attention Generative Adversarial Networks”, ICML 2018 Slide credit: Justin Johnson

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 72 May 02, 2023
 Example: CNN with Self-Attention

Attention Weights
Queries:
Transpose (H x W) x (H x W)
C’ x H x W

Input Image 1x1 Conv

softmax
x CxHxH
Keys:
CNN C’ x H x W

1x1 Conv
Features:
CxHxW C’ x H x W
Cat image is free to use under the Pixabay License

Values:
C’ x H x W
x 1x1 Conv
1x1 Conv

Zhang et al, “Self-Attention Generative Adversarial Networks”, ICML 2018 Slide credit: Justin Johnson

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 73 May 02, 2023
 Example: CNN with Self-Attention
Residual Connection

Attention Weights
Queries:
Transpose (H x W) x (H x W)
C’ x H x W

Input Image 1x1 Conv

softmax
x CxHxW
Keys:
CNN C’ x H x W
+
1x1 Conv
Features:
CxHxW C’ x H x W
Cat image is free to use under the Pixabay License

Values:
C’ x H x W
x 1x1 Conv
1x1 Conv

Self-Attention Module
Zhang et al, “Self-Attention Generative Adversarial Networks”, ICML 2018 Slide credit: Justin Johnson

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 74 May 02, 2023
Comparing RNNs to Transformer

RNNs
(+) LSTMs work reasonably well for long sequences.
(-) Expects an ordered sequences of inputs
(-) Sequential computation: subsequent hidden states can only be computed after the previous
ones are done.

Transformer:
(+) Good at long sequences. Each attention calculation looks at all inputs.
(+) Can operate over unordered sets or ordered sequences with positional encodings.
(+) Parallel computation: All alignment and attention scores for all inputs can be done in parallel.
(-) Requires a lot of memory: N x M alignment and attention scalers need to be calculated and
stored for a single self-attention head. (but GPUs are getting bigger and better)

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 75 May 02, 2023
 “ImageNet Moment for Natural Language Processing”

Pretraining:
Download a lot of text from the internet

Train a giant Transformer model for language modeling

Finetuning:
Fine-tune the Transformer on your own NLP task

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 76 May 02, 2023
Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 77 May 02, 2023
Image Captioning using Transformers
Input: Image I
Output: Sequence y = y1, y2,..., yT

z0,0 z0,1 z0,2

z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2

Extract spatial Features:

features from a HxWxD
pretrained CNN

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 78 May 02, 2023
Image Captioning using Transformers
Input: Image I
Output: Sequence y = y1, y2,..., yT

Encoder: c = TW(z)
where z is spatial CNN features
TW(.) is the transformer encoder

z0,0 z0,1 z0,2 c0,0 c0,1 c0,2 ... c2,2

z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2 Transformer encoder

Extract spatial Features:

features from a HxWxD z0,0 z0,1 z0,2 ... z2,2
pretrained CNN

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 79 May 02, 2023
Image Captioning using Transformers
Input: Image I Decoder: yt = TD(y0:t-1, c)
Output: Sequence y = y1, y2,..., yT where TD(.) is the transformer decoder

Encoder: c = TW(z) person wearing hat [END]

where z is spatial CNN features
TW(.) is the transformer encoder y1 y2 y3 y4

z0,0 z0,1 z0,2 c0,0 c0,1 c0,2 ... c2,2

Transformer decoder
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2 Transformer encoder

Extract spatial Features:

y0 y1 y2 y3
features from a HxWxD z0,0 z0,1 z0,2 ... z2,2
pretrained CNN
[START] person wearing hat

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 80 May 02, 2023
 ...
The Transformer encoder block

c0,0 c0,1 c0,2 ... c2,2

Transformer encoder

xN
Made up of N encoder blocks.

In vaswani et al. N = 6, Dq= 512

z0,0 z0,1 z0,2 ... z2,2

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 81 May 02, 2023
 ...
The Transformer encoder block

c0,0 c0,1 c0,2 ... c2,2

Transformer encoder

xN
Let's dive into one encoder block

z0,0 z0,1 z0,2 ... z2,2 x0 x1 x2 x2

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 82 May 02, 2023
 ...
The Transformer encoder block

c0,0 c0,1 c0,2 ... c2,2

Transformer encoder

xN

Positional encoding Add positional encoding

z0,0 z0,1 z0,2 ... z2,2 x0 x1 x2 x2

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 83 May 02, 2023
 ...
The Transformer encoder block

c0,0 c0,1 c0,2 ... c2,2

Transformer encoder

xN

Multi-head self-attention Attention attends over all the vectors

Positional encoding Add positional encoding

z0,0 z0,1 z0,2 ... z2,2 x0 x1 x2 x2

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 84 May 02, 2023
 ...
The Transformer encoder block

c0,0 c0,1 c0,2 ... c2,2

Transformer encoder

xN

+ Residual connection

Multi-head self-attention Attention attends over all the vectors

Positional encoding Add positional encoding

z0,0 z0,1 z0,2 ... z2,2 x0 x1 x2 x2

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 85 May 02, 2023
 ...
The Transformer encoder block

c0,0 c0,1 c0,2 ... c2,2

Transformer encoder

xN
Layer norm LayerNorm over each vector individually

+ Residual connection

Multi-head self-attention Attention attends over all the vectors

Positional encoding Add positional encoding

z0,0 z0,1 z0,2 ... z2,2 x0 x1 x2 x2

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 86 May 02, 2023
 ...
The Transformer encoder block

c0,0 c0,1 c0,2 ... c2,2

Transformer encoder

xN MLP MLP over each vector individually

Layer norm LayerNorm over each vector individually

+ Residual connection

Multi-head self-attention Attention attends over all the vectors

Positional encoding Add positional encoding

z0,0 z0,1 z0,2 ... z2,2 x0 x1 x2 x2

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 87 May 02, 2023
 ...
The Transformer encoder block

c0,0 c0,1 c0,2 ... c2,2

+ Residual connection
Transformer encoder

xN MLP MLP over each vector individually

Layer norm LayerNorm over each vector individually

+ Residual connection

Multi-head self-attention Attention attends over all the vectors

Positional encoding Add positional encoding

z0,0 z0,1 z0,2 ... z2,2 x0 x1 x2 x2

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 88 May 02, 2023
 ...
The Transformer encoder block
y0 y1 y2 y3
c0,0 c0,1 c0,2 ... c2,2 Transformer Encoder Block:
Layer norm
Inputs: Set of vectors x
+ Outputs: Set of vectors y
Transformer encoder

MLP Self-attention is the only

xN
interaction between vectors.
Layer norm

+
Layer norm and MLP operate
independently per vector.
Multi-head self-attention
Highly scalable, highly
Positional encoding parallelizable, but high memory usage.

z0,0 z0,1 z0,2 ... z2,2 x0 x1 x2 x2

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 89 May 02, 2023
The
...
Transformer
decoder block
person wearing hat [END]

y0 y1 y2 y3

Transformer decoder
Made up of N decoder blocks.

xN In vaswani et al. N = 6, Dq= 512

c0,0

c0,1

c0,2
...

c2,2
y0 y1 y2 y3

[START] person wearing hat Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 90 May 02, 2023
The
...
Transformer y0 y1 y2 y3

decoder block FC

person wearing hat [END]

y0 y1 y2 y3

Transformer decoder
Let's dive into the
c0,0
transformer decoder block
xN
c0,1
c0,0
c0,2

...
c0,1

c0,2 c2,2
...

c2,2
y0 y1 y2 y3

[START] person wearing hat x0 x1 x2 x3

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 91 May 02, 2023
The
...
Transformer y0 y1 y2 y3

Decoder block FC
Layer norm
person wearing hat [END]
+

y0 y1 y2 y3
MLP
Layer norm

Transformer decoder
+ Most of the network is the
c0,0
same the transformer
xN encoder.
c0,1
c0,0
c0,2
Layer norm

...
c0,1
+

c0,2 c2,2
Masked Multi-head
self-attention
...

c2,2 Positional encoding

y0 y1 y2 y3

[START] person wearing hat x0 x1 x2 x3

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 92 May 02, 2023
The
...
Transformer y0 y1 y2 y3

Decoder block FC
Layer norm
person wearing hat [END]
+

y0 y1 y2 y3
MLP
Layer norm

Transformer decoder
+ Multi-head attention block
c0,0
attends over the transformer
xN Multi-head attention encoder outputs.
c0,1 k v q
c0,0
c0,2 For image captioning, this is
Layer norm
how we inject image

...
c0,1
+
features into the decoder.
c0,2 c2,2
Masked Multi-head
self-attention
...

c2,2 Positional encoding

y0 y1 y2 y3

[START] person wearing hat x0 x1 x2 x3

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 93 May 02, 2023
The
...
Transformer y0 y1 y2 y3

Decoder block FC
Layer norm
Transformer Decoder Block:

person wearing hat [END] Inputs: Set of vectors x and

+
Set of context vectors c.
y0 y1 y2 y3 Outputs: Set of vectors y.
MLP
Layer norm
Masked Self-attention only

Transformer decoder
c0,0
+ interacts with past inputs.
xN Multi-head attention
c0,1 k v q Multi-head attention block is
c0,0
NOT self-attention. It attends
c0,2 over encoder outputs.
Layer norm

...
c0,1
+
Highly scalable, highly
c0,2 c2,2
Masked Multi-head parallelizable, but high memory
self-attention usage.
...

c2,2 Positional encoding

y0 y1 y2 y3

[START] person wearing hat x0 x1 x2 x3

Vaswani et al, “Attention is all you need”, NeurIPS 2017

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 94 May 02, 2023
Image Captioning using transformers
- No recurrence at all

person wearing hat [END]

y1 y2 y3 y4

z0,0 z0,1 z0,2 c0,0 c0,1 c0,2 ... c2,2

Transformer decoder
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2 Transformer encoder

Extract spatial Features:

y0 y1 y2 y3
features from a HxWxD z0,0 z0,1 z0,2 ... z2,2
pretrained CNN
[START] person wearing hat

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 95 May 02, 2023
Image Captioning using transformers
- Perhaps we don't need
convolutions at all?

person wearing hat [END]

y1 y2 y3 y4

z0,0 z0,1 z0,2 c0,0 c0,1 c0,2 ... c2,2

Transformer decoder
z1,0 z1,1 z1,2
CNN
z2,0 z2,1 z2,2 Transformer encoder

Extract spatial Features:

y0 y1 y2 y3
features from a HxWxD z0,0 z0,1 z0,2 ... z2,2
pretrained CNN
[START] person wearing hat

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 96 May 02, 2023
 Image Captioning using ONLY transformers
- Transformers from pixels to language

person wearing hat [END]

y1 y2 y3 y4

c0,0 c0,1 c0,2 ... c2,2

Transformer decoder

Transformer encoder

y0 y1 y2 y3
...

Dosovitskiy et al, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, ArXiv 2020
[START] person wearing hat
Colab link to an implementation of vision transformers

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 97 May 02, 2023
 Vision Transformers vs. ResNets

Dosovitskiy et al, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, ArXiv 2020
Colab link to an implementation of vision transformers

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 98 May 02, 2023
Vision Transformers

Liu et al, “Swin Transformer: Hierarchical Vision

Fan et al, “Multiscale Vision Transformers”, ICCV 2021 Transformer using Shifted Windows”, CVPR 2021

Carion et al, “End-to-End Object Detection with Transformers”,

ECCV 2020

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 99 May 02, 2023
 ConvNets strike back!

A ConvNet for the 2020s. Liu et al. CVPR 2022

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 100 May 02, 2023
Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 101 May 02, 2023
 Summary
- Adding attention to RNNs allows them to "attend" to different
parts of the input at every time step
- The general attention layer is a new type of layer that can be
used to design new neural network architectures
- Transformers are a type of layer that uses self-attention and
layer norm.
○ It is highly scalable and highly parallelizable
○ Faster training, larger models, better performance across
vision and language tasks
○ They are quickly replacing RNNs, LSTMs, and may(?) even
replace convolutions.

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 102 May 02, 2023
 Next time: Video Understanding

Fei-Fei Li, Yunzhu Li, Ruohan Gao Lecture 9 - 103 May 02, 2023