Lecture 5:

Convolutional Neural Networks

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 1 April 18, 2017
 Administrative

Assignment 1 due Thursday April 20, 11:59pm on Canvas

Assignment 2 will be released Thursday

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 2 April 18, 2017
 Last time: Neural Networks
Linear score function:
2-layer Neural Network

x W1 h W2 s
3072 100 10

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 3 April 18, 20173
 Next: Convolutional Neural Networks

Illustration of LeCun et al. 1998 from CS231n 2017 Lecture 1

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 4 April 18, 20174
 A bit of history...
The Mark I Perceptron machine was the first
implementation of the perceptron algorithm.

The machine was connected to a camera that used

2020 cadmium sulfide photocells to produce a 400-pixel
image.

recognized
letters of the alphabet

update rule:

Frank Rosenblatt, ~1957: Perceptron

This image by Rocky Acosta is licensed under CC-BY 3.0

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 5 April 18, 2017
 A bit of history...

These figures are reproduced from Widrow 1960, Stanford Electronics Laboratories Technical

Widrow and Hoff, ~1960: Adaline/Madaline Report with permission from Stanford University Special Collections.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 6 April 18, 2017
 A bit of history...

recognizable math

Illustration of Rumelhart et al., 1986 by Lane McIntosh,

copyright CS231n 2017

Rumelhart et al., 1986: First time back-propagation became popular

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 7 April 18, 2017
 A bit of history...

[Hinton and Salakhutdinov 2006]

Reinvigorated research in
Deep Learning

Illustration of Hinton and Salakhutdinov 2006 by Lane

McIntosh, copyright CS231n 2017

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 8 April 18, 2017
 First strong results
Acoustic Modeling using Deep Belief Networks
Abdel-rahman Mohamed, George Dahl, Geoffrey Hinton, 2010
Context-Dependent Pre-trained Deep Neural Networks
for Large Vocabulary Speech Recognition
George Dahl, Dong Yu, Li Deng, Alex Acero, 2012

Imagenet classification with deep convolutional

Illustration of Dahl et al. 2012 by Lane McIntosh, copyright
neural networks CS231n 2017

Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton, 2012

Figures copyright Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, 2012. Reproduced with permission.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 9 April 18, 2017
A bit of history:

Hubel & Wiesel,

1959
RECEPTIVE FIELDS OF SINGLE
NEURONES IN
THE CAT'S STRIATE CORTEX

1962
RECEPTIVE FIELDS, BINOCULAR
INTERACTION
AND FUNCTIONAL ARCHITECTURE IN
THE CAT'S VISUAL CORTEX
Cat image by CNX OpenStax is licensed

1968... under CC BY 4.0; changes made

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 10

10 April 18, 2017
 A bit of history Human brain

Topographical mapping in the cortex:

nearby cells in cortex represent
nearby regions in the visual field
Visual
cortex

Retinotopy images courtesy of Jesse Gomez in the

Stanford Vision & Perception Neuroscience Lab.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 11

11 April 18, 2017
Hierarchical organization

Illustration of hierarchical organization in early visual

pathways by Lane McIntosh, copyright CS231n 2017

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 12

12 April 18, 2017
A bit of history:

Neocognitron
[Fukushima 1980]

sandwich architecture (SCSCSC)

simple cells: modifiable parameters
complex cells: perform pooling

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 13

13 April 18, 2017
A bit of history:
Gradient-based learning applied to
document recognition
[LeCun, Bottou, Bengio, Haffner 1998]

LeNet-5

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 14

14 April 18, 2017
A bit of history:
ImageNet Classification with Deep
Convolutional Neural Networks
[Krizhevsky, Sutskever, Hinton, 2012]

Figure copyright Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, 2012. Reproduced with permission.

AlexNet

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 15

15 April 18, 2017
 Fast-forward to today: ConvNets are everywhere
Classification Retrieval

Figures copyright Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, 2012. Reproduced with permission.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 16 April 18, 2017
 Fast-forward to today: ConvNets are everywhere
Detection Segmentation

Figures copyright Shaoqing Ren, Kaiming He, Ross Girschick, Jian Sun, 2015. Reproduced with Figures copyright Clement Farabet, 2012.
permission. Reproduced with permission. [Farabet et al., 2012]
[Faster R-CNN: Ren, He, Girshick, Sun 2015]

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 17 April 18, 2017
 Fast-forward to today: ConvNets are everywhere

This image by GBPublic_PR is

licensed under CC-BY 2.0

NVIDIA Tesla line

(these are the GPUs on rye01.stanford.edu)

Note that for embedded systems a typical setup

Photo by Lane McIntosh. Copyright CS231n 2017.
would involve NVIDIA Tegras, with integrated
self-driving cars GPU and ARM-based CPU cores.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 18 April 18, 2017
 Fast-forward to today: ConvNets are everywhere

[Taigman et al. 2014] Activations of inception-v3 architecture [Szegedy et al. 2015] to image of Emma McIntosh,
used with permission. Figure and architecture not from Taigman et al. 2014.

Illustration by Lane McIntosh,

Figures copyright Simonyan et al., 2014. photos of Katie Cumnock
[Simonyan et al. 2014] Reproduced with permission. used with permission.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 19 April 18, 2017
 Fast-forward to today: ConvNets are everywhere

Images are examples of pose estimation, not actually from Toshev & Szegedy 2014. Copyright Lane McIntosh.
[Toshev, Szegedy 2014]

[Guo et al. 2014] Figures copyright Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard Lewis,
and Xiaoshi Wang, 2014. Reproduced with permission.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 20 April 18, 2017
 Fast-forward to today: ConvNets are everywhere

[Levy et al. 2016] Figure copyright Levy et al. 2016.

Reproduced with permission.

Photos by Lane McIntosh.

[Sermanet et al. 2011] Copyright CS231n 2017.

From left to right: public domain by NASA, usage permitted by

ESA/Hubble, public domain by NASA, and public domain. [Ciresan et al.]
[Dieleman et al. 2014]

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 21 April 18, 2017
 This image by Christin Khan is in the public domain Photo and figure by Lane McIntosh; not actual
and originally came from the U.S. NOAA. example from Mnih and Hinton, 2010 paper.

Whale recognition, Kaggle Challenge Mnih and Hinton, 2010

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 22 April 18, 2017
 No errors Minor errors Somewhat related
Image
Captioning
[Vinyals et al., 2015]
[Karpathy and Fei-Fei,
2015]

A white teddy bear sitting in A man in a baseball A woman is holding a

the grass uniform throwing a ball cat in her hand

All images are CC0 Public domain:

https://pixabay.com/en/luggage-antique-cat-1643010/
https://pixabay.com/en/teddy-plush-bears-cute-teddy-bear-1623436/
https://pixabay.com/en/surf-wave-summer-sport-litoral-1668716/
https://pixabay.com/en/woman-female-model-portrait-adult-983967/
https://pixabay.com/en/handstand-lake-meditation-496008/
A man riding a wave on A cat sitting on a A woman standing on a https://pixabay.com/en/baseball-player-shortstop-infield-1045263/

top of a surfboard suitcase on the floor beach holding a surfboard Captions generated by Justin Johnson using Neuraltalk2

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 23 April 18, 2017
 Original image is CC0 public domain
Starry Night and Tree Roots by Van Gogh are in the public domain
Bokeh image is in the public domain Gatys et al, Image Style Transfer using Convolutional Neural Networks, CVPR 2016
Figures copyright Justin Johnson, 2015. Reproduced with permission. Generated using the Inceptionism approach
Stylized images copyright Justin Johnson, 2017; Gatys et al, Controlling Perceptual Factors in Neural Style Transfer, CVPR 2017
from a blog post by Google Research.
reproduced with permission

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 24 April 18, 2017
 Convolutional Neural Networks
(First without the brain stuff)

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 25 April 18, 2017
Fully Connected Layer
32x32x3 image -> stretch to 3072 x 1

input activation

1 1
10 x 3072
3072 10
weights

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 26 April 18, 2017
Fully Connected Layer
32x32x3 image -> stretch to 3072 x 1

input activation

1 1
10 x 3072
3072 10
weights
1 number:
the result of taking a dot product
between a row of W and the input
(a 3072-dimensional dot product)

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 27 April 18, 2017
Convolution Layer
32x32x3 image -> preserve spatial structure

32 height

32 width
3 depth

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 28 April 18, 2017
Convolution Layer
32x32x3 image

5x5x3 filter
32

Convolve the filter with the image

i.e. slide over the image spatially,
computing dot products

32
3

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 29 April 18, 2017
Convolution Layer Filters always extend the full
depth of the input volume
32x32x3 image

5x5x3 filter
32

Convolve the filter with the image

i.e. slide over the image spatially,
computing dot products

32
3

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 30 April 18, 2017
Convolution Layer
32x32x3 image
5x5x3 filter
32

1 number:
the result of taking a dot product between the
filter and a small 5x5x3 chunk of the image
32 (i.e. 5*5*3 = 75-dimensional dot product + bias)
3

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 31 April 18, 2017
Convolution Layer
activation map
32x32x3 image
5x5x3 filter
32

28

convolve (slide) over all

spatial locations

32 28
3 1

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 32 April 18, 2017
 consider a second, green filter
Convolution Layer
32x32x3 image activation maps
5x5x3 filter
32

28

convolve (slide) over all

spatial locations

32 28
3 1

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 33 April 18, 2017
 For example, if we had 6 5x5 filters, well get 6 separate activation maps:
activation maps

32

28

Convolution Layer

32 28
3 6

We stack these up to get a new image of size 28x28x6!

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 34 April 18, 2017
 Preview: ConvNet is a sequence of Convolution Layers, interspersed with
activation functions

32 28

CONV,
ReLU
e.g. 6
5x5x3
32 filters 28
3 6

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 35 April 18, 2017
 Preview: ConvNet is a sequence of Convolutional Layers, interspersed with
activation functions

32 28 24

.
CONV, CONV, CONV,
ReLU ReLU ReLU
e.g. 6 e.g. 10
5x5x3 5x5x6
32 filters 28 filters 24
3 6 10

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 36 April 18, 2017
 Visualization of VGG-16 by Lane McIntosh. VGG-16
Preview [Zeiler and Fergus 2013] architecture from [Simonyan and Zisserman 2014].

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 37 April 18, 2017
Preview

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 38 April 18, 2017
 one filter =>
one activation map example 5x5 filters
(32 total)

We call the layer convolutional

because it is related to convolution
of two signals:

elementwise multiplication and sum of

a filter and the signal (image)
Figure copyright Andrej Karpathy.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 39 April 18, 2017
preview:

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 40 April 18, 2017
A closer look at spatial dimensions:
activation map
32x32x3 image
5x5x3 filter
32

28

convolve (slide) over all

spatial locations

32 28
3 1

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 41 April 18, 2017
A closer look at spatial dimensions:

7
7x7 input (spatially)
assume 3x3 filter

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 42 April 18, 2017
A closer look at spatial dimensions:

7
7x7 input (spatially)
assume 3x3 filter

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 43 April 18, 2017
A closer look at spatial dimensions:

7
7x7 input (spatially)
assume 3x3 filter

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 44 April 18, 2017
A closer look at spatial dimensions:

7
7x7 input (spatially)
assume 3x3 filter

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 45 April 18, 2017
A closer look at spatial dimensions:

7
7x7 input (spatially)
assume 3x3 filter

=> 5x5 output

7

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 46 April 18, 2017
A closer look at spatial dimensions:

7
7x7 input (spatially)
assume 3x3 filter
applied with stride 2

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 47 April 18, 2017
A closer look at spatial dimensions:

7
7x7 input (spatially)
assume 3x3 filter
applied with stride 2

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 48 April 18, 2017
A closer look at spatial dimensions:

7
7x7 input (spatially)
assume 3x3 filter
applied with stride 2
=> 3x3 output!
7

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 49 April 18, 2017
A closer look at spatial dimensions:

7
7x7 input (spatially)
assume 3x3 filter
applied with stride 3?

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 50 April 18, 2017
A closer look at spatial dimensions:

7
7x7 input (spatially)
assume 3x3 filter
applied with stride 3?

7 doesnt fit!
cannot apply 3x3 filter on
7x7 input with stride 3.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 51 April 18, 2017
 N
Output size:
(N - F) / stride + 1
F
e.g. N = 7, F = 3:
F N
stride 1 => (7 - 3)/1 + 1 = 5
stride 2 => (7 - 3)/2 + 1 = 3
stride 3 => (7 - 3)/3 + 1 = 2.33 :\

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 52 April 18, 2017
 In practice: Common to zero pad the border
0 0 0 0 0 0
e.g. input 7x7
0 3x3 filter, applied with stride 1
0 pad with 1 pixel border => what is the output?
0

(recall:)
(N - F) / stride + 1

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 53 April 18, 2017
 In practice: Common to zero pad the border
0 0 0 0 0 0
e.g. input 7x7
0 3x3 filter, applied with stride 1
0 pad with 1 pixel border => what is the output?
0
7x7 output!
0

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 54 April 18, 2017
 In practice: Common to zero pad the border
0 0 0 0 0 0
e.g. input 7x7
0 3x3 filter, applied with stride 1
0 pad with 1 pixel border => what is the output?
0
7x7 output!
0
in general, common to see CONV layers with
stride 1, filters of size FxF, and zero-padding with
(F-1)/2. (will preserve size spatially)
e.g. F = 3 => zero pad with 1
F = 5 => zero pad with 2
F = 7 => zero pad with 3

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 55 April 18, 2017
 Remember back to
E.g. 32x32 input convolved repeatedly with 5x5 filters shrinks volumes spatially!
(32 -> 28 -> 24 ...). Shrinking too fast is not good, doesnt work well.

32 28 24

.
CONV, CONV, CONV,
ReLU ReLU ReLU
e.g. 6 e.g. 10
5x5x3 5x5x6
32 filters 28 filters 24
3 6 10
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 56 April 18, 2017
 Examples time:

Input volume: 32x32x3

10 5x5 filters with stride 1, pad 2

Output volume size: ?

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 57 April 18, 2017
 Examples time:

Input volume: 32x32x3

10 5x5 filters with stride 1, pad 2

Output volume size:

(32+2*2-5)/1+1 = 32 spatially, so
32x32x10

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 58 April 18, 2017
 Examples time:

Input volume: 32x32x3

10 5x5 filters with stride 1, pad 2

Number of parameters in this layer?

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 59 April 18, 2017
 Examples time:

Input volume: 32x32x3

10 5x5 filters with stride 1, pad 2

Number of parameters in this layer?

each filter has 5*5*3 + 1 = 76 params (+1 for bias)
=> 76*10 = 760

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 60 April 18, 2017
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 61 April 18, 2017
 Common settings:

K = (powers of 2, e.g. 32, 64, 128, 512)

- F = 3, S = 1, P = 1
- F = 5, S = 1, P = 2
- F = 5, S = 2, P = ? (whatever fits)
- F = 1, S = 1, P = 0

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 62 April 18, 2017
 (btw, 1x1 convolution layers make perfect sense)

1x1 CONV
56 with 32 filters
56
(each filter has size
1x1x64, and performs a
64-dimensional dot
56 product)
56
64 32

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 63 April 18, 2017
Example: CONV
layer in Torch

Torch is licensed under BSD 3-clause.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 64 April 18, 2017
Example: CONV
layer in Caffe

Caffe is licensed under BSD 2-Clause.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 65 April 18, 2017
 The brain/neuron view of CONV Layer

32x32x3 image
5x5x3 filter
32

1 number:
32 the result of taking a dot product between
the filter and this part of the image
3
(i.e. 5*5*3 = 75-dimensional dot product)

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 66 April 18, 2017
 The brain/neuron view of CONV Layer

32x32x3 image
5x5x3 filter
32

Its just a neuron with local

connectivity...
1 number:
32 the result of taking a dot product between
the filter and this part of the image
3
(i.e. 5*5*3 = 75-dimensional dot product)

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 67 April 18, 2017
 The brain/neuron view of CONV Layer

32

28 An activation map is a 28x28 sheet of neuron

outputs:
1. Each is connected to a small region in the input
2. All of them share parameters

32
28 5x5 filter -> 5x5 receptive field for each neuron
3

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 68 April 18, 2017
 The brain/neuron view of CONV Layer

32

E.g. with 5 filters,

28 CONV layer consists of
neurons arranged in a 3D grid
(28x28x5)

There will be 5 different

32 28 neurons all looking at the same
3 region in the input volume
5

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 69 April 18, 2017
Reminder: Fully Connected Layer
Each neuron
32x32x3 image -> stretch to 3072 x 1 looks at the full
input volume
input activation

1 1
10 x 3072
3072 10
weights
1 number:
the result of taking a dot product
between a row of W and the input
(a 3072-dimensional dot product)

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 70 April 18, 2017
two more layers to go: POOL/FC

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 71 April 18, 2017
 Pooling layer
- makes the representations smaller and more manageable
- operates over each activation map independently:

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 72 April 18, 2017
 MAX POOLING

Single depth slice

1 1 2 4
x max pool with 2x2 filters
5 6 7 8 and stride 2 6 8

3 2 1 0 3 4

1 2 3 4

y
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 73 April 18, 2017
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 74 April 18, 2017
 Common settings:

F = 2, S = 2
F = 3, S = 2

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 75 April 18, 2017
 Fully Connected Layer (FC layer)
- Contains neurons that connect to the entire input volume, as in ordinary Neural
Networks

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 76 April 18, 2017
 [ConvNetJS demo: training on CIFAR-10]

http://cs.stanford.edu/people/karpathy/convnetjs/demo/cifar10.html

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 77 April 18, 2017
 Summary

- ConvNets stack CONV,POOL,FC layers

- Trend towards smaller filters and deeper architectures
- Trend towards getting rid of POOL/FC layers (just CONV)
- Typical architectures look like
[(CONV-RELU)*N-POOL?]*M-(FC-RELU)*K,SOFTMAX
where N is usually up to ~5, M is large, 0 <= K <= 2.
- but recent advances such as ResNet/GoogLeNet
challenge this paradigm

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 78 April 18, 2017