1/14/2024

Applied Neural
Networks
Unit 9

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 1

Lecture Outline
▪ Coevolutionary Neural Networks
▪ Edge Detection
▪ Padding and Stride
▪ Pooling
▪ Residual Networks
▪ Transfer Learning

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 2

1
 1/14/2024

Computer Vision Problem

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 3

CNN Motivation

W [1] = (1000, 1million)

64 x 64 x 3 = 12288

{
1 million

640 x 480 x 3 ≈ 1 million

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 4

2
 1/14/2024

Convolution Operation

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 5

Vertical Edge Detection

-5 -4 0 8
1 0 -1 -10 -2 2 3
* 1 0 -1 = 0 -2 -4 -7
1 0 -1 -3 -2 -3 -16
3x3
Filter/
Kernel

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 6

3
 1/14/2024

Convolution Operation

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 7

Vertical Edge Detection

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 8

4
 1/14/2024

Vertical Edge Detection (Dark to Light Transition)

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 9

Vertical and Horizontal Edge Detection

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 10

10

5
 1/14/2024

Right Weights for Edge Detection

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 11

11

Learning to Detect Edges

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 12

12

6
 1/14/2024

Padding

* =

3x3 4x4
fxf
n-f+1 = 4
6x6
nxn

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 13

13

Padding
• Padding: artificially fill borders of image
• Usually: fill with 0s
• Useful to keep spatial dimension constant
across filters
• The size of the output of a padded image can
be extracted using
• n+2p-f+1 x n+2p-f+1
• Where p is the padding size (p = 1 for this case)

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 14

14

7
 1/14/2024

Valid and Same Convolutions (How much to pad?)

Common Practices
Valid Padding: Same Padding
n-f+1 x n-f+1
n+2p-f+1 x n+2p-f+1
For n = n+2p-f+1
p = (f-1)/2
f is usually an odd
number

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 15

15

Strided Convolution
• Strides: increment step size for the convolution operator
• Reduces the size of the output map
▪ If you convolve an n x n image with f x f filter with padding p and
stride s then the output image size will be
𝑛+2p−𝑓 𝑛+2p−𝑓
▪ +1 x + 1
s s

91 100 83
69 91 127
44 72 74

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 16

16

8
 1/14/2024

Convolution on RGB Images

Colored image = height, width, channels
Convolutions are usually computed for each channel and summed

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 17

17

Convolution on RGB Images Cont’d

…..

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 18

18

9
 1/14/2024

The Filters

Edge

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 19

19

Multiple Filters
Horizontal Edge

Vertical Edge

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 20

20

10
 1/14/2024

Multiple Filters
Horizontal Edge across RGB

Horizontal Edge Feature for the image

(RGB Combined)

n x n x nc * f x f x nc
…. …. =
n – f+1 x n – f+1 x n’c

Vertical Edge Feature for the image

Vertical Edge across RGB (RGB Combined)

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 21

21

A Single Layer of CNN

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 22

22

11
 1/14/2024

Dimensions of a CNN

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 23

23

Example of a ConvNet

f [1] = 3 f [2] = 5 f [3] = 5

s [1] = 1 s [2] = 2 s [3] = 2
p [1] = 0 p [2] = 0 p [2] = 0 Output
10 filters 20 filters 40 filters Logistic/
Softmax
37 x 37 x 10 17 x 17 x 20 7 x 7 x 40
Input Image X nh[3] = nw[3]= 7
nh[1] = nw[1]= 37 nh[2] = nw[2]= 17
39 x 39 x 3 nc[3] = 40
nc[1] = 10 nc[2] = 20
nh[0] = nw[0]=39 1960
nc = 3 Fully Connected Layer

𝑛 + 2p − 𝑓 𝑛 + 2p − 𝑓
+1 x +1
s s

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 24

24

12
 1/14/2024

Types of Layers in CNN

▪ Convolution Layer
▪ Pooling Layer
▪ Fully Connected Layer

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 25

25

Pooling Layer: Max Pooling

Hyperparameters

No parameters to learn!

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 26

26

13
 1/14/2024

Pooling Layer: Max Pooling

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 27

27

Pooling Layer: Average Pooling

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 28

28

14
 1/14/2024

Hyper Parameters for Pooling Layer

▪f: filter size
▪s: Stride size
▪Max or Average Pooling

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 29

29

CNN Final Structure

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 30

30

15
 1/14/2024

Usually, a layer
Example: Digit Recognition of 400 neurons
in between

400

60,000 Parameters

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 31

31

CNN Parameters Count

Activation Shape Activation Size # Parameters
Input: 32, 32, 3 3,072 0
CONV1 (f=5, s=1) 28,28, 6 4,704 456
POOL 1 (f=2, s=2) 14, 14, 6 1,176 0
CONV2 (f=5, s=1) 10,10,16 1,600 2,416
POOL 2 (f=2, s=2) 5,5,16 400 0
FC3 120,1 120 48,120
FC4 84,1 84 10,164
Softmax 10,1 10 850

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 32

32

16
 1/14/2024

Why CNNs?

▪ Parameter sharing: A feature detector (such as a vertical edge detector) that’s

useful in one part of the image is probably useful in another part of the image.
▪ Sparsity of connections: In each layer, each output value depends only on a
small number of inputs.

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 33

33

Putting it all together

Use gradient descent to optimize parameters to reduce J

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 34

34

17
 1/14/2024

Usually, a layer
LeNet 5 of 400 neurons
in between

400

60,000 Parameters

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 35

35

AlexNet 60 Million Parameters

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 36

36

18
 1/14/2024

VGG-16

138 Million Parameters

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 37

37

Residual Networks

𝒂 𝒍+𝟐 = 𝒈 𝐳 𝒍+𝟐 + 𝒂 𝒍

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 38

38

19
 1/14/2024

Residual Networks

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 39

39

Why Residual Networks Work

▪ In the very deep plain nets (without residual blocks) making the
network deeper makes even the learning of an identity function
quite difficult. Therefore, adding a lot of layers, starts to produce
worse results instead of making them better.
▪ On the other hand, the residual networks with the residual
blocks:
• Can learn identity function
• Do not hurt performance
• And after a residual block, the gradient descent is capable to
improve further on.
▪ The main reason the residual network works is that it is so easy
for those extra layers to learn the residual.

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 40

40

20
 1/14/2024

1x1 Convolution

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 41

41

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 42

42

21
 1/14/2024

Pre-Trained Models
and Transfer Learning
Training a CNN model on ImageNet
from scratch takes days or weeks.
Many models trained on ImageNet
and their weights are publicly
available!

Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 43

43

Freeze
Train
Dr. Muhammad Usman Arif; Applied Neural Networks 1/14/2024 44

44

22