Convolutional Neural

Network (CNN)

Day 3
CNN Architectures Transfer Learning
 CNN
Architectures
CNN Architecture Decisions

➢ Number of Layers
➢ Number of filters
➢ Filter or Kernel Size
➢ Pooling
➢ Stride
➢ Fully Connected Layers
➢ Regularizers e.g. Batch Norm, Dropout
 What are the
Best Practices?

Review work in related domain and follow best practices..

 ILSVRC
(Imagenet Large Scale Visual Recognition Challenge)
What is ImageNet

- Large Image Dataset

- 14+ Million images

- ~22K Categories

- Human labeled

- ‘Describes’ the world around us

image-net.org
 Top 5% Error Rate

CNN Based

Accuracy measured on test dataset for 1000 Categories

AlexNet (2012)

➢ Reduced Error rate from 26% to 15%

○ A watershed moment in Computer Vision
➢ Used a Deep Architecture
➢ ReLU
➢ Dropout
➢ Data Augmentation
➢ Inference Augmentation
 AlexNet
SoftMax
FC 1000

FC 4096

FC 4096

Pool 3x3 S:2

- 5 Convolutional Layers
Conv 256 3x3, S:1, P:1

Conv 384 3x3, S:1, P:1 - 3 Max Pool Layers

Conv 384 3x3, S:1, P:1
- 3 Fully Connected Layers
Pool 3x3 S:2

Conv 256 5x5, S:1, P:2

- GTX580 , 5-6 Days

Pool 3x3 S:2

Conv 96 11x11, S:4, P:0

Input 227x227x3
 AlexNet
SoftMax
FC 1000

FC 4096

FC 4096
Conv 2
Conv 1
Pool 3x3 S:2 ? Overlapping
? ?
Input Image Max Pool 256
Conv 256 3x3, S:1, P:1 96
227x227x3 5x5
11x11 3x3
Stride=2 Stride = 1
Conv 384 3x3, S:1, P:1 Stride = 4
Padding = 2

Conv 384 3x3, S:1, P:1

Pool 3x3 S:2

Conv 256 5x5, S:1, P:2

Pool 3x3 S:2 - Output size : (N - F + 2P)/s + 1

Conv 96 11x11, S:4, P:0
- How many Weights to learn?
Input 227x227x3
 Overlapping Max Pool

1 4 5 2 7

5 3 6 3 6

7 2 1 1 4 - 3 x 3 Filter

- Stride 2
3 9 4 6 7

4 2 5 1 2
 Overlapping Max Pool

1 4 5 2 7

5 3 6 3 6

7 2 1 1 4 - 3 x 3 Filter

- Stride 2
3 9 4 6 7

4 2 5 1 2
 AlexNet - Overlapping Max Pool

1 4 5 2 7

5 3 6 3 6

7 2 1 1 4 - 3 x 3 Filter

- Stride 2
3 9 4 6 7

4 2 5 1 2
 Relu vs tanh

hyperbolic tangent max(0,x)

Relu helps with Vanishing Gradients issue

 Dropout

Dropout applied with Fully connected Layers

Data Augmentation

Horizontal Flip
Data Augmentation

Random Crop
Inference Augmentation

Multiple Images for

Prediction
Average
Output for
Model Prediction

Prediction Time Augmentation

Summary - AlexNet (2012)

➢ Deep Architecture with Convolutional Layers

➢ Trained on ImageNet
➢ Used ReLU instead of tanh
➢ Dropout with FC Layers
➢ Data Augmentation - Horizontal flips, Translations
➢ Trained on GPU
 Top 5% Error Rate

CNN Based

Accuracy measured on test dataset for 1000 Categories

 ZF Net
SoftMax
FC 1000

FC 4096

FC 4096

Pool 3x3 S:2

- Similar to AlexNet
Conv 512 3x3, S:1

Conv 1024 3x3, S:1 - Smaller filter size but more filters
Conv 512 3x3, S:1
- GTX580 , 11-12 Days
Pool 3x3 S:2

Conv 256 3x3, S:1

- Error rate of 11.7%

Pool 3x3 S:2

Conv 96 7x7, S:2

Input 224x224x3
 Building
Deeper Networks
 VGG (2014)

researchgate.net
 SoftMax
FC 1000
FC 4096
FC 4096
Pool 3x3
Conv 3x3, 512
All Conv filters : 3x3 stride 1 pad 1

Conv 3x3, 512

Conv 3x3, 512
Pool 3x3
Very simple architecture

Nvidia Titan 2-3 Weeks

Max Pool : 2x2 stride 2
VGGNet

Conv 3x3, 512

Conv 3x3, 512 Error rate of 7.3%
Conv 3x3, 512
Pool
Conv 3x3, 512
Conv 3x3, 256
Conv 3x3, 256
Pool
-

-
Conv 3x3, 128
Conv 3x3, 128
Pool
Conv 3x3, 64
Conv 3x3, 64
Input
 What should be Filter Size?

Smaller size
filter OR
Smaller
Receptive field

- Region that a CNN Filter gets to look at in the input is call

its Receptive Field
- Filter or Kernel capture Pixel Level interaction
Larger size
- What should be filter size? filter OR
Larger
Receptive field
How to achieve Larger Receptive field

➢ Larger Kernels e.g. 5x5, 7x7, 11x11

○ Downside -> More Weights

➢ Pooling

○ Downside -> Information Loss

➢ Using multiple layers of smaller filter e.g. 3x3

 5x5 Filter

x x x x x

5x5 Filter
With Relu

x
 Multi-layer 3x3 Filter

x x x x x

3x3 Filter
With Relu

x x x

3x3 Filter
With Relu

Additional non-linearity (Relu twice in 3x3 vs one in 5x5)

 How many Parameters?

Input Input
30x30x64 30x30x64

Conv 1 Conv 1
64, 3x3 64, 5x5,
S=1, Relu S=1, Relu

Conv 1
64, 3x3, S=1, Relu

Reduces Model Size

3x3x64xx64 + 5x5x64xx64
3x3x64x64 = 25x64x64
= 18x64x64
 Increasing Filters with Depth

- Initial layers capture low level

information e.g edges etc

- Later layers combine initial

features to learn for higher level
info

researchgate.net
 Ensembles

Model # 1

Average of
Model # 2 Multiple
Predictions

Model # n
Ensembles in VGG

- VGG16
- VGG19

Reduces Overfitting,
Improves Accuracy
Summary - VGG (2014)

➢ Use of only 3x3 filters

➢ Increasing filters with depth
➢ Using Ensembles to improve results
➢ Top-5 error rate 7.3%
Moving away from ‘Simple’
 SoftMax
FC
Avg Pool
9. Inception
No FC Layer except last one

8. Inception
Stacked Inception modules

7. Inception
9 Inception Modules
GoogLeNet

6. Inception
Error rate of 6.7%
5. Inception
4. Inception
3. Inception
2. Inception
1. Inception
-

-
Pool
Conv
Conv
Pool
Conv
Input
Convolution OR Pooling?

What Size Convolution?

 Using all the options

Concatenation

1x1 Conv 3x3 Conv 5x5 Conv 3x3 Max Pool

Previous Layer

Naive Inception module

But it does not work :(
 Naive Inception Module

28x28x(128+192+96+256)
=28x28x672

Depth-wise Number of Ops:

Concatenation
1x1 Conv : 28x28x128x1x1x256

28x28x128 28x28x192 28x28x96 28x28x256

3x3 Conv : 28x28x192x3x3x256

128 1x1 Conv 192 3x3 Conv 96 5x5 Conv 3x3 MaxPool, 5x5 Conv : 28x28x96x5x5x256
S: 1, P:0 S:1, P:1 S:1, P:2 S:1, P:1
Total: 854M

28x28x256

Input
Computationally very very complex
Power of 1x1 Convolution
28x28x256 28x28x32

1x1 Conv with

32 filters

Reduces depth
 Efficient Inception Module

Concatenation

3x3 Conv 5x5 Conv 1x1 Conv

1x1 Conv

1x1 Conv 1x1 Conv 3x3 Max Pool

Previous Layer
 Efficient Inception Module

28x28x480 Number of Ops:

Depth-wise
Concatenation 1x1 Conv : 28x28x128x1x1x256
1x1 Conv : 28x28x64x1x1x256
1x1 Conv : 28x28x64x1x1x256
28x28x192 28x28x96 28x28x64
3x3 Conv : 28x28x192x3x3x64
192 3x3 Conv 96 5x5 Conv 64 1x1 Conv
28x28x128 5x5 Conv : 28x28x96x5x5x64
128 1x1 Conv 1x1 Conv : 28x28x64x1x1x256
28x28x64 28x28x64 28x28x256 Total: 358M
64 1x1 Conv 64 1x1 Conv 3x3 MaxPool

28x28x256

Input
GoogLeNet Architecture
 Auxiliary Loss

➢ Calculate Loss for earlier Layers

➢ Combine Auxiliary Loss with Final Loss

➢ Why have Auxiliary loss?

○ Reduce Vanishing Gradient for earlier layers
 No Fully Connected Layer

Conv Conv

7 x 7 x 1024 7 x 7 x 1024
Earlier Network
approaches GoogLeNet
approach
FC Layer
Global Average
Pooling
1024

1024 1 x 1 x 1024

How many Weights?

 No Fully Connected Layer

Conv Conv

7 x 7 x 1024 7 x 7 x 1024
Earlier Network
approaches GoogLeNet
approach
FC Layer
Global Average
Pooling
1024

7 x 7 x 1024 x 1024 = 50M 0

Reduces Model size significantly

Summary - GoogLeNet (2014)

➢ Use of Inception Module

➢ 1 x 1 Convolution
➢ Auxiliary Loss
➢ Avoid FC Layers to reduce Size
➢ Global Average Pooling
How deep can we
really go?
Accuracy saturates and then
degrades
ILSVRC Winners

Deeper Deep?
Networks
ResNet (2015)

➢ 1st Place in ILSVRC 2015

➢ 1st Place in COCO Detection & Segmentation
➢ Replacing VGG-16 with ResNet 101 in Faster-RCNN improved results
by 28%
➢ Efficiently trained networks with 100 layers and 1000 layers
 ResNet
SoftMax
FC 1000

Pool

3x3 Conv 64

3x3 Conv 64
- Ultra deep : 152 Layers
Conv 128 3x3

Conv 128 3x3 - Residual blocks

- Error Rate 3.7%

Conv 128 3x3 - 8 GPUs , 2-3 Weeks

Conv 128 3x3

Pool

Conv 64 7x7

Input
 Residual Block

H(X relu
F(X) + X +
)

Conv Conv

X
relu relu

Conv Conv

X X

Regular Stacking Residual Block

 Residual Block

relu
F(X) + X +

H(x) = F(X) + X
Conv

X
relu F(x) = H(X) - X

Conv
Smaller value,
easier to Optimize
X

Residual Block
Summary - ResNet (2015)

➢ Residual Blocks with Skip connection

➢ Batch Normalization
➢ No Dropout
➢ No FC Layer
Deep CNNs require lots of Data
Transfer Learning
 Retrained

SoftMax
FC 1000
FC 4096
FC 4096
Pool 3x3
Conv 3x3, 512
Conv 3x3, 512
Conv 3x3, 512
Conv 3x3, 512
Pool 3x3
VGGNet

Conv 3x3, 512

Conv 3x3, 512
Conv 3x3, 512
Conv 3x3, 512
Frozen

Pool
Conv 3x3, 256
Conv 3x3, 256
Pool
Conv 3x3, 128
Conv 3x3, 128
Pool
Conv 3x3, 64
Conv 3x3, 64
Input
 Identifying Flowers

Daisy Roses

Dandelion

Tulips
Sunflowers
 Applying Transfer Learning

Daisy

Roses
Fully
Fully
Connected
Connected Dandelion
5
ResNet 200
(SoftMax)
(Frozen Layers) Tulips

Sunflowers

Flatten
Do we keep all Layers
Frozen?
 More Options

Similar to Original Different from Original

Small Dataset

for layer in model.layers: for layer in model.layers[:10]:

layer.trainable = False layer.trainable = False
Large Dataset

for layer in model.layers[:10]: for layer in model.layers:

layer.trainable = False layer.trainable = True