Module - 3

Deep Learning and Computer

Vision with OpenVINO Toolkit
Units for Discussion

Fundamentals of Computer Vision Computer Vision

Deep Learning with OpenVINO

Unit - 1 Unit - 2 Unit - 3

Introduction to
Gen-AI

Unit - 4
 Unit - 1

Fundamentals of Deep Learning

 DISCLAIMER
The content is curated from online/offline resources
and used for educational purpose only.
Learning Objectives
• Deep Learning
• Deep Neural Network
• DL Vs ML
• Forward and Back Propagation
• Hyperparameters in Neural Networks
• Multilayer Perceptron

Source :
www.freepik.com/
What is Deep Learning?

• Also known as deep structured learning

• Part of a broader family of machine learning methods
• Based on artificial neural networks
• Learning can be supervised, semi-supervised or
unsupervised.​

Source :
Reference link
Machine Learning Vs Deep Learning​

Source :
Reference
Deep Neural Networks(DNN)​

• A hierarchical organization of neurons with

connections to other neurons is a simplified
representation of a DNN.
• Deep Neural Networks gained their name from the
fact that they utilized a lot of hidden layers to learn
more intricate patterns, giving them the label "deep".
• Based on the received input, these neurons send a
message or signal to other neurons, forming a
complex network that learns through a feedback loop.

Source :
Reference link
Neurons

• Neurons in deep learning models are nodes through

which data and computations flow.
• Neurons receive one or more input signals.
• These input signals can come from either the raw
data set or from neurons positioned at a previous
layer of the neural net.

Source :
Reference link
Weights and Bias

Input 1

• Weights enable the artificial neural network to dial up Bias

or dial down connections between neurons.
Weight 1
• Bias can be used to make adjust-
ments within neurons.
Input 2 Output
• Bias can be positive or negative, increasing or
decreasing a neuron’s output
Weight 2

Source :
Reference link
Components of Artificial Neuron

Source :
Reference link
Single Neuron

Source :
Reference link
Representation of DNN
Forward Propagation

• Forward propagation is how neural networks make

predictions. Input data is “forward propagated”
through the network layer by layer to the final layer
which outputs a prediction.

• Forward propagation (or forward pass) refers to the

calculation and storage of intermediate variables
(including outputs) for a neural network in order from
the input layer to the output layer.
Backward Propagation

• In machine learning, backpropagation is a widely used

algorithm for training feedforward neural networks.
Generalizations of backpropagation exist for other
artificial neural networks (ANNs), and for functions
generally. These classes of algorithms are all referred
to generically as "backpropagation".
• In fitting a neural network, backpropagation computes
the gradient of the loss function with respect to the
weights of the network for a single input–output
example, and does so efficiently, unlike a naive direct
computation of the gradient with respect to each weight
individually.

Source :
Reference link
Working of ANN
Epochs

• One Epoch is when an ENTIRE dataset is passed

forward and backward through the neural network
only ONCE.
• Since one epoch is too big to feed to the computer at
once we divide it in several smaller batches.

Batch Size
• Total number of training examples present in a single
batch.
• Note: Batch size and number of batches are two
different things.
Hyperparameters in Neural Network

• Hyperparameters are the variables.

• It determines the network structure
Example: Number of Hidden Units and Learning Rate.
• Hyperparameters are set before training(before optimizing the weights and bias).
Hyperparameters related to Network Structure

Number of Hidden Layers and Units

• Hidden layers are the layers between input layer

and output layer.

• “Very simple. Just keep adding layers until the test

error does not improve anymore.”

• Many hidden units within a layer with

regularization techniques can increase accuracy.
Smaller number of units may cause underfitting.
Hyperparameters related to Network Structure

Dropout

• Random neurons are cancelled

• Dropout is regularization technique to avoid overfitting

(increase the validation accuracy) thus increasing the
generalizing power.

• Generally, use a small dropout value of 20%-50% of

neurons with 20% providing a good starting point. A
probability too low has minimal effect and a value too
high results in under-learning by the network.
Hyperparameters related to Network Structure

Activation Function

• Activation functions are used to introduce nonlinearity to models, which allows deep learning
models to learn nonlinear prediction boundaries.
• The Activation Functions can be basically divided into 2 types-
1. Linear Activation Function
2. Non-linear Activation Functions
• Several different types of Activation functions are used in Deep Learning.
• Few examples are Sigmoid function, Tanh function, ReLu function, Leaky ReLu function, Maxout
function, ELU function, etc.
Hyperparameters related to Network Structure

Sigmoid Activation Function

● Sigmoid function takes any real value as

input and outputs values in the range of 0 to
1.
● It is commonly used for models where
we have to predict the probability as an
output. Like Binary Classification problem
where the output is between 0 and 1.

Source :
Reference link
Hyperparameters related to Network Structure

Tanh or hyperbolic tangent Activation Function

● Tanh function takes any real value as input

and outputs values in the range of -1 to 1.
● The output of the tanh activation function is
Zero centered; hence we can easily map
the output values as strongly negative,
neutral, or strongly positive.

Source :
Reference link
Hyperparameters related to Network Structure

ReLU (Rectified Linear Unit) Activation Function

● ReLU function takes any real value as input

and outputs values in the range of 0 to ∞
with the formula f(x) = max(0, x).
● ReLU function does not activate all the
neurons at the same time. Since only a
certain number of neurons are activated,
the ReLU function is far more
computationally efficient when compared to
the sigmoid and tanh functions and are used
heavily in hidden layers.
Source :
Reference link
Hyperparameters related to Network Structure

Leaky ReLU Activation Function

● Leaky ReLU function takes any real value as

input and outputs values in the range of -∞
to ∞ with the formula f(x) = max(0.1* x, x).
● Leaky ReLU is an improved version
of ReLU function to solve the
dying ReLU problem which refers to the
problem when ReLU neurons becoming
inactive and only outputs 0 for any input.

Source :
Reference link
Hyperparameters related to Network Structure

Maxout Activation Function

● The Maxout activation is a generalization of the ReLU and the leaky ReLU functions.
● It is used in conjunction with the dropout regularization technique.

Source :
Reference link
Hyperparameters related to Network Structure

ELU Activation Function

● ELU is a function that tends to converge

faster and produce more accurate results.
● Capture negative value bet-
ter than ReLU activation function.

Source :
Reference link
Vanishing Gradient Descent

● Vanishing Gradient Descent is a challenge that emerges during backpropagation when the
derivatives or slopes of the activation functions become progressively smaller as we move
backward through the layers of a neural network.
● The vanishing gradient problem is particularly associated with the sigmoid and hyperbolic
tangent (tanh).
● We can fix this problem using different methods. Some methods includes Batch Normalization,
choosing right activation functions. For eg, choosing Activation function like Rectified Linear
Unit (ReLU) for the hidden layers in neural network.
Hyperparameters related to Training Algorithm

Learning Rate

• The learning rate defines how quickly a network updates

its parameters.

• Low learning rate slows down the learning process but

converges smoothly. Larger learning rate speeds up the
learning but may not converge.

• Usually, a decaying Learning rate is preferred.

Gradient Descent

• Gradient Descent is known as one of the most used optimization algorithms to train machine learning
models by means of minimizing errors between actual and expected results.
• Gradient descent was initially discovered by "Augustin-Louis Cauchy" in the mid 18th century.
• Gradient Descent is defined as one of the most used iterative optimization algorithms of machine
learning to train the machine learning and deep learning models. It helps in finding the local minimum
of a function.

The best way to define the local minimum or local maximum of a function using gradient descent
is as follows:
• If we move towards a negative gradient or away from the gradient of the function at the current point,
it will give the local minimum of that function.
• Whenever we move towards a positive gradient or towards the gradient of the function at the current
point, we will get the local maximum of that function.
Gradient Descent

Source :
Reference link
Gradient Descent

Source :
Reference link
Hyperparameters related to Training Algorithm

Number of epochs

Number of epochs is the number of times the whole training data is shown to the network while training.

Increase the number of epochs until the validation accuracy starts decreasing even when training accuracy is
increasing(overfitting).

Batch size

Mini batch size is the number of sub samples given to the network after which parameter update happens.

A good default for batch size might be 32. Also try 32, 64, 128, 256, and so on.
Loss Functions

A loss function, that can be used to estimate the 1. Binary Classification Loss Functions
loss of the model so that the weights can be • Binary Cross-Entropy
updated to reduce the loss on the next evaluation.
• Hinge Loss
• Regression Loss Functions
• Squared Hinge Loss
• Mean Squared Error Loss

• Mean Squared Logarithmic Error Loss

2. Multi-Class Classification Loss Functions
• Mean Absolute Error Loss
• Multi-Class Cross-Entropy Loss

• Sparse Multiclass Cross-Entropy Loss

• Kullback Leibler Divergence Loss

Multilayer Perceptron

• Multilayer Perceptron(MLP) is a neural network that

has multiple layers which transform any input
dimension to the desired dimension.

• MLPs can be used to solve complex nonlinear

problems. Their ability to learn nonlinear relationships
in data makes them suitable for tasks such as
classification, regression and pattern recognition.

• MLPs handle both small and large input data

effectively. This flexibility allows them to work well
with various datasets.
Source :
Reference link
Advantages of DL over ML​

• DL models do not require any pre-processing for

feature extraction and are capable of categorizing
data into multiple classes and categories on their
own.

• Raw Data is given as an input to DL model.

• Pre-processed data is given to ML model.

• After a certain point, the accuracy of ML models

stops rising with more data, but the accuracy of
DL models continues to increase with increasing
data.
Source :
Reference link
TensorFlow 2.0 & Keras API

Tensor

• TensorFlow 2.0 is a library that provides a

comprehensive ecosystem of tools for developers,
researchers, and organizations who want to build
Machine language
scalable Machine Learning and Deep Learning Face detection in translation through
Fraud detection in
the banking and
Object detections
electronic devices. apps such as on videos.
financial sectors.
Google Translate.
applications.

• TensorFlow Applications

Syntax:
import tensorflow as tf
Building a model in Keras
1
Define Network

2
Compile Network

3
Fit Network

4
Evaluate Network

5
Make Predictions
What is Keras?

• Keras is a deep learning API written in Python, running on top of the machine learning platform TensorFlow.
• It was developed with a focus on enabling fast experimentation. Being able to go from idea to result as fast
as possible is key to doing good research.

Source :
Reference link
 Lab - 1

First Neural Network with Keras

 Conclusion
• So we have learned what is Deep Learning , and how it is related to AI and how it is different
from AI and ML
• What is deep Neural Network, neurons and components of Artificial neuron
• Forward and Backward Propagation
• Then we have learned Hyperparameter related to Network structure
• Advantage of DL over ML
• And at last Neural network using keras
References
• https://www.javatpoint.com/deep-learning-algorithms
• https://www.guru99.com/deep-learning-tutorial.html
• https://www.tutorialspoint.com/python_deep_learning/index.htm
• https://www.datacamp.com/tutorial/tutorial-deep-learning-tutorial
• https://towardsdatascience.com/what-are-hyperparameters-and-how-to-tune-the-
hyperparameters-in-a-deep-neural-network-d0604917584a
Let’s Start
Quiz

1. Which of the following is not a common activation

function used in deep neural networks?

a) ReLU (Rectified Linear Activation)

b) Sigmoid
c) Tanh (Hyperbolic Tangent)
d) Mean Squared Error

Answer: D
Mean Squared Error
Quiz

2. In deep learning, what is backpropagation used for?

a) Initializing neural network weights

b) Calculating the loss function
c) Updating neural network weights
d) Determining the number of hidden layers

Answer: C
Updating neural network weights
Quiz

3. What is an epoch in the context of training deep neural

networks?

a) A single forward pass through the network

b) A single backward pass through the network
c) One complete iteration through the training dataset
d) The number of layers in the neural network

Answer: C
One complete iteration through the training dataset
Quiz

4. Which deep learning technique is used to prevent the

vanishing gradient problem in very

a) Regularization
b) Batch normalization
c) Data augmentation
d) Max-pooling

Answer: B
Batch normalization
Quiz

5. What is the primary objective of deep learning?

a) Feature engineering
b) Data visualization
c) Automatic feature learning
d) Data preprocessing

Answer: C
Automatic feature learning
Thank You