Scribd
Upload
3 views

NN BackProp (1)

The document outlines the structure and training process of multilayered neural networks, detailing the roles of input, hidden, and output layers. It describes the backpropagation algorithm, including the steps for error computation, weight updates, and the use of gradient descent for minimizing errors. Additionally, it explains the concept of momentum in weight updates to enhance learning stability and efficiency.

Uploaded by

srohithkanna
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
3 views

NN BackProp (1)

The document outlines the structure and training process of multilayered neural networks, detailing the roles of input, hidden, and output layers. It describes the backpropagation algorithm, including the steps for error computation, weight updates, and the use of gradient descent for minimizing errors. Additionally, it explains the concept of momentum in weight updates to enhance learning stability and efficiency.

Uploaded by

srohithkanna
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 34

Multilayered Network Architectures

Input layer Hidden layer Output layer

Linear neuron Sigmoidal neuron


Supervised Learning Procedure

Error information fed back for network adaptation

Xk Sk Error

Dx
Neural Network
Backpropagation Weight Update
Procedure
Procedure
1. Select a pattern Xkfrom the training set T present it to
the network.
2. Forward Pass: Compute activations and signals of input,
hidden and output neurons in that sequence.
3. Error Computation: Compute the error over the output
neurons by comparing the generated outputs with the
desired outputs.
4. Compute Weight Changes: Use the error to compute the
change in the hidden to output layer weights, and the
change in input to hidden layer weights such that a global
error measure gets reduced.
Backpropagation Weight Update
Procedure
Procedure
5. Update all weights of the network.

6.Repeat Steps 1 through 5 until the global error


falls below a predefined threshold.
Square Error Function
† The instantaneous summed squared error kεis
the sum of the squares of each individual
output error ejk, scaled by one-half:
Gradient Descent Procedure
Recall: Gradient Descent Update
Equation
Equation

† It follows logically therefore, that the weight


component should be updated in proportion
with the negative of the gradient as follows:
Neuron Signal Functions
† Input layer neurons are
linear.

† Hidden and output layer


neurons are sigmoidal.

† A training data set is


assumed to be given
which will be used to
train the network.
Notation
Notationfor
for
Backpropagation
Backpropagation
Algorithm Derivation
The General Idea Behind
Iterative Training…
Iterative Training…
† Employ the gradient of the pattern error in order
to reduce the global error over the entire training
set.
† Compute the error gradient for a pattern and use
it to change the weights in the network.
† Such weight changes are effected for a sequence
of training pairs (X 1,D 1), (X2,D2), . . . , (Xk ,Dk ), . . .
picked from the training set.
† Each weight change perturbs the existing neural
network slightly, in order to reduce the error on
the pattern in question.
Square Error Performance
Function
Function
† The k thtraining pair (Xk,Dk) then defines the
instantaneous error:
„ E k = Dk − S(Yk) where
„E= k (e 1, . . . , e p )
k k

„ = (d − 1 S(y 1 ), . . . , dp − S(yp ))
k k k k

† The instantaneous summed squared error kEis


the sum of the squares of each individual
output error ejk, scaled by one-half:
The Difference Between Batch and
PatternUpdate
Pattern Update
Derivation of BP Algorithm:
Forward Pass-Input
Forward Pass-Input Layer
Layer
Derivation of BP Algorithm:
Forward Pass-Hidden Layer
Derivation of BP Algorithm:
Forward Pass-Output Layer
Recall the Gradient Descent
Update Equation
Update Equation
† A weight gets
updated based on
the negative of the
error gradient with
respect to the
weight
Derivation of BP Algorithm:
Computation
Computation ofof Gradients
Gradients
Derivation of BP Algorithm:
Computation
Computation ofof Gradients
Gradients
Derivation of BP Algorithm:
Computation
Computation ofof Gradients
Gradients
Derivation of BP Algorithm:
Computation
Computation ofof Gradients
Gradients
Derivation of BP Algorithm:
Computation
Computation ofof Gradients
Gradients
GeneralizedDelta
Generalized DeltaRule:
Rule:Momentum
Momentum
† Increases the rate of learning while
maintaining stability
How Momentum Works
† Momentum should be less than 1
for convergent dynamics.
† If the gradient has the same
sign on consecutive iterations
the net weight change increases
over those iterations
accelerating the descent.
† If the gradient has different
signs on consecutive iterations
then the net weight change
decreases over those iterations
and the momentum decelerates
the weight space traversal. This
helps avoid oscillations.
Derivation of BP Algorithm:
Finally…!
Finally…!
Backpropagation Algorithm:
Operational Summary
Operational Summary
Backpropagation Algorithm:
Operational Summary(contd.)
Summary(contd
.)
Hand-worked
Hand- Example
worked Example
Forward Pass 1/Backprop Pass 1
Weight Changes: Pass 1
Network N2 after first Iteration
Forward Pass 2/Backprop Pass 2
Weight Changes: Pass 2
Network
NetworkNN3after second Iteration
Activation functions: https://towardsdatascience.com/activation-
functions-neural-networks-1cbd9f8d91d6

You might also like