A Gentle Introduction to Backpropagation

Shashi Sathyanarayana, Ph.D

July 22, 2014

Contents
1 Why is this article being written?

2 What is so difficult about designing a neural network?

3 Backpropagation

4 Easy as 1-2-3
4.1 Boxing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Weight updates . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4
4
5
9

5 The miraculous feat of cognition

6 Backpropagation Algorithm Outline. Training Wheels for Training Neural Networks

11
6.1 Summary of the backpropagation algorithm . . . . . . . . . . . . 12
6.2 Details of the backpropagation algorithm . . . . . . . . . . . . . 13
7 About the author

14

8 LEGAL NOTICE

15

Why is this article being written?

Neural networks have always fascinated me ever since I became aware of them
in the 1990s. I was initially drawn to the hypnotizing array of connections with
which they are often depicted. In the last decade, deep neural networks have
dominated pattern recognition, often replacing other algorithms in applications
c
Numeric
Insight, Inc. All rights reserved. See LEGAL NOTICE on page 15.

Figure 1: An example of a multi-layered neural network that can be used to

associate an input consisting of 10 numbers with one of 4 decisions or predictions.
like computer vision and voice recognition. At least in specialized tasks, they
indeed come close to mimicking the miraculous feats of cognition our brains are
capable of.
While neural networks are capable of such feats, the very discovery of a
method of programming such a computational device is to me, in itself, a miraculous feat of cognition worthy of celebration. My purpose in writing this article
is to share my perspective on an amazing algorithm made widely known by a
1986 publication in Nature.
I will assume that the reader has some understanding of the meaning and
purpose of the various elements of a neural network such as the one shown in
Figure 1. I have provided a little bit of background below. Other than that, this
article should be an easy read for those with some familiarity of basic calculus.

What is so difficult about designing a neural

network?

To appreciate the difficulty involved in designing a neural network, consider

this: The neural network shown in Figure 1 can be used to associate an input
consisting of 10 numbers with one of 4 decisions or predictions. For example,
the neural network shown may be used by a bank to determine if credit should
be extended to a customer. In this case, the 10 input numbers represent various
parameters relevant to an individuals financial responsibility, such as balance
in savings accounts, outstanding loan amounts, number of years of employment,
and so on. The neural network takes in these 10 numbers, performs calculations
and produces an output consisting of 4 numbers. Depending on the position at
which the maximum of these 4 numbers appears in the output, the prediction
could be one of the following:
1. Excellent creditworthiness with high spending limit
2. Average creditworthiness with moderate spending limit
2

3. Low creditworthiness with low spending limit

4. High default risk.
Based on this prediction, the bank would take the appropriate decision.
A neural network essentially like the one shown in Figure 1 can perform this
miraculous feat of cognition only if it is specifically trained to do so. For the network to work correctly, the weight at each of its (104)+(46)+(68)+(84)
=144 connections have to be carefully chosen such that the network classifies
every input drawn from a known training set with a high degree of accuracy.
This is a classic application of the supervised learning paradigm in machine
learning.
There are, of course, no formulas in existence to directly set the values of
the 144 weights. The only recourse is to start with some initial values for the
144 weights, check how good the resulting neural network is, and repeatedly
refine the weights to progressively make the network more and more accurate.
So, what is needed is a method of refining the weights.
If one were to think of the accuracy as some grand function of the weights,
it makes sense to refine each weight by changing it by an amount proportional
to the partial derivative of that grand function with respect to that weight.
Why bring in partial derivatives? Because they, by definition, predict how the
accuracy responds to small changes in the weights. In fact, at every iteration,
performing a refinement guided by the partial derivatives results in a more
advantageous gain in accuracy compared to any other method of refinement.
The method of steepest descent does exactly what is suggested here with
the minor, but entirely equivalent, goal of seeking to progressively decrease the
error, rather than increase the accuracy.
To keep things straight, let me list the concepts I have described thus far:
1. The weights of the neural network must be set such that the error calculated on a known training set is minimized. Ideally, the network must
yield zero error on a large and representative training data set.
2. We adopt the following strategy in order to arrive at the weights that minimize error: Given an arbitrary choice of weights, refine them by changing
each weight by a small amount proportional to the partial derivative of
the error with respect to that weight. The partial derivatives themselves
change after any such refinement, and must be recomputed before the next
round of refinement can be applied.
3. We can keep on refining the weights in this manner, and stop refining
when the error is zero. In real life, we call it done when the error is low
enough. Or refuses to fall anymore. Or we run out of time after a few
million rounds of refinements.
Seems simple, right? Yes. As long as there is a simple way of calculating the
partial derivative of error with respect to every single weight, at every single
3

iteration. In principle, the partial derivatives can be calculated by systematically

perturbing the weights and measuring the resulting changes in error. A word to
the wise, dont try this at home. Or even on your neighborhood supercomputer.

Backpropagation

In fact, this error minimization problem that must be solved to train a neural
network eluded a practical solution for decades till D. E. Rumelhart, C. E. Hinton, and R. J. Williams (drawing inspiration from other researchers) demonstrated a technique, which they called backpropagation, and made it widely
known (Nature 323, 533-536, 9 October 1986). It is essentially by building upon
their method that today others have ventured to program neural networks with
60 million weights, with astounding results.
According to Bernard Widrow, now Professor Emeritus at Stanford University and one of the pioneers of neural networks, The basic concepts of backpropagation are easily grasped. Unfortunately, these simple ideas are often obscured
by relatively intricate notation, so formal derivations of the backpropagation
rule are often tedious. This is indeed unfortunate because the backpropagation
rule is one of the most elegant applications of calculus that I have known.

Easy as 1-2-3

Once you appreciate the fact that, in order to train a neural network, you need
to somehow calculate the partial derivatives of the error with respect to weights,
backpropagation can be easily and qualitatively derived by reducing it to three
core concepts. It also helps immensely to keep the notation intuitive and easy
to connect to the concept being symbolized.

4.1

Boxing

Since training the neural network is all about minimizing the training error, the
first step in the derivation involves tacking on an extra computational block to
calculate the error between the actual output {o1 , o2 , o3 , o4 } and a known target
{t1 , t2 , t3 , t4 }. This is shown as a triangular block in Figure 2. For now, let us
think of the output and the target as known and fixed entities. Although we
need not concern ourselves with the exact formula to compute the error, I offer
the familiar sum-of-squares error as an example
e = (o1 t1 )2 + (o2 t2 )2 + (o3 t3 )2 + (o4 t4 )2
Next, we choose one of the layers (say Layer 3) and enclose that layer and
all following layers (including the error calculating block) in a box, as shown
in gray in Figure 2. Keep in mind that this is just one of several nested boxes
we can construct in order to compartmentalize the network. Let us resolve
not to worry about anything going on inside the box but simply think of the

Figure 2: The derivation of the backpropagation algorithm is simplified by

adding an extra computational block to calculate the error and also by boxing
parts of the network. Compare with Figure 1.
relationship between the input to the box and the output (i.e the error) coming
out of the box. We will call this box the current box, and call the input to this
box the current input, {c1 , c2 , c3 , c4 , c5 , c6 }. It is important to recognize that
the functional relationship between the current input to the box and the output
emanating from the box is completely defined and can be computed. Let us
denote this relationship as e = E(c1 , c2 , c3 , c4 , c5 , c6 ).

4.2

Sensitivity

As our journey through backpropagation continues, I gently request you to

E E E
assume that the vector of partial derivatives c
,
,
, . . . of the function
1 c2 c3
E(c1 , c2 , c3 , c4 , c5 , c6 ) is known. This might seem like asking for too much! After
all, we have set out to find a method to compute some other (equally unrealized)
partial derivatives. But I assure you it will all work out in the end. To emphasize
the crucial nature of this simple concept, it has been given a name: Sensitivity.
Let us denote the sensitivity of our current box as {c1 , c2 , c3 , c4 , c5 , c6 .}
Remember, sensitivity is a vector, not a single number.
With the help of Figure 3 to hold these concepts in our mind, we can concretely think about how the output of the current box responds to a small
perturbation applied to any of its current inputs. For example, if the fourth
component of the current input changes by the small amount c4 , we can expect the error at the output to change by c4 c4 . Further, in addition to the
hypothetical change in component 4, if there is a simultaneous change of c6
in component 6, we can expect the error at the output to change by an additional amount, making the total change c4 c4 + c6 c6 . The effect of small
simultaneous changes in the current input components simply add up at the
output.
Knowing the sensitivity of the current box, what can we say about the
sensitivity of the preceding box? Keep in mind that the preceding box encloses
Layer 2 and all following layers, including the error calculating block. For our
5

Figure 3: This diagram shows the boxed portion of the neural network of Figure 2. By hiding details of internal connections within the box, this scheme
allows us to think about the broad relationship between the input and the output pertaining to this box.
specific example, let us call the input to the preceding box the preceding input,
{p1 , p2 , p3 , p4 .} It follows quite logically that the sensitivity of the preceding box
(which we will naturally denote as {p1 , p2 , p3 , p4 }) must be related to the
sensitivity of the current box and the extra neural network elements making up
the difference between the two nested boxes. The extra elements are the very
vital nonlinear activation function units, summing junctions and weights.
Figure 4 (top) shows the current box and the extra elements that must be
added to construct the preceding box. For clarity, all the elements not relevant
to the calculation of the first component of sensitivity (p1 ) have been grayed
out. Look closely at the top and bottom parts of Figures 4 to understand how
the sensitivity of the preceding box can easily be derived from first principles.
e
Specifically, the bottom part of Figure 4 provides insight into how p1 (= p
)
1
can be computed by allowing the input component p1 to change by a small quantity p1 and following the resulting changes in the network. Notes: (i) The notation A0 (p1 ) has been used for the derivative of the activation function evaluated
at p1 . (ii) For clarity, not all changes in signals have been explicitly labeled.
Those that are not labeled can easily be determined since they all follow an
obvious pattern.
This is indeed a deep result. (And one which we have managed to arrive
at without recourse to tedious calculus and numbered equations.) By repeated
application, this result allows us to work backwards and calculate the sensitivity
of the error to changes in the input at every layer.
The algorithm gets the name backpropagation because the sensitivities are
6

Figure 4: The drawing on the top shows the current box and the extra elements
that must be added to construct the preceding box. The drawing on the bottom
e
provides insight into how the sensitivity component p1 (= p
) can be computed
1
by allowing the input component p1 to change by a small quantity p1 and
following the resulting changes in the network.

Figure 5: Similar to Figure 4, the drawing on the top shows the current box
and the extra elements that must be added to construct the preceding box. The
e
drawing on the bottom provides insight into how the partial derivative w
11
(used to guide weight refinement) can be computed by allowing the weight w11
to change by a small quantity w11 and following the resulting changes in the
e
network. Using this intuition, w
can be computed as A(p1 )c1 .
11

propagated backwards as they are calculated in sequence. The textbook formula

to express the sensitivity of the preceding layer in terms of the sensitivity of the
current layer is easily seen to be
X
pi = A0 (pi )
wij cj
j

A starting point is all we need to completely calculate all the sensitivity terms
throughout the neural network. To do this, we consider the error computing
block itself as the first box. For this box, the input is {o1 , o2 , o3 , o4 }, and the
output is e as given in the sum-of-squares error formula we have seen before.
Simple calculus gives us the components of the sensitivity of the error computing
block
{2 (o1 t1 ) , 2 (o2 t2 ) , 2 (o3 t3 ) , 2 (o4 t4 )}

4.3

Weight updates

At this point, the last section writes itself. Following the same strategy outlined
in the previous figure, look at Figure 5 to intuitively understand how the error
changes in response to a small change in one of the weights, say w11 . Once
again in these figures, details of connections not immediately relevant to this
calculation have been grayed out. The much sought after partial derivative
of error with respect to the specific weight w11 is easily seen to be A(p1 )c1 .
Generalizing on this, the textbook formula to compute the partial derivative of
the error with respect to any weight is easily seen to be
e
= A (pi ) cj
wij
Now that we have a formula to calculate the partial derivative of error with
respect to every weight in the network, we can proceed to iteratively refine the
weights and minimize the error using the method of steepest descent.
In the most popular version of backpropagation, called stochastic backpropagation, the weights are initially set to small random values and the training set
is randomly polled to pick out a single input-target pair. The input is passed
through the network to compute internal signals (like A(p1 ) and A0 (p1 ) shown
in Figures 4 and 5) and the output vector. Once this is done, all the information
needed to initiate backpropagation becomes available. The partial derivatives
of error with respect to the weights can be computed, and the weights can be
refined with intent to reduce the error. The process is iterated using another
randomly chosen input-target pair.

The miraculous feat of cognition

I am in awe of the miraculous feat of cognition that lead early neural network
researchers to arrive at the backpropagation algorithm. They clearly had the

ability to see patterns and make elegant groupings which ultimately made it
possible to train huge networks. Their work not only resulted in the neural
network applications we use today, but have also inspired a host of other related
algorithms which depend on error minimization.
Although this algorithm has been presented here as a single established
method, it should be regarded as a framework. In my experience, appreciating
how an algorithm is derived leads to insight which makes it possible to explore
beneficial variations. The process of designing robust and efficient scientific
algorithms frequently leads us to regard established frameworks as substrates
upon which to build new and better algorithms.

The algorithm outline which follows on the next page has

been provided to make the concepts presented in this article
more concrete, and enable a working multi-layer neural network to be actually programmed and trained in your favorite
language. Note that, although the notation used in the preceding pages no longer applies to the algorithm outline on
the next pages, we continue to keep the notation intuitive
and make it easy to connect variable names to concepts. A
table of notations is provided. Also note that the addition
of bias terms results in a slight change in the architecture
of the layers. This is an important consideration because
the use of bias terms makes the training process converge
faster. We hope that enabling encouraging results in early
experimentation will lead one to try out more complex configurations of the algorithm on practical applications or new
and interesting problems.

10

Backpropagation Algorithm Outline. Training Wheels for Training Neural Networks

Notation
General Symbols
L
l
Nl
Iil

N L+1
Oj
A()
A0 ()
l
wij

il
tj
l
gij

r
I

Number of layers
Layer index
Number of input components going into layer l.
ith component of the input going into layer l. Note that
i = {0, 1, 2, . . . , N l }. See Figure 6 to understand why i starts
from 0.
Number of output components coming out of layer L (i.e. the
last layer, or output layer.)
j th component of the output coming out of layer L. Note that
j = {1, 2, 3, . . . , N L+1 }
The activation function. For example, A(I32 ) = tanh(I32 )
The derivative of the activation function. For example,
A0 (I32 ) = 1 tanh2 (I32 )
The weight element connecting the ith input to the j th output in layer l. Note the ranges l = {1, 2, 3, . . . , L}, i =
{0, 1, 2, . . . , N l }, and j = {1, 2, 3, . . . , N l+1 .}
ith component of the sensitivity measured at the input to layer
l. Note that i = {1, 2, 3, . . . , N l }
j th component of the target corresponding to the chosen input.
Note that j = {1, 2, 3, . . . , N L+1 }
The partial derivative of the error e with respect to the weight
l
wij
. Note that we have chosen the symbol g to suggest gradient.
The learning rate.
Training Set
Training set input. Note that I is a matrix whose rows are
selected randomly during training to extract vectors of the form
1
{I11 , I21 , I31 , . . . , IN
1 } that feed into the network
Training set target. Note that at every instance of selection
of a row within I, the corresponding row of T is selected to
extract a vector of the form {t1 , t2 , t3 , . . . , tN L+1 }. It is this
vector that is compared with the actual output of the network
during training iterations
Table continues on next page . . .

Superscripts are used to index layers as in, for example, I23 . This is an intuitive notation
and it should be clear from context if a quantity is actually being raised to a power.

11

. . . table continued from previous page

Stopping Criteria
gMin The threshold value of gradient
k
Iteration index
kMax Maximum number of iterations allowed

Figure 6: The modified structure of Layer 2 of the multi-layer neural network

shown in previous figures. The addition of bias results in a slight change in the
architecture, and a slight change in the formulas. The input to each layer is
augmented by adding an extra bias element I0l , equal to 1 by definition. The
use of bias generally results in a faster convergence of the training process. See
additional notes in Step 3.

6.1

Summary of the backpropagation algorithm

The overall goal of the algorithm is to attempt to find weights such that, for every input vector in the training set, the neural network yields an output vector
closely matching the prescribed target vector.

12

Step
Step
Step
Step
Step
Step

1
2
3
4
5
6

6.2

Initialize the weights and set the learning rate and the stopping criteria.
Randomly choose an input and the corresponding target.
Compute the input to each layer and the output of the final layer.
Compute the sensitivity components.
Compute the gradient components and update the weights.
Check against the stopping criteria. Exit and return the weights or loop back to Step 2.

Details of the backpropagation algorithm

l
to small random values between -1 and +1.
1. Initialize the weights wij
Set the learning rate and stopping criteria. Try the following settings for
initial experimentation

= 102

gMin = 103
k

=0

kMax = 105
1
2. Randomly choose (with replacement) an input vector {I11 , I21 , I31 , . . . , IN
1}
from I and the corresponding target {t1 , t2 , t3 , . . . , tN L+1 } from T .
l
3. Compute the input to each layer, {I1l , I2l , I3l , . . . , Ijl , . . . , IN
l }, and output
{O1 , O2 , O3 , . . . , Oj , . . . , ON L+1 } coming out of layer L.
1

Ij2

N
X

Ii1

1
wij

for j = 1, 2, 3, . . . , N 2 .

i=0

Ijl =

l1
N
X

l1
A(Iil1 ) wij

for l = 3, 4, 5, . . . , L,
for j = 1, 2, 3, . . . , N l .

i=0
L

Oj =

N
X

L
A(IiL ) wij

for j = 1, 2, 3, . . . , N L+1 .

i=0

Note that bias (see Figure 6) is implemented by allowing the index i to

start from 0 in the above summations. Terms such as I01 , A(I0l1 ), and
A(I0L ) must be set to 1.
l
4. Compute the sensitivity components {1l , 2l , 3l , . . . , il , . . . , N
l}

iL+1 = 2(Oi ti )
il

= A (Iil )

l+1
N
X

j=1

for i = 1, 2, 3, . . . , N L+1 .

l l+1
wij
j

for l = L, L 1, L 2, . . . , 2,
for i = 1, 2, 3, . . . , N l .

13

l
5. Compute the gradient components gij
(partial derivative of error with rel
spect to weights) and update the weights wij

e
for l = 1, 2, 3, . . . , L,
l l+1
l

=
A(I
)
gij
=
i j
l
wij
for i = 0, 1, 2, . . . , N l ,

l
l
l
wij = wij r gij
for j = 1, 2, 3, . . . , N l+1

Note that terms such as A(I0l ) must be set to 1.

6. Check against the stopping criteria
(a) Increment iteration index, k = k + 1.
If k > kMax , EXIT and return weights. Else,
(b) Check for convergence
l
If |gij
| gMin for evey l = {1, 2, 3, . . . , L}, i = {0, 1, 2, . . . , N l }, and
j = {1, 2, 3, . . . , N l+1 }, EXIT and return the weights. Else,
loop back to Step 2.

About the author

Shashi Sathyanarayana Ph.D, the founder of Numeric Insight, Inc (www.numericinsight.com) has several decades of expertise in scientific programming, algorithm development and teaching.
At Boston Scientific Corp, he was responsible for the
creation of algorithms that power the companys imaging
devices employed to assist detection of atherosclerosis.

He is a named inventor on about a dozen patents in machine learning and

ultrasound imaging. His interest in medical imaging began when he was a postdoctoral fellow with Professor Richard Gordon, one of the pioneers of Computed
Tomography algorithms. It was further shaped by a research fellowship at the
Cleveland Clinic Foundation where he contributed to advancements in echocardiography at the world-renowned Heart Center.
Shashi is passionate about data, algorithms, and teaching. He is an active
developer of products and creates highly efficient and effective algorithms. He
enjoys creating technical vision and actively engages in understanding customer
needs.
Shashis commitment to judiciously apply common sense and advanced analysis to improve products and processes earned him the Six Sigma Black Belt
certification by the American Society for Quality.
Although frequently called upon to work on complex algorithms, he believes
in the value of keeping it basic; he recognizes the universal customer need for
intuitiveness and simplicity in a product.
Email comments to [email protected]

14

LEGAL NOTICE

This material is the property of Numeric Insight, Inc and has been provided for information
purposes only. You are permitted to print and download this article only for your personal
reference, and only provided that: no documents or related graphics are modified or copied
in any way; no graphics from this article or the referring websites www.numericinsight.com
and www.numericinsight.blogspot.com are used separately from accompanying text; and the
status of Numeric Insight, Inc (and that of any identified contributors) as the author of this
material is always acknowledged.
This legal notice applies to the entire contents of this article and the referring websites
www.numericinsight.com and www.numericinsight.blogspot.com. By using or visiting this
article you accept and agree to the terms of this legal notice. If you do not accept these terms,
please do not use this material.
You understand and agree that Numeric Insight, Inc and any of its principals or affiliates
shall in no event be liable for any direct, indirect, incidental, consequential, or exemplary
damages (including, but not limited to, equipment malfunction, loss of use, data, or profits;
or business interruption) however caused and on any theory of liability, whether in contract,
strict liability, or tort (including negligence or otherwise) arising in any way (including due to
redistribution of software, algorithm, solution or information without including this notice)
out of the use of this software, algorithm, solution or information. You understand and agree
that your use of this software, algorithm, solution or information is entirely at your own risk.
Redistributions of and references to this software, algorithm, solution or information must
retain this notice.
The material in this article and the referring websites www.numericinsight.com and
www.numericinsight.blogspot.com is provided as is, without any conditions, warranties or
other terms of any kind. Numeric Insight, Inc and its respective partners, officers, members,
employees and advisers shall not be liable for any direct or indirect or consequential loss or
damage, nor for any loss of revenue, profit, business, data, goodwill or anticipated savings
suffered by any person as a result of reliance on a statement or omission on, or the use of, this
material.
Numeric Insight, Inc will not be liable for any loss or damage caused by a distributed
denial-of-service attack, viruses or other technologically harmful material that may infect
your computer equipment, computer programs, data or other proprietary material due to
your use of the associated websites or to your downloading of any material posted on it, or
on any website linked to it.
Numeric Insight, Inc may revise the content of this article, including this legal notice, at
any time. This entire notice must be included with any subsequent copies of this material.

15