ARTIFICIAL NEURAL NETWORK

A neural network consists of inter connected processing elements called neurons that work together to
produce an output function. The output of a neural network relies on the cooperation of the individual
neurons within the network to operate. Well designed neural networks are trainable systems that can often
“learn” to solve complex problems from a set of exemplars and generalize the “acquired knowledge” to
solve unforeseen problems, i.e. they are self-adaptive systems. A neural network is used to refer to a
network of biological neurons. A neural network consists of a set of highly interconnected entities called
nodes or units. Each unit accepts a weighted set of inputs and responds with an output.

Mathematically let I = (I1, I2, … … In) represent the set of inputs presented to the unit U. Each input has an
associated weight that represents the strength of that particular connection. Let W=
(W1, W2, … … Wn) represent the weight vector corresponding to the input vector X. By applying to V,
these weighted inputs produce a net sum at U given by

S = SUM (Wi × Ii)

A neural network is first and foremost a graph, with patterns represented in terms of numerical values
attached to the nodes of the graph and transformations between patterns achieved via simple message-
passing algorithms. The graph contains a number of units and weighted unidirectional connections
between them. The output of one unit typically becomes an input for another. There may also be units
with external inputs and outputs. The nodes in the graph are generally distinguished as being input nodes
or output nodes and the graph as a whole can be viewed as a representation of a multivariate functions
linking inputs to outputs. Numerical values (weights) are attached to the links of the graphs,
parameterizing the input/ output function and allowing it to be adjusted via a learning algorithm. A
broader view of a neural network architecture involves treating the network as a statistical processor
characterized by making particular probabilistic assumptions about data. Figure illustrates one example
of a possible neural network structure.
Figure An example of a Neural network structure

Patterns appearing on the input nodes or the output nodes of a network are viewed as samples from
probability densities and a network is viewed as a probabilistic model that assigns probabilities to
patterns. Biologically, we can also define a neuron. The human body is made up of a vast array of living
cells. Certain cells are interconnected in a way that allows them to communicate pain or to actuate fibres
or tissues. Some cells control the opening and closing of minuscule valves in the veins and arteries. These
specialized communication cells are called neurons. Neurons are equipped with long tentacle like
structures that stretch out from the cell body, permitting them to communicate with other neurons. The
tentacles that take in signals from other cells and the environment itself are called dendrites, while the
tentacles that carry signals from the neuron to other cells are called axons.

Dendrites

Nucleus

Cell Body Mitochondria

Axons

Figure A Neuron
FEATURES OF ARTIFICIAL NETWORK (ANN)
Artificial neural networks may by physical devices or simulated on conventional computers. From a
practical point of view, an ANN is just a parallel computational system consisting of many simple
processing elements connected together in a specific way in order to perform a particular task. There are
some important features of artificial networks as follows.

(1) Artificial neural networks are extremely powerful computational devices (Universal computers).
(2) ANNs are modeled on the basis of current brain theories, in which information is represented by
weights.
(3) ANNs have massive parallelism which makes them very efficient.
(4) They can learn and generalize from training data so there is no need for enormous feats of
programming.
(5) Storage is fault tolerant i.e. some portions of the neural net can be removed and there will be only a
small degradation in the quality of stored data.
(6) They are particularly fault tolerant which is equivalent to the “graceful degradation” found in
biological systems.
(7) Data are naturally stored in the form of associative memory which contrasts with conventional
memory, in which data are recalled by specifying address of that data.
(8) They are very noise tolerant, so they can cope with situations where normal symbolic systems would
have difficulty.
(9) In practice, they can do anything a symbolic/ logic system can do and more.
(10) Neural networks can extrapolate and intrapolate from their stored information. The neural networks
can also be trained. Special training teaches the net to look for significant features or relationships of
data.

TYPES OF NEURAL NETWORKS

There are a number of models of neural networks have been developed, each defined at a different level
of abstraction and trying to model different aspects of neural systems. They range from models of the
short term behaviour of individual neurons, through models of how the dynamics of neural circuitry arise
from interactions between individual neurons, to models of how behaviour can arise from abstract neural
modules that represent complete sub systems. Some neural networks have been illustrated below.

Single Layer Network

A single layer neural network consists of a set of units organized in a layer. Each unit Un receives a
weighted input Ijwith weight Wjn. Figure shows a single layer neural network with j inputs and n outputs.
Figure A single Layer neural Network

Let I = (i1, i2 … … . ij) be the input vector and let the activation function f be simply, so that the activation
value is just the net sum to a unit. The jxn weight matrix is calculated as follows.

W11 W12 … … … W1n

W21 W22
W = [ q q
W2nq
]
Wj1 Wj2................Wjn

Thus the output Ox at unit Ux is

I1
I
Ok = (W1K, W2K … … . . (2
Wji) )q
Ij

Multilayer Network
A multilayer network has two or more layers of units, with the output from one layer serving as input to
the next. Generally in a multilayer network there are 3 layers present like, input layer, output layer and
hidden layer. The layer with no external output connections are referred to as hidden layers. A multilayer
neural network structure is given in figure.
Figure A multilayer neural network

Any multilayer system with fixed weights that has a linear activation function is equivalent to a single
layer linear system, for example, the case of a two layer system. The input vector to the first layer is Irthe
output O = W1 × I and the second layer produces output O2 = W2 × O.HenceO2 = W2 × (W1 × I)

= (W2 × W1) × I

So a linear system with any number n of layers is equivalent to a single layer linear system whose weight
matrix is the product of the n intermediate weight matrices. A multilayer system that is not linear can
provide more computational capability than a single layer system. Generally multilayer networks have
proven to be very powerful than single layer neural network. Any type of Boolean function can be
implemented by such a network. At the output layer of a multilayer neural network the output vector is
compared to the expected output. If the difference is zero, no changes are made to the weights of
connections. If the difference is not zero, the error is calculated and is propagated back through the
network.

Feed Forward neural network

The neural networks consist of multiple layers of computational units, usually interconnected in a feed
forward way. The feed forward neural networks are the first, simplest type of artificial neural networks
devised. In this network, the information moves in only one direction, forward from the input nodes,
through the hidden nodes and to the output nodes. There are no cycles or loops in the network. In other
way we can say the feed forward neural network is one that does not have any connections from output to
input. All inputs with variable weights are connected with every other node. A single layer feed forward
network has one layer of nodes, whereas a multilayer feed forward network has multiple layers of nodes.
The structure of a feed forward multilayer network is given in figure.
Figure Multilayer Feed Forward Neural Network

Data are introduced into the system through an input layer. This is followed by processing in one or more
intermediate (hidden layers). Output data emerge from the network’s final layer. The transfer functions
contained in the individual neurons can be almost anything. The input layer is also called as Zeroth layer, of
the network serves to redistribute input values and does no processing. The output of this layer is described
mathematically as follows.

OOM= im Where m = 1,2 … . NO

(NO representes the no. of neurons in the input or zeroth layer).

The input to each neuron in the first hidden layer in the network is a summation all weighted connections
between the input or Zeroth layer and the neuron in the first hidden layer. We will write the weighted sum
as net sum or net input. We can write the net input to a neuron from the first layer as the product of that
input vector im and weight factor wm plus a bias term . The total weighted input to the neuron is a
summation of these individual input signals described as follows.

net sum = Σ WmXm + θ

m=1

Where N represents the number of neurons in the input layer.

The net sum to the neuron is transformed by the neuron’s activation or transfer function, f to produce a
new output value for the neuron. With back propagation, this transfer function is most commonly either a
sigmoid or a linear function. In addition to the net sum, a bias term  is generally added to offset the
input. The bias is designed as a weight coming from a unitary valued input and denoted as W 0. So, the final
output of the neuron is given by the following equation.
 Output = f (net sum)

= f ( Σ wmim + θ)
m=1

N0

= f ( Σ wm im
O
+ wO )
m=1

But one question may arise in reader’s mind. Why we are using the hidden layer between the input and
output layer? The answer to this question is very silly. Each layer in a multilayer neural network has its
own specific function. The input layer accepts input signals from the outside world and redistributes these
signals to all neurons in the hidden layer. Actually, the input layer rarely includes computing neurons and
thus does not process input patterns. The output layer accepts output signals, or in other words a stimulus
patterns, from the hidden layer and established the output patterns of the entire network. Neurons in the
hidden layer detect the features, the weights of the neurons represent the features hidden in the input
patterns. These features are then used by the output layer in determining the output patterns. With one
hidden layer we can represent any continuous function of the input signals and with two hidden layers
even discontinuous functions can be represented. A hidden layer hides its desired output. Neurons in the
hidden layer cannot be observed through the input/ output behaviour of the network. The desired output
of the hidden layer is determined by the layer itself. Generally, we can say there is no obvious way to
know what the desired output of the hidden layer should be.

Back Propagation neural network

Multilayer neural networks use a most common technique from a variety of learning technique, called the
back propagation algorithm. In back propagation neural network, the output values are compared with the
correct answer to compute the value of some predefined error function. By various techniques the error is
then fed back through the network. Using this information, the algorithms adjust the weights of each
connection in order to reduce the value of the error function by some small amount. After repeating this
process for a sufficiently large number of training cycles the network will usually converge to some state
where the error of the calculation is small.

The goal of back propagation, as with most training algorithms, is to iteratively adjust the weights in the
network to produce the desired output by minimizing the output error. The algorithm’s goal is to solve
credit assignment problem. Back propagation is a gradient-descent approach in that it uses the minimization
of first-order derivatives to find an optimal solution. The standard back propagation algorithm is given
below.

Step1:
Build a network with the choosen number of input, hidden and output units.

Step2:

Initialize all the weights to low random values.

Step3:

Randomly, choose a single training pair.

Step4:

Copy the input pattern to the input layer.

Step5:

Cycle the network so that the activation from the inputs generates the activations in the hidden and output
layers.

Step6:

Calculate the error derivative between the output activation and the final output.

Step7:

Apply the method of back propagation to the summed products of the weights and errors in the output
layer in order to calculate the error in the hidden units.

Step8:

Update the weights attached the each unit according to the error in that unit, the output from the unit
below it and the learning parameters, until the error is sufficiently low.

To derive the back propagation algorithm, let us consider the three layer network shown in figure .
Figure Three layer back-propagation neural network

To propagate error signals, we start at the output layer and work backward to the hidden layer. The error
signal at the output of neuron k at iteration x is defined as

Where is the desired output of neuron k at iteration x.

Generally, computational learning theory is concerned with training classifiers on a limited amount of data.
In the context of neural networks a simple heuristic, called early stopping often ensures that the network
will generalize well to examples not in the training set. There are some problems with the back propagation
algorithm like speed of convergence and the possibility of ending up in a local minimum of the error
function. Today there are a variety of practical solutions that make back propagation in multilayer
perceptrons the solution of choice for many machine learning tasks.