• An Artificial Neural Network (ANN) is a

mathematical model that tries to simulate the

structure and functionalities of biological
neural networks.
• Basic building block of every artificial neural
network is artificial neuron, that is, a simple
mathematical model (function).
• Such a model has three simple sets of rules:
multiplication, summation and activation.
• Such a model has three simple sets of rules:
multiplication, summation and activation.
• In the middle section of artificial neuron is
sum function that sums all weighted inputs
and bias.
• At the exit of artificial neuron the sum of
previously weighted inputs and bias is passing
through activation function that is also called
transfer function.
 Structure and function of ANN
• An artificial neuron is a mathematical function conceived as a model of
biological neurons, a neural network.
• Artificial neurons are elementary units in an artificial neural network.
• The artificial neuron receives one or more inputs and sums them to produce
an output.
• Usually each input is separately weighted, and the sum is passed through a
non-linear function known as an activation function or transfer function.
• The transfer functions usually have a sigmoid shape, but they may also take
the form of other non-linear functions, piecewise linear functions, or step
functions.
• They are also often monotonically increasing, continuous, differentiable and
bounded.
• The threshold function has inspired building logic gates referred to as
threshold logic applicable to building logic circuits resembling brain processing.
 Biological Vs Artificial neuron
• Size: Our brain contains about 86 billion neurons and more than a 100 synapses
(connections). The number of “neurons” in artificial networks is much less than
that.
• Signal transport and processing: The human brain works asynchronously, ANNs
work synchronously.
• Processing speed: Single biological neurons are slow, while standard neurons in
ANNs are fast.
• Topology: Biological neural networks have complicated topologies, while ANNs
are often in a tree structure.
• Speed: certain biological neurons can fire around 200 times a second on average.
artificial neurons do not experience “fatigue”. They are functions that can be
calculated as many times and as fast as the computer architecture would allow.
• Fault-Tolerance: Biological neuron networks due to their topology are also fault-
tolerant. Artificial neural networks are not modeled for fault tolerance or self
regeneration though recovery is possible by saving the current state (weight
values) of the model and continuing the training from that save state.
• Power consumption:
– Biological neural networks are highly energy-efficient, with the brain
consuming relatively low amounts of energy compared to the computational
power it exhibits.
– Artificial neural networks implemented on electronic hardware may consume
more energy, especially in large-scale applications. Efforts are ongoing to
develop more energy-efficient hardware architectures for ANNs.
• Learning: Biological networks usually don't stop / start learning. ANNs
have different fitting (train) and prediction (evaluate) phases.
• Field of application: ANNs are specialized. They can perform one task.
They might be perfect at playing chess, but they fail at playing go (or
vice versa). Biological neural networks can learn completely new tasks.
• Training algorithm: ANNs use Gradient Descent for learning. Human
brains use something different
 Basic building blocks of ANN
• Processing of ANN depends upon the
following three building blocks:
1. Network Topology
2. Adjustments of Weights or Learning
3. Activation Functions
1. Network Topology: A network topology is the arrangement of a
network along with its nodes and connecting lines. According
to the topology, ANN can be classified as the following kinds:
A. Feed forward Network:
 It is a non-recurrent network having processing units/nodes in
layers and all the nodes in a layer are connected with the nodes of
the previous layers.
 The connection has different weights upon them.
 There is no feedback loop means the signal can only flow in one
direction, from input to output.
 It may be divided into the following two types: Single layer feed
forward network and Multilayer feed forward network
• Single layer feed forward network: The
concept is of feed forward ANN having only
one weighted layer. In other words, we can say
the input layer is fully connected to the output
layer.
• Multilayer feed forward network: The concept
is of feed forward ANN having more than one
weighted layer. As this network has one or
more layers between the input and the output
layer, it is called hidden layers.
B. Feedback Network: As the name suggests, a
feedback network has feedback paths, which
means the signal can flow in both directions using
loops. This makes it a non-linear dynamic system,
which changes continuously until it reaches a state
of equilibrium. It may be divided into the following
types:
• Recurrent networks: They are feedback networks
with closed loops. Following are the two types of
recurrent networks.
• Fully recurrent network: It is the simplest
neural network architecture because all nodes
are connected to all other nodes and each
node works as both input and output.
• Jordan network − It is a closed loop network in
which the output will go to the input again as
feedback as shown in the following diagram.
2. Adjustments of Weights or Learning: Learning, in
artificial neural network, is the method of modifying
the weights of connections between the neurons of a
specified network. Learning in ANN can be classified
into three categories namely supervised learning,
unsupervised learning, and reinforcement learning.
• Supervised Learning: As the name suggests, this type of learning is done under the
supervision of a teacher. This learning process is dependent.
• During the training of ANN under supervised learning, the input vector is
presented to the network, which will give an output vector.
• This output vector is compared with the desired output vector. An error signal is
generated, if there is a difference between the actual output and the desired
output vector.
• On the basis of this error signal, the weights are adjusted until the actual output is
matched with the desired output.
• Unsupervised Learning: As the name suggests, this type of learning is
done without the supervision of a teacher. This learning process is
independent.
• During the training of ANN under unsupervised learning, the input vectors
of similar type are combined to form clusters.
• When a new input pattern is applied, then the neural network gives an
output response indicating the class to which the input pattern belongs.
• There is no feedback from the environment as to what should be the
desired output and if it is correct or incorrect.
• Hence, in this type of learning, the network itself must discover the
patterns and features from the input data, and the relation for the input
data over the output.
• Reinforcement Learning: As the name suggests, this type of learning is
used to reinforce or strengthen the network over some critic information.
• This learning process is similar to supervised learning, however we might
have very less information.
• During the training of network under reinforcement learning, the network
receives some feedback from the environment. This makes it somewhat
similar to supervised learning.
• However, the feedback obtained here is evaluative not instructive, which
means there is no teacher as in supervised learning.
• After receiving the feedback, the network performs adjustments of the
weights to get better critic information in future.
3. Activation Functions: An activation function is a mathematical
equation that determines the output of each element (perceptron or
neuron) in the neural network.
• It takes in the input from each neuron and transforms it into an
output, usually between one and zero or between -1 and one.
• It may be defined as the extra force or effort applied over the input
to obtain an exact output.
• In ANN, we can also apply activation functions over the input to get
the exact output. Followings are some activation functions of
interest
i) Linear Activation Function: It is also called the identity function as
it performs no input editing. It can be defined as: F(x) = x
ii) Sigmoid Activation Function: It is of two types
Binary sigmoidal function: This activation function performs input editing between 0
and 1. It is positive in nature.
• It is always bounded, which means its output cannot be less than 0 and more than
1.
• It is also strictly increasing in nature, which means more the input higher would
be the output.
• It can be defined as
F(x)=sigm(x)=11+exp(−x)F(x)=sigm(x)=11+exp(−x)
Bipolar sigmoidal function: This activation function performs input editing between -
1 and 1.
• It can be positive or negative in nature. It is always bounded, which means its
output cannot be less than -1 and more than 1.
• It is also strictly increasing in nature like sigmoid function.
• It can be defined as
F(x)=sigm(x)=21+exp(−x)−1=1−exp(x)1+exp(x)
WHAT IS A NEURAL NETWORK ACTIVATION FUNCTION?

• In a neural network, inputs, which are typically real values, are fed into the
neurons in the network.
• Each neuron has a weight, and the inputs are multiplied by the weight and fed into
the activation function.
• Each neuron’s output is the input of the neurons in the next layer of the network,
and so the inputs cascade through multiple activation functions until eventually,
the output layer generates a prediction.
• Neural networks rely on nonlinear activation functions—the derivative of the
activation function helps the network learn through the back propagation process.
SOME COMMON ACTIVATION FUNCTIONS
1. The sigmoid function has a smooth gradient and outputs values between zero and
one. For very high or low values of the input parameters, the network can be very
slow to reach a prediction, called the vanishing gradient problem.
2. The TanH function is zero-centered making it easier to model inputs that are strongly
negative strongly positive or neutral.
3. The ReLu function is highly computationally efficient but is not able to process inputs
that approach zero or negative.
4. The Leaky ReLu function has a small positive slope in its negative area, enabling it to
process zero or negative values.
5. The Parametric ReLu function allows the negative slope to be learned, performing
backpropagation to learn the most effective slope for zero and negative input values.
6. Softmax is a special activation function use for output neurons. It normalizes outputs
for each class between 0 and 1, and returns the probability that the input belongs to
a specific class.
7. Swish is a new activation function discovered by Google researchers. It performs
better than ReLu with a similar level of computational efficiency.
 APPLICATIONS OF ANN
1. Data Mining: Discovery of meaningful patterns (knowledge) from large volumes of data.
2. Expert Systems: A computer program for decision making that simulates thought process of a
human expert.
3. Fuzzy Logic: Theory of approximate reasoning.
4. Artificial Life: Evolutionary Computation, Swarm Intelligence.
5. Artificial Immune System: A computer program based on the biological immune system.
6. Medical: At the moment, the research is mostly on modelling parts of the human body and
recognizing diseases from various scans (e.g. cardiograms, CAT scans, ultrasonic scans,
etc.).Neural networks are ideal in recognizing diseases using scans since there is no need to
provide a specific algorithm on how to identify the disease. Neural networks learn by example so
the details of how to recognize the disease are not needed. What is needed is a set of examples
that are representative of all the variations of the disease. The quantity of examples is not as
important as the 'quantity'. The examples need to be selected very carefully if the system is to
perform reliably and efficiently
7. Computer Science: Researchers in quest of artificial intelligence have created spin offs like
dynamic programming, object oriented programming, symbolic programming, intelligent storage
management systems and many more such tools. The primary goal of creating an artificial
intelligence still remains a distant dream but people are getting an idea of the ultimate path,
which could lead to it.
8. Aviation: Airlines use expert systems in planes to monitor atmospheric conditions and system status. The plane can be put on
autopilot once a course is set for the destination.
9. Weather Forecast: Neural networks are used for predicting weather conditions. Previous data is fed to a neural network, which
learns the pattern and uses that knowledge to predict weather patterns.
10. Neural Networks in business: Business is a diverted field with several general areas of specialization such as accounting or
financial analysis. Almost any neural network application would fit into one business area or financial analysis.
11. There is some potential for using neural networks for business purposes, including resource allocation and scheduling.
12. There is also a strong potential for using neural networks for database mining, which is, searching for patterns implicit within
the explicitly stored information in databases. Most of the funded work in this area is classified as proprietary. Thus, it is not
possible to report on the full extent of the work going on. Most work is applying neural networks, such as the Hopfield-Tank
network for optimization and scheduling.
13. Marketing: There is a marketing application which has been integrated with a neural network system. The Airline Marketing
Tactician (a trademark abbreviated as AMT) is a computer system made of various intelligent technologies including expert
systems. A feed forward neural network is integrated with the AMT and was trained using back-propagation to assist the
marketing control of airline seat allocations. The adaptive neural approach was amenable to rule expression. Additionally,
the application's environment changed rapidly and constantly, which required a continuously adaptive solution.
14. Credit Evaluation: The HNC company, founded by Robert Hecht-Nielsen, has developed several neural network applications.
One of them is the Credit Scoring system which increases the profitability of the existing model up to 27%. The HNC neural
systems were also applied to mortgage screening. A neural network automated mortgage insurance under writing system
was developed by the Nestor Company. This system was trained with 5048 applications of which 2597 were certified. The
data related to property and borrower qualifications. In a conservative mode the system agreed on the under writers on
97% of the cases. In the liberal model the system agreed 84% of the cases. This is system run on an Apollo DN3000 and used
250K memory while processing a case file in approximately 1 sec.
 Advantages of ANN
1. Adaptive learning: An ability to learn how to do tasks based on the data given for training or initial
experience.
2. Self-Organisation: An ANN can create its own organisation or representation of the information it
receives during learning time.
3. Real Time Operation: ANN computations may be carried out in parallel, and special hardware
devices are being designed and manufactured which take advantage of this capability.
4. Pattern recognition: is a powerful technique for harnessing the information in the data and
generalizing about it. Neural nets learn to recognize the patterns which exist in the data set.
5. The system is developed through learning rather than programming.. Neural nets teach themselves
the patterns in the data freeing the analyst for more interesting work.
6. Neural networks are flexible in a changing environment. Although neural networks may take some
time to learn a sudden drastic change they are excellent at adapting to constantly changing
information.
7. Neural networks can build informative models whenever conventional approaches fail. Because
neural networks can handle very complex interactions they can easily model data which is too
difficult to model with traditional approaches such as inferential statistics or programming logic.
8. Performance of neural networks is at least as good as classical statistical modelling, and better on
most problems. The neural networks build models that are more reflective of the structure of the
data in significantly less time.
 LIMITATIONS OF ANN
In this technological era everything has Merits and some Demerits in others
words there is a Limitation with every system which makes this ANN
technology weak in some points. The various Limitations of ANN are:-
1. ANN is not a daily life general purpose problem solver.
2. There is no structured methodology available in ANN.
3. There is no single standardized paradigm for ANN development.
4. The Output Quality of an ANN may be unpredictable.
5. Many ANN Systems does not describe how they solve problems.
6. Black box Nature
7. Greater computational burden.
8. Proneness to over fitting.
9. Empirical nature of model development.