Artificial Neural Network

Artificial Neural Network

• The term "Artificial Neural Network" is derived from Biological neural networks that
develop the structure of a human brain.

• Similar to the human brain that has neurons interconnected to one another, artificial neural
networks also have neurons that are interconnected to one another in various layers of the
networks.

• These neurons are known as nodes.

Typical diagram of Biological Neural Network. Typical Artificial Neural Network
• An Artificial Neural Network in the field of Artificial intelligence where it attempts to
mimic the network of neurons makes up a human brain so that computers will have an
option to understand things and make decisions in a human-like manner.

• The artificial neural network is designed by programming computers to behave simply like
interconnected brain cells.

• There are around 1000 billion neurons in the human brain. Each neuron has an association
point somewhere in the range of 1,000 and 100,000.

• In the human brain, data is stored in such a manner as to be distributed, and we can extract
more than one piece of this data when necessary from our memory parallelly. We can say
that the human brain is made up of incredibly amazing parallel processors.
 The architecture of an artificial neural network:
• To understand the concept of the architecture of an artificial neural network, we have to
understand what a neural network consists of.

• In order to define a neural network that consists of a large number of artificial neurons,
which are termed units arranged in a sequence of layers.

• Artificial Neural Network primarily consists of three layers:

Input Layer:
As the name suggests, it accepts inputs in several different formats provided by the programmer.

Hidden Layer:
The hidden layer presents in-between input and output layers. It performs all the calculations to find
hidden features and patterns.

Output Layer:
The input goes through a series of transformations using the hidden layer, which finally results in output
that is conveyed using this layer.

The artificial neural network takes input and computes the weighted sum of the inputs and includes a
bias. This computation is represented in the form of a transfer function.

It determines weighted total is passed as an input to an activation function to produce the output.
Activation functions choose whether a node should fire or not. Only those who are fired make it to the
output layer. There are distinctive activation functions available that can be applied upon the sort of task
we are performing.
 How do artificial neural networks work?

• Artificial Neural Network can be best represented as a weighted directed graph, where the
artificial neurons form the nodes.

• The association between the neurons outputs and neuron inputs can be viewed as the
directed edges with weights.

• The Artificial Neural Network receives the input signal from the external source in the
form of a pattern and image in the form of a vector.

• These inputs are then mathematically assigned by the notations x(n) for every n number of
inputs.
• Afterward, each of the input is multiplied by its corresponding weights ( these weights are the details
utilized by the artificial neural networks to solve a specific problem ).

• In general terms, these weights normally represent the strength of the interconnection between neurons
inside the artificial neural network. All the weighted inputs are summarized inside the computing unit.

• If the weighted sum is equal to zero, then bias is added to make the output non-zero or something else
to scale up to the system's response.

• Bias has the same input, and weight equals to 1. Here the total of weighted inputs can be in the range of
0 to positive infinity.

• Here, to keep the response in the limits of the desired value, a certain maximum value is benchmarked,
and the total of weighted inputs is passed through the activation function.

• The activation function refers to the set of transfer functions used to achieve the desired output.

• There is a different kind of the activation function, but primarily either linear or non-linear sets of
functions. Some of the commonly used sets of activation functions are the Binary, linear, Tangent
hyperbolic, sigmoidal, etc. activation functions.
 Activation Function

• Activation functions refer to the functions used in neural networks to compute the weighted
sum of input and biases, which is used to choose the neuron that can be fire or not.

• It controls the presented information through some gradient processing, normally gradient
descent. It produces an output for the neural network that includes the parameters in the
data.

• Activation function can either be linear or non-linear, relying on the function it shows. It is
used to control the output of outer neural networks across various areas, such as speech
recognition, segmentation, fingerprint detection, cancer detection system, etc.

• In the artificial neural network, we can use activation functions over the input to get the
precise output. These are some activation functions that are used in ANN.
Linear Activation Function:
• The equation of the linear activation function is the same as the equation of a straight line
i.e.
Y= mx+ c
• If we have many layers and all the layers are linear in nature, then the final activation
function of the last layer is the same as the linear function of the first layer. The range of a
linear function is –infinitive to + infinitive.

• Linear activation function can be used at only one place that is the output layer.
Binary Step Function
Binary step function depends on a threshold value that decides whether a neuron should be
activated or not.

The input fed to the activation function is compared to a certain threshold; if the input is
greater than it, then the neuron is activated, else it is deactivated, meaning that its output is not
passed on to the next hidden layer.

Mathematically it can be
represented as:
Sigmoid function:
• Sigmoid function refers to a function that is projected as S - shaped graph.
A = 1/(1+e-x)

• This function is non-linear. The value of X is directly proportional to the values of Y. It

means a slight change in the value of x would also bring about changes in the value of
y.
Tanh Function:
• The activation function, which is more efficient than the sigmoid function is Tanh
function.

• Tanh function is also known as Tangent Hyperbolic Function.

• It is a mathematical updated version of the sigmoid function.

• Sigmoid and Tanh function are similar to each other and can be derived from each
other.
F(x)= tanh(x) = 2/(1+e-2X) - 1
OR
Tanh (x) = 2 * sigmoid(2x) – 1

• This function is non-linear, and the value range lies between -1 to +1

ReLU (Rectified Linear Unit) Activation Function

• The ReLU is the most used activation function in the world right now. Since, it is used
in almost all the convolutional neural networks or deep learning.

• As you can see, the ReLU is half rectified (from bottom). f(z) is zero when z is less
than zero and f(z) is equal to z when z is above or equal to zero.

• Range: [ 0 to infinity)
Summarization (including other functions)
 Types of Artificial Neural Network

• Feedforward networks output units

• These compute a series of
transformations hidden units
• Typically, the first layer is the
input and the last layer is the
output. input units

• Recurrent networks
• These have directed cycles in
their connection graph. They
can have complicated
dynamics.
• More biologically realistic.
Different Network Topologies
• Single layer feed-forward networks
• Input layer projecting into the output layer

Single layer
network

Input Output
layer layer
Different Network Topologies
• Multi-layer feed-forward networks
• One or more hidden layers. Input projects only from previous layers
onto a layer.

2-layer or
1-hidden layer
fully connected
network
Input Hidden Output
layer layer layer
Different Network Topologies

• Multi-layer feed-forward networks

Input Hidden Output

layer layers layer
 Recurrent Neural Network
• A recurrent neural network (RNN) is a type of artificial neural network which uses
sequential data or time series data.

• Like feedforward and convolutional neural networks (CNNs), recurrent neural

networks utilize training data to learn.

• They are distinguished by their “memory” as they take information from prior
inputs to influence the current input and output.

• While traditional deep neural networks assume that inputs and outputs are
independent of each other, the output of recurrent neural networks depend on the
prior elements within the sequence.
Different Network Topologies
• Recurrent networks
• A network with feedback, where some of its inputs are connected to
some of its outputs (discrete time).

Recurrent
network

Input Output
layer layer
Recurrent Neural Network vs. Feedforward Neural Network
Types of recurrent neural networks