Artificial Intelligence

Dr. Shail Kumar Dinkar

GBPIET,Pauri Garhwal
Table of Contents
1. Introduction of Artificial Intelligence
2. Applications of Artificial Intelligence
3. Techniques of Artificial Intelligence
3.1. Artificial Neural Network (ANN)
3.1.1. Learning
3.1.2. Supervised Learning
3.1.3. Unsupervised Learning
3.2. Properties of Neural Network
3.3. Other properties of Neural Network
3.4. How do our brains work?
3.5. Terminology of Human Brain Processing Systems
3.6. Neural Networks
3.7. Artificial Neural Network Concepts
3.7.1. Block Diagram of ANN
3.7.2. ANN Working Methodology
Table of Contents
3.7.3. Neural Network working methodology
3.8. ANN Activations Functions
3.9. Most Popular Activation Functions
4.0. Generalized Activations Functions
4.1. Single Layer Perceptron
4.2. ANN Multi-Layer Perceptron
4.3. Feed Forward Neural Network
4.4. Backpropagation in ANN
4.5. Loss Functions
4.6. Understanding Backpropagation
4.7. Mathematics behind Backpropagation
Table of Contents
4.8. Various Optimizers in ANN
4.9. Most Popular: Gradient Descent
5.0. Conclusion
1. Artificial Intelligence
Artificial intelligence (AI) is intelligence demonstrated by machines, as opposed
to natural intelligence displayed by animals including humans. Leading AI textbooks define
the field as the study of "intelligent agents": any system that perceives its environment and
takes actions that maximize its chance of achieving its goals. Some popular accounts use the
term "artificial intelligence" to describe machines that mimic "cognitive" functions that
humans associate with the human mind, such as "learning" and "problem solving", however,
this definition is rejected by major AI researchers
2. Applications of Artificial Intelligence
3. Techniques of Artificial Intelligence
✓Machine Learning
✓NLP (Natural Language Processing)
✓Automation and Robotics
✓Machine Vision
✓Neural Networks
➢ Artificial Neural Networks
➢ Convolutional Neural Networks
✓Support Vector Machines
✓Optimization Techniques
✓Heuristics Search
✓Evolutionary Computing
✓Markov Decision Process
3.1. Artificial Neural Network (ANN)
✓Artificial Neural Networks are an architecture or technique to trained
computer machine like as human brain and take decisions to perceive inputs
from their surrounding by through an agent and to do some significant tasks.

How do our brains work?

✓ The Brain is A massively parallel information processing system.
✓ Our brains are a huge network of processing elements. A typical brain
contains a network of 10 billion neurons.
3.1.1. Learning
✓The procedure that consists in estimating the parameters of neurons so that the whole
network can perform a specific task

✓2 types of learning
➢ The supervised learning
➢ The unsupervised learning

✓The Learning process (supervised)

➢ Present the network a number of inputs and their corresponding outputs
➢ See how closely the actual outputs match the desired ones
➢ Modify the parameters to better approximate the desired outputs
3.1.2. Supervised learning
✓The desired response of the neural network in function of particular
inputs is well known.
✓A “Professor” may provide examples and teach the neural network
how to fulfill a certain task
3.1.3. Unsupervised learning
✓Idea : group typical input data in function of resemblance criteria
un-known a priori
✓Data clustering
✓No need of a professor
➢ The network finds itself the correlations between the data
Examples of such networks :
Kohonen feature maps
3.2. Properties of Neural Networks
✓Supervised networks are universal approximators (Non recurrent networks)
✓Theorem : Any limited function can be approximated by a neural network
with a finite number of hidden neurons to an arbitrary precision
✓Type of Approximators
➢Linear approximators : for a given precision, the number of parameters grows
exponentially with the number of variables (polynomials)
➢Non-linear approximators (NN), the number of parameters grows linearly with the
number of variables
3.3. Other properties of Neural Networks
✓Adaptivity
➢Adapt weights to environment and retrained easily
✓Generalization ability
➢May provide against lack of data
✓Fault tolerance
➢Graceful degradation of performances if damaged => The information is
distributed within the entire net.
3.4. How do our brains work?
✓ A processing element

Dendrites: Input
Cell body: Processor
Synaptic: Link
Axon: Output

Fig a: Human Nervous Systems

3.5. Terminology of Human Brain
Processing Systems
✓ A neuron is connected to other neurons through about 10,000 synapses
✓ A neuron receives input from other neurons. Inputs are combined.
✓ Once input exceeds a critical level, the neuron discharges a spike ‐ an electrical
pulse that travels from the body, down the axon, to the next neuron(s).
✓ The axon endings almost touch the dendrites or cell body of the next neuron.
✓ Transmission of an electrical signal from one neuron to the next is effected by
neurotransmitters.
✓Neurotransmitters are chemicals which are released from the first neuron and which
bind to the Second.
✓ This link is called a synapse. The strength of the signal that reaches the next neuron
depends on factors such as the amount of neurotransmitter available.
3.6. Neural Networks : How ANN works?
✓A mathematical model to solve engineering problems
➢Group of highly connected neurons to realize compositions of non linear functions
✓Tasks
➢Classification
➢Discrimination
➢Estimation
✓2 types of networks
➢Feed forward Neural Networks
➢Recurrent Neural Networks
3.6. Neural Networks : How ANN works?

Fig b: Biological Neuron and Artificial Neuron

3.7. Artificial Neural Network Concepts
✓ ANN possess a large number of processing elements called nodes/neurons
which operate in parallel.
✓ Neurons are connected with others by connection link.
✓ Each link is associated with weights which contain information about the
input signal.
✓ Each neuron has an internal state of its own which is a function of the inputs
that neuron receives- Activation level.
3.7.1. Block Diagram of ANN
x1
X1
w1

Y y

x2
w2
y = f ( yin )
X2

yin = x1w1 + x2 w2
3.7.2. ANN Working Methodology
✓ Now, let us have a look at the model of an artificial neuron.

Fig c: Structure of ANN Networks

3.7.3. Neural Network Working Methodology
✓ Single Layer Neural Network

Fig d: Single Layer Neural Networks.

3.7.3. Neural Network Working Methodology

𝒀 = (𝒙𝟏 ∗ 𝒘𝟏 + 𝒙𝟐 ∗ 𝒘𝟐 + 𝒙𝟑 ∗ 𝒘𝟑 + ⋯ . + 𝒙𝒏 ∗ 𝒘𝒏 ) + 𝒃𝒏
Here 𝒃𝒏 𝒓𝒆𝒑𝒓𝒆𝒔𝒆𝒏𝒕𝒔 𝒃𝒊𝒂𝒔.
Activation Function: Generally Sigmoid
𝟏
Sigmoid =
𝟏+𝒆−𝒚
3.8. ANN Activations Functions
3.9. Most Popular Activations Functions

Sigmoid

Relu
𝑹 𝒁 = 𝒎𝒂𝒙(𝟎, 𝒛)
4.0. Generalized Activations Functions
20

18

Linear
16

14

12

y=x
10

0
0 2 4 6 8 10 12 14 16 18 20

1.5

1
Logistic
0.5

1
0

y=
1 + exp( − x)
-0.5

-1

-1.5

-2
-10 -8 -6 -4 -2 0 2 4 6 8 10

1.5

1
Hyperbolic tangent
exp( x) − exp( − x)
0.5

y=
0

exp( x) + exp( − x)
-0.5

-1

-1.5

-2
-10 -8 -6 -4 -2 0 2 4 6 8 10
4.1. Single Layer Perceptron

References: https://machinelearningknowledge.ai/animated-explanation-of-feed-forward-neural-
network-architecture/
4.2. ANN Multilayer Perceptron
4.2. ANN Multilayer Perceptron
4.2. ANN Multilayer Perceptron
4.3. Feed Forward Neural Network

References: https://machinelearningknowledge.ai/animated-explanation-of-feed-forward-neural-
network-architecture/
4.4. Backpropagation in ANN

References: https://machinelearningknowledge.ai/animated-explanation-of-feed-forward-
neural-network-architecture/
4.5. Loss Function
✓The Loss Function is one of the important components of Neural
Networks. Loss is nothing but a prediction error of Neural Net. And the
method to calculate the loss is called Loss Function.

✓In simple words, the Loss is used to calculate the gradients. And gradients are
used to update the weights of the Neural Net. This is how a Neural Net is
trained.
4.6. Understanding Backpropagation
Suppose this is a neural network
Suppose this ANN output is 𝑦 and this is
actual output and suppose let 𝑦ො is predicted
output and 𝑛 is the total number of neurons
or nodes, so here LOSS FUNCTION defined
as:

𝒏
𝟏
𝑳𝑶𝑺𝑺 𝒚, 𝒚ෝ = ෝ)𝟐
෍(𝒚 − 𝒚
𝟐𝒏
𝒊=𝟏

This is also called “Mean

Squared Error” (MSE).
4.6. Understanding Backpropagation
✓ If this Loss function produces values at very extreme level or maximize LOSS
FUNCTION value then “Backpropagation” have to be done.
✓ In this technique Weights will have updated by some mathematical rules in
backward manner.
✓Now again feed forward neural network trained and examine LOSS function
value and this will have done in similar fashion until we will get a very low or
minimal LOSS FUNCTION value.
✓Here LOSS Function also represents the accuracy of our neural network,
because LOW LOSS FUNCTION value represents the accuracy of neural
network.
4.7. Mathematics behind Backpropagation
Suppose the final outcome achieve at
neuron 𝑜1 is 𝑦, and 𝑦ො are predicted
function, so calculate the LOSS
function calculated are as follows:
𝒏
𝟏
𝑳𝑶𝑺𝑺 𝒚, 𝒚ෝ = ෝ)𝟐
෍(𝒚 − 𝒚
𝟐𝒏
𝒊=𝟏

And after calculated LOSS function

backpropagation will have done

Suppose the outcome achieved at

neuron ℎ1 is 𝑜𝑢𝑡1 and on neuron
ℎ2 is 𝑜𝑢𝑡2 . Then the weight
updation formula are as follows.
4.7. Mathematics behind Backpropagation

If we updated weight 𝑤5 then formula of weight

updation are as follows:

𝝏𝑳
𝒘𝒏𝒆𝒘
𝟓 = 𝒘𝒐𝒍𝒅
𝟓 −𝜼
𝝏𝒘𝒐𝒍𝒅
𝟓

Here L represents LOSS FUNCTION

𝜼 represents Learning Rate which is very minimum
e.g. 0.005.
4.7. Mathematics behind Backpropagation
Chain Rule in Backpropagation Suppose if we want to update weight of 𝒘𝟏 then chain rule of
differentiation have to be apply in this context are as follows:

𝝏𝑳
𝒘𝒏𝒆𝒘
𝟏 = 𝒘𝒐𝒍𝒅
𝟏 −𝜼
𝝏𝒘𝒐𝒍𝒅
𝟏
𝝏𝑳
Here we expand are as follows:
𝝏𝒘𝒐𝒍𝒅
𝟏

𝝏𝑳 𝝏𝑳 𝝏𝒘𝟓 𝝏𝒐𝒖𝒕𝟏
= × ×
𝝏𝒘𝒐𝒍𝒅
𝟏
𝝏𝒘𝟓 𝝏𝒐𝒖𝒕𝟏 𝝏𝒘𝒐𝒍𝒅
𝟏

This is the chain rule of differentiation

Here L represents LOSS Function and 𝒐𝒖𝒕𝟏 are the output

of neuron 𝒉𝟏 .
4.7. Mathematics behind Backpropagation
Now all weight updation formula are ✓ Updation of 𝒘𝒏𝒆𝒘
𝟓 :
𝝏𝑳
as follows: 𝒘𝒏𝒆𝒘
𝟓 = 𝒘𝒐𝒍𝒅
𝟓 −𝜼
𝝏𝒘𝒐𝒍𝒅
𝟓

✓ Updation of 𝒘𝒏𝒆𝒘
𝟔 :
𝝏𝑳
𝒘𝒏𝒆𝒘
𝟔 = 𝒘𝒐𝒍𝒅
𝟔 −𝜼
𝝏𝒘𝒐𝒍𝒅
𝟔
✓ Updation of 𝒘𝒏𝒆𝒘
𝟏 :
𝝏𝑳
𝒘𝒏𝒆𝒘
𝟏 = 𝒘𝒐𝒍𝒅
𝟏 −𝜼
𝝏𝒘𝒐𝒍𝒅
𝟏
𝝏𝑳
Chain rule for are as follows:
𝝏𝒘𝒐𝒍𝒅
𝟏

𝝏𝑳 𝝏𝑳 𝝏𝒘𝟓 𝝏𝒐𝒖𝒕𝟏
= × ×
𝝏𝒘𝒐𝒍𝒅
𝟏
𝝏𝒘𝟓 𝝏𝒐𝒖𝒕𝟏 𝝏𝒘𝒐𝒍𝒅
𝟏
4.7. Mathematics behind Backpropagation
✓ Updation of 𝒘𝒏𝒆𝒘
𝟐 :
𝝏𝑳
𝒘𝒏𝒆𝒘
𝟐 = 𝒘𝒐𝒍𝒅
𝟐 −𝜼
𝝏𝒘𝒐𝒍𝒅
𝟐

𝝏𝑳
Chain rule for are as follows:
𝝏𝒘𝒐𝒍𝒅
𝟐
𝝏𝑳 𝝏𝑳 𝝏𝒘𝟓 𝝏𝒐𝒖𝒕𝟏
= × ×
𝝏𝒘𝒐𝒍𝒅
𝟐
𝝏𝒘𝟓 𝝏𝒐𝒖𝒕𝟏 𝝏𝒘𝒐𝒍𝒅
𝟐

✓ Updation of 𝒘𝒏𝒆𝒘
𝟑 :
𝝏𝑳
𝒘𝒏𝒆𝒘
𝟑 = 𝒘𝒐𝒍𝒅
𝟑 −𝜼
𝝏𝒘𝒐𝒍𝒅
𝟑
𝝏𝑳
Chain rule for are as follows:
𝝏𝒘𝒐𝒍𝒅
𝟑
𝝏𝑳 𝝏𝑳 𝝏𝒘𝟔 𝝏𝒐𝒖𝒕𝟐
= × ×
𝝏𝒘𝒐𝒍𝒅
𝟑
𝝏𝒘𝟔 𝝏𝒐𝒖𝒕𝟐 𝝏𝒘𝒐𝒍𝒅
𝟑
4.7. Mathematics behind Backpropagation
Updation of 𝒘𝒏𝒆𝒘
𝟒 :
𝝏𝑳
𝒘𝒏𝒆𝒘
𝟒 = 𝒘𝒐𝒍𝒅
𝟒 −𝜼
𝝏𝒘𝒐𝒍𝒅
𝟒

𝝏𝑳
Chain rule for are as follows:
𝝏𝒘𝒐𝒍𝒅
𝟒

𝝏𝑳 𝝏𝑳 𝝏𝒘𝟔 𝝏𝒐𝒖𝒕𝟐
= × ×
𝝏𝒘𝒐𝒍𝒅
𝟒
𝝏𝒘𝟔 𝝏𝒐𝒖𝒕𝟐 𝝏𝒘𝒐𝒍𝒅
𝟒
4.8. Various Optimizers in ANN
✓Gradient Descent.
✓Stochastic Gradient Descent (SGD)
✓Mini Batch Stochastic Gradient Descent (MB-SGD)
✓SGD with momentum.
✓Nesterov Accelerated Gradient (NAG)
✓Adaptive Gradient (AdaGrad)
✓AdaDelta.
✓RMSprop.
4.9. Most Popular -> Gradient Descent
Suppose we have a scalar function f(w) :  → 

We want to find a local minimum.

Assume our current weight is w

GRADIENT DESCENT RULE: w  w −  f (w)
w

η is called the LEARNING RATE. A small positive number, e.g. η = 0.05

4.9. Most Popular -> Gradient Descent
Given f(w) :  → 
m

  
 f (w ) 
 w1  points in direction of steepest ascent.
f (w ) =   
  
 w f ( w )
 m   f (w )
is the gradient in that direction

GRADIENT DESCENT RULE: w  w - f (w )

Equivalently  ….where wj is the jth weight
wj  wj - f (w )
w j “just like a linear feedback system”
4.9. Most Popular -> Gradient Descent
4.9. Most Popular -> Gradient Descent
4.9. Most Popular -> Gradient Descent
For supervised learning, neural nets are also models with vectors of w
parameters in them. They are now called weights.
As before, we want to compute the weights to minimize sum-of-squared
residuals.
Which turns out, under “Gaussian noise”
assumption to be max. likelihood.
Instead of explicitly solving for max. likelihood weights, we use GRADIENT
DESCENT to SEARCH for them.
5.0. Conclusion
✓Artificial Intelligence and Machine Learning are products of both science and myth. The
idea that machines could think and perform tasks just as humans do is thousands of years
old. The cognitive truths expressed in AI and Machine Learning systems are not new either.
It may be better to view these technologies as the implementation of powerful and long-
established cognitive principles through engineering.

✓We should accept that there is a tendency to approach all important innovations as a
Rorschach test upon which we impose anxieties and hopes about what constitutes a good or
happy world. But the potential of AI and machine intelligence for good does not lie
exclusively, or even primarily, within its technologies. It lies mainly in its users. If we trust
(in the main) how our societies are currently being run then we have no reason not to trust
ourselves to do good with these technologies. And if we can suspend presentism and accept
that ancient stories warning us not to play God with powerful technologies are instructive
then we will likely free ourselves from unnecessary anxiety about their use.
END