Unit 1 Introduction to

Soft Computing
Outline
• What is Computing?
• Soft Computing & Hard Computing
• Introduction to Neural Network: Biological Neuron
• Fundamental concept of Artificial Neural Network & Biological
Neural Network
• Evolution of Neural Networks
• Basic Models of Artificial Neural Networks
• Types of Learning
• Important Terminologies of ANNs
• Idea of computational units & Activation Functions
• McCulloch–Pitts unit and Thresholding logic
•
Computing
• Before diving into soft computing, it's essential to first
understand what computing is and familiarize ourselves with the
core terminologies related to computing.
• The following diagram represents the basic process that occurs in
computing:
• In this diagram, we observe that there is a function, f, which
takes some input, x, and produces an output, y. The input is
referred to as the Antecedent, while the output is called
the Consequent.
• The function f can also be called a formal method, algorithm,
or a mapping function. It represents the logic or steps used to
process the input and generate the output.
• The middle part of this diagram is known as the computing
unit, where the function resides. This is where we feed the input,
and a process occurs to convert that input into the desired
output y.
• In the computing process, the steps that guide how the input is
manipulated are called control actions.
• These control actions ensure that the input gradually approaches
the desired output. The moment the process completes, we
obtain the final result, known as the Consequent.
Basic Characteristics Related to
Computing
• Precise Solution: Computing is used to produce a solution that
is exact and definitive.
• Unambiguous & Accurate: The control actions within the
computing unit must be unambiguous, meaning they must have
only one interpretation. Each control action should also be
accurate to ensure that the process is valid and reliable.
• Mathematical Model: A well-defined algorithm is a requirement
for solving any problem through computing. This algorithm is
essentially a mathematical model that represents the problem
and its solution process.
Type of Computing
• Hard Computing

• Soft Computing
Hard Computing
• Hard computing refers to a traditional computing approach where
the results are always precise and exact.

• This is because hard computing relies on well-defined

mathematical models and algorithms to solve problems.

• There is no room for approximation, as every calculation or action

leads to a deterministic output.
Main Features of Hard Computing
• Exact Results: The results in hard computing are always exact. This
means, whenever we solve a problem using hard computing methods,
the answer will always be the same and correct. There is no room for
uncertainty or approximation.
• Clear Control Actions: In hard computing, the steps or actions we
take to solve a problem are clear and have only one meaning. For
example, when following an algorithm, each step must be clearly
understood and should only have one possible result. There should be
no confusion or multiple meanings for a single step. This is called
being "unambiguous."
• Based on Mathematical Models: All the actions in hard computing
are based on well-defined mathematical rules or formulas. This means
that each step we take while solving a problem has a proper
mathematical explanation behind it. These steps follow a fixed pattern
or method, which helps in solving the problem in a precise way.
Examples of Hard Computing
• Numerical Problems: Problems that require accurate
mathematical calculations, like solving equations or doing
arithmetic, are examples of hard computing.

• Searching and Sorting Algorithms: In computer science,

methods like searching for an item in a list or sorting a list in a
particular order are solved using hard computing techniques
because they always give an exact answer.

• Computational Geometry Problems: These are problems

related to shapes, distances, and areas in geometry. For example,
calculating the shortest distance between two points or finding the
area of a triangle requires exact methods, which is where hard
Soft Computing

“I think the brain is essentially a computer, and

consciousness is like a computer program. It will cease to
run when computer is turned off. Theoretically, it could be
recreated on a Neural Network, but that would require all
one’s memories.”
-Stephen
Hawkin
Theoretical, physicist, cosmologist
and author
• The ideology of soft computing was first brought in light by Sir
Lotfi A. Zadeh at the University of California, Berkley, U.S.A.

• He is proved to be a legend in the development of fuzzy

mathematics, which is known to be the roots of Artificial
Intelligence.

• Soft computing can tackle a collection of imprecise data and

process on it to extract the useful information out of it, without
exploiting and following the so called rules used to perform the
particular task, thus it is thought to be useful in applications like
image recognition, forecasting, optimization, natural language,
making decision out of common sense etc.

• It is a combination of techniques and methods which help to deal

• Soft Computing is an approach to computing that models the
human mind’s ability to make decisions in an uncertain, imprecise,
or complex environment.

• Unlike traditional, or "hard," computing, which relies on exact,

binary logic (0s and 1s), soft computing deals with approximation,
flexibility, and learning from experience to solve complex real-world
problems.

• Soft computing techniques focus on developing systems that can

handle ambiguity, uncertainty, and approximation, making them
well-suited for fields like artificial intelligence (AI), pattern
recognition, and robotics.
• Fuzzy set : for knowledge representation via fuzzy If – Then rules.
• Neural Networks : for learning and adaptation
• Genetic Algorithms : for evolutionary computation
Difference
between
Hard and
Soft
Computing
Requirement of Soft Computing
• Soft computing is essential because many real-world problems
are too complex for traditional computing methods. Systems
must handle:
• Uncertainty: Data may not always be precise, requiring flexible
models.
• Partial truths: Situations where a simple binary "true/false"
answer isn’t sufficient.
• Imprecision: Many problems, such as human language
processing, involve vague or imprecise inputs.
• Complex systems: Soft computing helps in dealing with
complex, non-linear systems, like weather prediction or stock
market analysis.
Importance of Soft Computing
• Handles Uncertainty and Approximation:
• Traditional algorithms often require exact inputs, but real-world problems
don't always have precise data.
• Soft computing methods can handle uncertainty, making them more
suitable for practical applications.
• Example: In medical diagnosis, data such as symptoms might be
uncertain or incomplete, and soft computing helps make decisions
despite that.

• Flexible and Adaptive:

• Soft computing systems are highly adaptive to changes and can learn
from new data. This makes them ideal for dynamic environments where
conditions or inputs change over time.
• Example: In stock market predictions, a neural network can learn from
past trends and adapt to new market conditions.
Importance of Soft Computing
• Tolerance to Noise and Incompleteness:
• Soft computing techniques can work with noisy data (data that is random or
disturbed) and incomplete information.
• They don’t require the exact and clean data that traditional methods do.
• Example: Image recognition systems in self-driving cars can still work even
when the images are slightly unclear or distorted.

• Ease of Implementation:
• Soft computing techniques like fuzzy logic and genetic algorithms are often
easier to implement for complex, real-world problems compared to
traditional methods.
• They don’t require a precise model of the system but instead learn from
examples.
• Example: In control systems, fuzzy logic can be quickly applied to model
human-like decision-making without needing complex mathematical
Importance of Soft Computing
• Incorporates Human-Like Reasoning:
• Soft computing allows for reasoning based on approximate data,
mimicking human decision-making.

• This is especially useful in applications where exact answers aren’t

available.

• Example: In customer service, fuzzy logic can simulate human-like

reasoning in responding to customer queries.
Various Techniques of Soft Computing
• Fuzzy Logic: Mimics human reasoning, allowing for more than just "true" or
"false" outcomes. It’s widely used in control systems, such as thermostats or
automated vehicle systems.

• Neural Networks: Modeled after the human brain, these networks learn
from data and improve over time. They are used in areas like image
recognition, speech processing, and autonomous systems.

• Genetic Algorithms: Inspired by natural selection, these algorithms

"evolve" solutions over generations. They are useful for optimization
problems, such as route planning or complex decision-making.

• Probabilistic Reasoning: Deals with uncertainty using probabilities. It is

useful in situations where outcomes are not deterministic, like predicting
stock market trends or diagnosing diseases.
Applications of Soft Computing
• Soft computing techniques are applied in a variety of fields, including:
• Artificial Intelligence (AI): Many AI systems rely on neural networks and
fuzzy logic to emulate human decision-making.
• Pattern Recognition: Soft computing helps identify patterns in complex
data, such as handwriting recognition or facial recognition.
• Robotics: Soft computing enables robots to navigate environments with
uncertain data, making them more adaptable.
• Data Mining: Soft computing techniques are used to discover patterns
and relationships in large datasets.
• Control Systems: Fuzzy logic controllers are used in industrial systems,
like temperature control in manufacturing plants.
• Medicine: Soft computing helps in diagnosing diseases, predicting patient
outcomes, and optimizing treatment plans.
• Economics and Finance: Used in predicting market trends, risk analysis,
Advantages of Soft Computing
• Robustness: Soft computing techniques are robust and can handle
uncertainty, imprecision, and noise in data, making them ideal for
solving real-world problems.
• Approximate solutions: Soft computing techniques can provide
approximate solutions to complex problems that are difficult or
impossible to solve exactly.
• Non-linear problems: Soft computing techniques such as fuzzy
logic and neural networks can handle non-linear problems effectively.
• Human-like reasoning: Soft computing techniques are designed to
mimic human-like reasoning, which is often more effective in solving
complex problems.
• Real-time applications: Soft computing techniques can provide
real-time solutions to complex problems, making them ideal for use
in real-time applications.
Disadvantages of Soft Computing
• Approximate solutions: Soft computing techniques provide
approximate solutions, which may not always be accurate.
• Computationally intensive: Soft computing techniques can be
computationally intensive, making them unsuitable for use in some
real-time applications.
• Lack of transparency: Soft computing techniques can sometimes
lack transparency, making it difficult to understand how the
solution was arrived at.
• Difficulty in validation: The approximation techniques used in
soft computing can sometimes make it difficult to validate the
results, leading to a lack of confidence in the solution.
• Complexity: Soft computing techniques can be complex and
difficult to understand, making it difficult to implement them
Introduction to Neural Computing
• Neural Computing can be referred as the next major achievement
in the industry of Artificial Intelligence.
• In simple words, neural computers can be defined as systems,
comprised of artificial neural networks that are closely related to
Biological Neural Networks in the technique of solving problems.
• The purpose behind building such systems is to mimic the natural
processing of human brains.
• Dr. Robert Hecht-Nielsen, the inventor of the first neuro-
computers, defines a neural network as:

“A computing system made up of a number of simple, highly

interconnected processing elements, which process information by
their dynamic state response to external inputs”.
Fundamentals of ANN
• Artificial Neural Networks (ANNs) are a key component of modern
artificial intelligence, inspired by the way biological neural networks in
the human brain function.

• ANNs are designed to mimic the brain's ability to process information,

learn from data, and make decisions.

• This section explores the fundamentals of ANNs, starting with an

understanding of biological neural networks, which serve as the
foundation for developing these sophisticated computational models.

• By examining how neurons interact in the human brain, we can better

appreciate how artificial networks are structured and how they
perform various tasks.
Biological Neural Network
• Computation in the brain
• The neural system is not just the neural network; the overall
neural system of the human body comprised of three primary
constituents:
• Receptors
• A neural network
• Effectors.

• The function of receptors is to receive the stimuli either internally

(in the form of signals from internal organs like heart, liver etc.)
or externally (through sensory organs, like eyes, ears, tongue
etc.), this information is then passed onto the neurons in the form
of electrical impulses.
Biological Neural Network
• The neural network then in response to the incoming information
took a decision of the outputs.
• At last, the effectors convert incoming electrical impulses out of the
neural network into responses to the external signals received by the
atmosphere.
• Lets understand this phenomena with a simple practical example,
suppose, a ball of fire is coming with enormous speed towards you;
then the receptors, will send this information to the neural network,
• The neural network will take a decision in response to the incoming
danger, by generating a signal like run, or lean down, or move to
right or left, according to the circumstances.
• In response to the generated signal, the effectors will make the body
parts to respond, like legs will start running or anything like that.
Biological Neural Network

• Figure shows the bidirectional

communication between stages
for feedback
Biological
Neural Network
• First, we should understand that
a neural network is a massive
collection of neurons that are
interconnected with one another.
• The human brain contains
billions of neurons,
approximately 1013, and all these
neurons are interconnected to
form a complex network.
• The following is a diagram of a
single biological neuron:
Biological Neural Network
 Biological
Neural Network
• Dendrites are branching fibers that extend from
the cell body or soma.
• Soma or cell body of a neuron contains the
nucleus and other structures, support chemical
processing and production of neurotransmitters.
• Axon is a singular fiber carries information away
from the soma to the synaptic sites of other
neurons (dendrites ans somas), muscels, or
glands.
• Axon hillock is the site of summation for
incoming information. At any moment, the
collective influence of all neurons that conduct
impulses to a given neuron will determine whether
or not an action potential will be initiated at the
axon hillock and propagated along the axon.
 Biological
Neural Network
• Myelin sheath consists of fat-containing cells that insulate
the axon from electrical activity. This insulation acts to
increase the rate of transmission of signals. A gap exists
between each myelin sheath cell along the axon. Since fat
inhibits the propagation of electricity, the signals jump from
one gap to the next.
• Nodes of Ranvier are the gaps (about 1 μm) between
myelin sheath cells. Since fat serves as a good insulator, the
myelin sheaths speed the rate of transmission of an electrical
impulse along the axon.
• Synapse is the point of connection between two neurons or a
neuron and a muscle or a gland. Electrochemical
communication between neurons take place at these
junctions.
• Terminal buttons of a neuron are the small knobs at the end
of an axon that release chemicals called neurotransmitters.
Biological Neural Network
Information flow in a neural cell
• A biological neuron consists of dendrites, which are responsible for
collecting inputs.

• Inside the neuron, there is a cell body (also known as the soma),
and within the cell body, we find the nucleus.

• The dendrites collect the input signals, and the cell body processes
that input. Once processed, the output is transferred via the axon.

• The axon is the part of the neuron that carries the output signals to
other neurons.
Biological Neural Network
• In the human brain, we have billions of neurons, and the
dendrites of one neuron connect with the axons of other neurons.
• The point where two neurons connect is called a synapse.
Information is stored and transferred across these synapses.
• As we store more information in our brains, the synapses
strengthen, allowing us to retain that information.
• With billions of neurons, we also have billions of synapses, which
play a key role in memory and learning.
• For example, when we read a question in an exam, the
information is sent to our brain through neurons. Another
example is when a mosquito bites us; the signal of pain is sent to
our brain, and we respond to it through neural signals.
Biological Neural Network
Definition of Biological Neural Network
• A biological neural network is a network of neurons that processes
and transmits information in the brain. It is the basis of how humans
think, learn, and make decisions. The neurons communicate with
each other through synapses, which strengthen as we store more
information, allowing us to retain and recall data.

They wanted to replicate this ability in machines. This is how the

concept of artificial neural networks (ANNs) was born.
Researchers aimed to build machines that could think and act like
humans, think rationally, and perform tasks without human
intervention. The structure and function of ANNs are modeled on
biological neural networks, which is why BNNs played a significant
role in the development of ANNs.
Artificial Neural Network
• An Artificial Neural Network (ANN) is designed to develop a
computational system that models the human brain.

• This model allows ANNs to perform a variety of tasks more

efficiently than traditional computer systems.

• For example, while traditional computers might take a long time

to perform a task, an ANN can complete the same task in a much
shorter period. ANNs are used for various applications such as
classification, pattern recognition, and optimization.
• Essentially, ANNs are developed to mimic the functionality of the
human brain and perform tasks faster and more efficiently than
conventional systems.
Artificial Neural Network
• The main objective of an ANN is to process and analyze data in a
manner similar to how the human brain does, allowing it to make
decisions, recognize patterns, and solve complex problems with
speed and accuracy.
• ANNs are designed to process information in a way that is similar
to how the brain processes information.
• An ANN consists of highly interconnected processing elements
known as nodes. In the human brain, these processing elements
are neurons.
• Similarly, in ANNs, nodes act like neurons and are interconnected
with each other through links. Each node receives inputs,
processes them, and sends outputs to other nodes. This network
of nodes and connections allows ANNs to perform complex tasks
Artificial Neural Network
Characteristics of Artificial Neural Networks
• It is a mathematical model consists of computational elements
implemented neurally.
• Large number of highly interconnected processing elements
known as neurons are prominent in ANN
• The interconnections with their weights are associated with
neurons.
• The input signals arrive at the processing elements through
connections and weights.
• ANNs collective behavior is characterized by their ability to learn,
recall and generalize from the given data.
• A single neuron carries no specific information.
Evolution of Artificial Neural Network
• 1943 − It has been assumed that the concept of neural network
started with the work of physiologist, Warren McCulloch, and
mathematician, Walter Pitts, when in 1943 they modeled a simple
neural network using electrical circuits in order to describe how
neurons in the brain might work.
• 1949 − Donald Hebbs book, The Organization of Behavior, put
forth the fact that repeated activation of one neuron by another
increases its strength each time they are used.
• 1956 − An associative memory network was introduced by Taylor.
• 1958 − A learning method for McCulloch and Pitts neuron model
named Perceptron was invented by Rosenblatt.
• 1960 − Bernard Widrow and Marcian Hoff developed models called
"ADALINE" and MADALINE.
Evolution of Artificial Neural Network
• 1961 − Rosenblatt made an unsuccessful attempt but proposed the
backpropagation scheme for multilayer networks.
• 1964 − Taylor constructed a winner-take-all circuit with inhibitions
among output units.
• 1971 − Kohonen developed Associative memories.
• 1976 − Stephen Grossberg and Gail Carpenter developed Adaptive
resonance theory.
• 1982 − The major development was Hopfields Energy approach.
• 1985 − Boltzmann machine was developed by Ackley, Hinton, and
Sejnowski.
• 1986 − Rumelhart, Hinton, and Williams introduced Generalised Delta
Rule
• 1988 − Kosko developed Binary Associative Memory BAM and also
Difference between ANN and
BNN

Biological Neural Artificial Neural

Network Network
Soma Node
Dendrites Input
Weights or
Synapse
Interconnections
Axon Output
Difference between ANN and
BNN
Criteria BNN ANN
Massively parallel, slow but superior Massively parallel, fast but inferior than
Processing
than ANN BNN

102 to104 nodes mainly depends on the

type of application and network
Size 1011 neurons and 1015 interconnections
designer mainly depends on the type of
application and network designer

Very precise, structured and formatted

Learning They can tolerate ambiguity
data is required to tolerate ambiguity

It is capable of robust performance,

Performance degrades with even
Fault tolerance hence has the potential to be fault
partial damage
tolerant

Stores the information in continuous

Storage capacity Stores the information in the synapse
memory locations
Model of Artificial Neural
Network
Model of Artificial Neural Network
• For example, consider an ANN with two input nodes (A and B)
connected to one output node (C).

• Node A is connected to node C, and node B is also connected to

node C.

• Suppose node A receives an input signal called x1 and node B

receives an input signal called x2.

• The output node C will receive a combined signal y. We assign

weights to the connections between the nodes: w1 for the
connection between A and C, and w2 for the connection between
Model of Artificial Neural Network
• The weights are crucial because they determine the strength and
importance of each input signal.

• By adjusting the weights, the ANN can learn which inputs are
more significant for making accurate predictions or decisions. The
net input to node C is calculated as follows:

Net Input = x1 * w1 + x2 * w2
Model of Artificial Neural Network
• This calculation provides the net input to node C, but to determine the
actual output, we need to apply an activation function to the net input.

• The activation function helps to decide whether the neuron should be

activated or not. After applying the activation function, we get the final
output for node C.

• This example illustrates a simple ANN structure. In more complex ANNs,

there can be multiple input nodes (A, B, C, D, E, F), each with its own
input signals (x1, x2, x3, x4, x5, etc.) and weights (w1, w2, etc.).

• The net input to the output node is calculated similarly, but with many
more inputs and weights. The final output is determined after applying
the activation function to the net input.
Model of Artificial Neural Network
• For above neuron architecture , the net input has to be calculated in the way.
yin = x1w1+x2w2

• where x1 and x2 are the activations of the input neurons X1 and X2 .

• The output yin of the output neuron Y can be obtained by applying activations
over the net input .

y =f(yin)

Output = Function ( net input calculated )

• The function to be applied over the net input is called activation function
Basic Models of Artificial Neural Networks
• To understand Artificial Neural Networks (ANN), we must first examine its
foundational components.
• Just like any structure, a neural network is built upon certain key elements that
define how it processes information, learns from data, and makes predictions.
• Architecture of ANN
• Architecture refers to the layout or structure of an ANN. It is how the neurons
(units) are organized in layers and how these layers are connected to each other.
• Layers: An ANN typically consists of three types of layers:
• Input Layer: Receives the raw data (like images, numbers, etc.) and passes it on to the next
layer.
• Hidden Layer(s): These layers process the input and extract patterns. The number of hidden
layers and neurons in each layer can vary.
• Output Layer: Provides the final result (e.g., a classification label or prediction).
Basic Models of Artificial Neural Networks
• The architecture of a neural network defines how data flows through the network,
from input to output, and how decisions are made at each step.
• Connection with Weights: The architecture alone does not make the network
functional.
• Weights determine the strength and impact of each connection on the final output.

Setting of Weights
• Weights are numerical values that control how much influence one neuron has
over another.
• Every connection between neurons has a weight associated with it, and these
weights are learned during the training process.
Basic Models of Artificial Neural Networks
• Initial Weights: At the start, weights are usually assigned randomly. These
weights are adjusted as the network learns from the data.
• Role of Weights: The higher the weight between two neurons, the stronger the
connection, meaning the output from one neuron has a greater influence on the
next neuron’s decision.
• Learning: During training, the network adjusts the weights to minimize errors in
its predictions. This process is called training the network, and it’s essential for the
network’s ability to learn from data.
Basic Models of Artificial Neural Networks
• Connection with Activation Functions: Once the weights determine how much
influence one neuron’s output
• need a mechanism to decide whether the output should be passed forward or
"activated.“

Activation Functions
• An activation function is a mathematical function that determines whether a
neuron should be activated (fired) based on the weighted sum of its inputs.
• It introduces non-linearity to the network, allowing it to solve complex problems.
Basic Models of Artificial Neural Networks
• Importance of activation function: let us assume a person is performing some
task.
• To make the task more efficient and to obtain correct results, some force or
motivation may be given in achieving the correct results.
• In a similar way, the activation function is applied over the net input of the
network to calculate the output of an ANN so that we get a better output or result.
• Type of Activation Function
• 1: Identity Function
• 2: Binary Step Function
• 3: Bipolar Step Function
• 4: Sigmoidal Activation Functions
• Ramp Activation Function
Identity
Function
• Also known as the linear function because the
output is identical to the input.
• It can be defined as:
f(x) = x for all x.
Here, if the net input is x, then the output will
also be x.
• The input layer often uses the identity
activation function because no transformation
is needed.
 Binary Step
Function
• This function can be defined as:
f(x) = 1 if x ≥ θ
f(x) = 0 if x < θ
• where θ is the threshold value. If the net input (x) is greater
than or equal to this threshold value, the output is 1,
otherwise, it is 0.
• It is widely used in single-layer neural networks where the
output needs to be binary (either 0 or 1).
Bipolar Step
Function
• Similar to the binary step function, but it outputs either
+1 or -1 instead of 1 or 0.
• It can be defined as:
f(x) = 1 if x ≥ θ
f(x) = -1 if x < θ
• Used when outputs are expected to be in the range of +1
and -1, often in single-layer networks.
Sigmoidal Activation Functions
• Sigmoidal functions introduce non-linearity to the network and are widely used in
backpropagation-based neural networks. There are two main types:
• a) Binary Sigmoidal Function (Logistic Function)
• b) Bipolar Sigmoidal Function
Sigmoidal Activation Functions
• Binary Sigmoidal Function (Logistic Function)
• It can be defined as:
f(x) = 1 / (1 + e^−λx)
where λ (lambda) is the steepness parameter, and x is the net input.
• The output of the binary sigmoidal function is always between 0 and 1.
• This function has a peculiar property: its derivative is:
f′(x) = λf (x)(1 − f(x))
Sigmoidal Activation Functions
• It can be defined as:
f(x) = (2 / (1 + e^−λx) )− 1
where λ (lambda) is the steepness parameter, and x is the net input.
• The output of the bipolar sigmoidal function ranges from -1 to +1.
• Its derivative is:
f′(x) = λ/2 * (1 + f(x)) (1 − f(x))
 Ramp Activation
Function
• This function is a combination of step and linear functions.
• It can be defined as:
f(x)=1 if x > 1
f(x)=0 if x < 0
f(x)=x if 0 ≤ x≤ 1
• It outputs 0 for inputs less than 0, increases linearly with inputs
between 0 and 1, and outputs 1 for inputs greater than 1.
• Activation functions play a vital role in artificial neural networks
by introducing non-linearity and helping the network learn
complex patterns.
Sigmoid Activation Function Solved
Example
• In this case, we have a neural network with an input layer and an
output layer. The input layer has three neurons, and the output
layer has one neuron.
• The inputs are 0.8, 0.6, and 0.4.
• The bias weight is 0.35.
• The weights associated with the three neurons are 0.1, 0.3,
and −0.2.
Sigmoid Activation Function Solved
Example
• To calculate the net input yn, we find the sum of the product of
the inputs and weights. The equation looks like this:
• yn=B+∑i=1n(xi⋅wi)In this case, n is the number of input neurons,
which is 3.
• Expanding the equation:
• yn=B+(x1⋅w1)+(x2⋅w2)+(x3⋅w3)Substituting the values:
Sigmoid Activation Function Solved
Example
• Bias (B) = 0.35
• x1=0.8,w1=0.1
• x2=0.6,w2=0.3
• x3=0.4,w3=−0.2
• Now solving the equation:
• yn=0.35+(0.8⋅0.1)+(0.6⋅0.3)+(0.4⋅−0.2)
• After calculation, we get yn=0.53 as the net input to the neuron.
• Now we can use this net input to find the outputs:
Sigmoid Activation Function Solved
Example
• Binary Sigmoid Activation Function: Using the net input in
the binary sigmoid equation, we get: output=1/1+e−0.53≈0.628
• Bipolar Sigmoid Activation Function: Using the net input in
the bipolar sigmoid equation, we
get: output=2/1+e−0.53−1≈0.257
Bias
• The bias is a constant value included in the network. Its impact is seen in
calculating the net input.
• The bias is included by adding a component x0 =1 to the input vector X.
• Bias can be positive or negative.
• The positive bias helps in increasing the net input of the network.
• The negative bias helps in decreasing the net input of the network.
Threshold
• Threshold is a set value used in the activation function.
• In ANN, based on the threshold value the activation functions are defined and the
output is calculated.
Learning Rate
• The learning rate is used to control the amount of weight adjustment at each step
of training.
• The learning rate ranges from 0 to 1.
• It determines the rate of learning at each time step.
McCulloch-Pitts Neuron
• he McCulloch-Pitts (MP) neuron, introduced in 1943 by Warren McCulloch and
Walter Pitts, is one of the earliest models of artificial neural networks.
• This pioneering work laid the foundation for the field of neural computing and
artificial intelligence.
• The MP neuron was designed to mimic the functioning of biological neurons in
the human brain, aiming to represent logical operations in a simplified manner.
• Unlike more complex models, the MP neuron operates based on binary inputs and
outputs, effectively simulating the way neurons fire in response to stimuli.
McCulloch-Pitts Neuron
• it serves as a fundamental building block in understanding neural networks and
provides insights into how information processing occurs within more
sophisticated models.
• This model is crucial for grasping essential concepts in neural networks, including
activation functions, thresholds, and the basic principles of learning.

Architecture of McCulloch-Pitts Neuron

• The architecture of the MP neuron consists of two layers:
• Input Layer: Contains the input neurons.
• Output Layer: Contains the output neuron.
McCulloch-Pitts Neuron
McCulloch-Pitts Neuron
McCulloch-Pitts Neuron
• The input layer neurons are connected to the output neuron through directed
edges, which can have either positive or negative weights.
• Positive weights are associated with excitatory nodes, while negative weights are
associated with inhibitory nodes.

Activation Function and Threshold Value

• The firing of the output neuron depends on a threshold value.
• The activation function of this network can be defined as follows:
• Let F(yn) be the activation function, where yn is the net input.
McCulloch-Pitts Neuron
• The function can be expressed as:
F(yn) = 1 if yn ≥ θ
F(yn) = 0 otherwise
• Here, θ is the threshold value. For the neuron to fire, the net input yn must be
greater than or equal to the threshold value.

Determining the Threshold Value

• The value of θ should be greater than θ = n⋅W − P, where:
• n = Number of neurons in the input layer
• W = Positive weight
• P = Negative weight
AND Function Implementation Using
McCulloch-Pitts Neuron
1. Truth Table of AND Function
• The truth table for the AND function clearly illustrates the
relationship between the inputs and the output.
• In the context of logic gates, an AND gate outputs a high signal
(1) only when all its inputs are high.
• From the truth table, it is evident that the output is high only
when both inputs are high (1).
• If either input is low (0), the output is also low (0).
• This binary relationship forms the basis of how the McCulloch-
Pitts neuron will function in this scenario.
AND Function Implementation Using
McCulloch-Pitts Neuron
2. Understanding Weights and Threshold
• One crucial aspect to note is that the McCulloch-Pitts neuron does
not utilize a built-in training algorithm like modern neural networks.
• Instead, we must manually analyze the input combinations to
determine the optimal weights and threshold values required for
the desired output behavior.
• For implementing the AND function, we can assume the following
weights for our inputs:
W1=1 W2=1
• These weights represent the contribution of each input to the net
input of the neuron.
• With these assumptions in place, we can now calculate the net
input at the output neuron based on various input combinations:
AND Function Implementation Using
McCulloch-Pitts Neuron
Net Input Calculations
• Let’s compute the net input yn for each possible combination of
inputs:
1. For inputs X1=1 and X2=1:
yn=(X1⋅W1)+(X2⋅W2)=(1⋅1)+(1⋅1)=2
• In this scenario, both inputs are high, resulting in a net input of 2.

2. For inputs X1=1 and X2=0:

yn=(1⋅1)+(0⋅1)=1
• Here, the first input is high while the second is low, yielding a net
input of 1.
AND Function Implementation Using
McCulloch-Pitts Neuron
3. For inputs X1=0 and X2=1:
yn=(0⋅1)+(1⋅1)=1
• Similar to the previous case, only one input is high, resulting in a
net input of 1.

4. For inputs X1=0 and X2=0:

yn=(0⋅1)+(0⋅1)=0
• Both inputs being low produces a net input of 0, indicating that
the neuron does not fire.
AND Function Implementation Using
McCulloch-Pitts Neuron Neuron
3. Determining the Threshold Value
• To ensure that the McCulloch-Pitts neuron fires (outputs 1) only
when both inputs are high (1), we need to establish an
appropriate threshold value, denoted as θ.
• Based on our previous calculations, we can conclude:
• If θ≥2, the neuron will fire when both inputs are high (1).
• If θ<2, the neuron will not fire in cases where either input is low
(0).
• Thus, we determine that the threshold value θ should be set to 2
to achieve the desired behavior of the AND function.
AND Function Implementation Using
McCulloch-Pitts Neuron
• Additionally, we can calculate the threshold value using the
following equation:
• Θ ≥ n⋅W − P
Where:
• n is the number of neurons in the input layer (in this case, n=2).
• W is the positive weight (here, W=1).
• P is the negative weight, which in this case is 0 since we do not
have inhibitory inputs.
AND Function Implementation Using
McCulloch-Pitts Neuron
• Substituting these values into the equation gives:
θ ≥ 2⋅1 − 0 = 2
• Thus, both methods confirm that the threshold value should be
set to 2.
AND Function Implementation Using
McCulloch-Pitts Neuron
4. Final Activation Function
• The final activation function for the McCulloch-Pitts neuron can be
expressed mathematically as:
F(yn) = 1if yn ≥ 2
F(yn) = 0 otherwise
• This function confirms that the neuron will fire (output 1) only
when both inputs X1 and X2 are equal to 1, effectively
implementing the AND logical function.
• The simplicity of the McCulloch-Pitts model highlights its
significance as a foundational concept in neural network theory,
paving the way for more complex learning algorithms and
structures in modern artificial intelligence.
ANDNOT Function Implementation Using
McCulloch-Pitts Neuron
1. Truth Table of ANDNOT Function
• The truth table for the ANDNOT function clearly illustrates the
relationship between the inputs and the output.
• In this case, we have two inputs: X1 and X2, and Y is the output.
The ANDNOT function outputs a high signal (1) only when X1 is
high (1) and X2 is low (0).
• This behavior can be represented as follows:
• From the truth table, it is evident that the output is high only
when X1 is high (1) and X2 is low (0).
• In all other cases, the output is low (0). This binary relationship
forms the basis of how the McCulloch-Pitts neuron will function in
this scenario.
ANDNOT Function Implementation Using
McCulloch-Pitts Neuron
2. Understanding Weights and Threshold
• One crucial aspect to note is that the McCulloch-Pitts neuron does
not utilize a built-in training algorithm. Therefore, we need to
analyze the input combinations to determine the optimal weights
and threshold values required for the desired output behavior.
• For implementing the ANDNOT function, we can assume the
following weights for our inputs:
W1 = 1
W2 = −1
• These weights indicate that the first input contributes positively
to the net input, while the second input contributes negatively.
With these assumptions in place, we can now calculate the net
input at the output neuron based on various input combinations:
ANDNOT Function Implementation Using
McCulloch-Pitts Neuron
Net Input Calculations
• Let’s compute the net input yn for each possible combination of
inputs:
1. For inputs X1=1 and X2=1:
yn=(X1⋅W1)+(X2⋅W2)=(1⋅1)+(1⋅−1)=0
• In this scenario, both inputs are high, resulting in a net input of 0.

2. For inputs X1=1 and X2=0:

yn=(1⋅1)+(0⋅−1)=1
• Here, the first input is high while the second is low, yielding a net
input of 1.
ANDNOT Function Implementation Using
McCulloch-Pitts Neuron
3. For inputs X1=0 and X2=1:
yn=(0⋅1)+(1⋅−1)=−1
• In this case, only the second input is high, resulting in a negative
net input of -1.

4. For inputs X1=0 and X2=0:

yn=(0⋅1)+(0⋅−1)=0
• Both inputs being low produces a net input of 0, indicating that
the neuron does not fire.
ANDNOT Function Implementation Using
McCulloch-Pitts Neuron
3. Determining the Threshold Value
• To ensure that the McCulloch-Pitts neuron fires (outputs 1) only
when X1 is high (1) and X2 is low (0), we need to establish an
appropriate threshold value, denoted as θ. Based on our previous
calculations, we can conclude:

• If θ≤1, the neuron will fire when X1 is 1 and X2 is 0.

• If θ>1, the neuron will not fire in cases where X2 is high (1).

• Thus, we determine that the threshold value θ should be set to 1

to achieve the desired behavior of the ANDNOT function.
ANDNOT Function Implementation Using
McCulloch-Pitts Neuron
• Additionally, we can calculate the threshold value using the
following equation:
θ≥n⋅W−P
Where:
• n is the number of neurons in the input layer (in this case, n=2).
• W is the positive weight (here, W=1).
• P is the negative weight (in this case, P=1).
• Substituting these values into the equation gives:
θ≥2⋅1−1=1
• Thus, both methods confirm that the threshold value should be
set to 1.
ANDNOT Function Implementation Using
McCulloch-Pitts Neuron
4. Final Activation Function
• The final activation function for the McCulloch-Pitts neuron can be
expressed mathematically as:
F(yn) = 1if yn≥1
F(yn) = 0 otherwise

• This function confirms that the neuron will fire (output 1) only
when X1 is equal to 1 and X2 is equal to 0, effectively
implementing the ANDNOT logical function.
Hebb Network / Hebbian Rule
• 1. Introduction to Hebbian Rule
• The Hebbian Rule is one of the simplest learning rules under artificial neural
networks. It was introduced in 1949 by Donald Hebb.
• According to Hebb, learning in the brain occurs due to changes in the synaptic
gap, which can be attributed to metabolic changes or growth. This rule is based on
the biological processes that occur in the brain during learning.
• To understand the Hebbian Rule, let’s first consider the structure of a biological
neuron. Each neuron has three main parts:
• Cell Body (Soma): This is where the nucleus is located.
• Axon: A long connection extending from one neuron to another.
• Dendrites: Small nerves connected to the cell body that transmit electrical
impulses.
Hebb Network / Hebbian Rule
• The electrical impulses are passed from one neuron to another through synapses,
which are small gaps between neurons. When learning occurs, metabolic changes
happen in the synaptic gap, leading to the formation of new connections between
neurons.
• 2. Hebbian Rule Definition
• Donald Hebb's definition of the Hebbian Rule is as follows:
• When an axon of cell A is near enough to excite cell B and repeatedly or
persistently fires it, some growth process or metabolic change takes place in one
or both cells.
Hebb Network / Hebbian Rule
• In simpler terms, if neuron A excites neuron B frequently, changes occur in their
synaptic gap, which strengthens the connection between the two neurons.
• Hebb's Rule was inspired by the way learning happens in the human brain.
• A relatable example of this can be seen when learning to drive.
• Initially, when you start driving, you are conscious of every action, like turning or
reversing.
• However, over time, as it becomes a habit, you can drive effortlessly while doing
other tasks, like listening to music.
• This example demonstrates how neurons become trained and perform tasks
automatically over time, which is the core idea of Hebb’s learning theory.
Hebb Network / Hebbian Rule
• 3. Principles of Hebbian Rule
The Hebbian Rule follows two basic principles:

• If two neurons on either side of a connection are activated synchronously (both

are on), the weight between them is increased.

• If two neurons on either side of a connection are activated asynchronously (one is

on, the other is off), the weight between them is decreased.
Hebb Network / Hebbian Rule
4. Hebbian Rule Formula
• According to Hebb's learning rule, when two interconnected neurons are activated
simultaneously, the weights between them increase. The change in weight is represented
by the following formula:
Wnew = Wold + ΔW

ΔW = xi * y
Where:
• Wnew: New weight after learning
• Wold: Initial weight
• ΔW: Change in weight (synaptic gap change)
• xi: Input vector
• y: Output vector
Hebb Network / Hebbian Rule
• Training Algorithm for Hebbian Network
• The training algorithm for the Hebbian Network follows these
steps:
• Initialize the weights and bias: Set the weights and bias to
either zero or random values.
• For each input-output pair: Perform the following:
• Set the activation for the input unit: Xi = Si
• Set the activation for the output unit: y = T
• Update the weights and bias using the formulas:
Wnew = Wold + xi * y

Bnew = Bold + y
Hebb Network / Hebbian Rule
• Repeat the process until there are no more input-output pairs.
• This training algorithm allows the Hebbian Network to learn and
update its weights and bias, forming the basis for unsupervised
learning in neural networks.
Hebb Network / Hebbian Rule
Applications of Hebbian Rule
• The Hebbian learning rule is widely used in various applications,
including:
• Pattern Association: Associating different patterns based on
inputs.
• Pattern Categorization: Categorizing inputs based on learned
patterns.
• Pattern Classification: Classifying new inputs based on
previously learned patterns.
• In conclusion, Hebb's rule and network play a foundational role in
understanding how neurons learn and adapt. The rule is simple
yet powerful, forming the basis for more complex neural network
models in artificial intelligence.
Hebbian Network for Logical AND
Function (Bipolar Inputs)
• We are tasked with designing a Hebbian network to implement
the logical AND function using bipolar inputs (1 or -1) and targets.

We will initialize the weights (W1, W2) and bias (B) to zero and use the Hebbian learning
rule to update the weights based on the input-output pairs.
Weight Update Rule
The weight and bias update rules in Hebbian learning are as follows:
W1(new) = W1(old) + X1 * Y

W2(new) = W2(old) + X2 * Y

B(new) = B(old) + Y
Step-by-Step Calculation

Initial Weights: W1 = 0, W2 = 0, B = 0
Hebbian Network for Logical AND
Function (Bipolar Inputs)
Hebbian Network for Logical AND
Function (Bipolar Inputs)
• There are 4 training samples, so there will be 4 iterations. Also, the
activation function used here is Bipolar Sigmoidal Function so the range is
[-1,1].
• Step 1 :
• Set weight and bias to zero, w = [ 0 0 0 ]T and b = 0.
• Step 2 :
• Set input vector Xi = Si for i = 1 to 4.
X1 = [ -1 -1 1 ]T
X2 = [ -1 1 1 ]T
X3 = [ 1 -1 1 ]T
X4 = [ 1 1 1 ]T
• Step 3 :
Hebbian Network for Logical AND
Function (Bipolar Inputs)
Step 4 :
• Modifying weights using Hebbian Rule:
• First iteration -
• w(new) = w(old) + x1y1 = [ 0 0 0 ]T + [ -1 -1 1 ]T . [ -1 ] = [ 1 1 -1 ]T
• For the second iteration, the final weight of the first one will be used
and so on.
• Second iteration -
• w(new) = [ 1 1 -1 ]T + [ -1 1 1 ]T . [ -1 ] = [ 2 0 -2 ]T
• Third iteration -
• w(new) = [ 2 0 -2]T + [ 1 -1 1 ]T . [ -1 ] = [ 1 1 -3 ]T
• Fourth iteration -
• w(new) = [ 1 1 -3]T + [ 1 1 1 ]T . [ 1 ] = [ 2 2 -2 ]T
Hebbian Network for Logical AND
Function (Bipolar Inputs)
• So, the final weight matrix is [ 2 2 -2 ]T
• Testing the network :
Hebbian Network for Logical AND
Function (Bipolar Inputs)
• For x1 = -1, x2 = -1, b = 1, Y = (-1)(2) + (-1)(2) + (1)(-2) = -6
• For x1 = -1, x2 = 1, b = 1, Y = (-1)(2) + (1)(2) + (1)(-2) = -2
• For x1 = 1, x2 = -1, b = 1, Y = (1)(2) + (-1)(2) + (1)(-2) = -2
• For x1 = 1, x2 = 1, b = 1, Y = (1)(2) + (1)(2) + (1)(-2) = 2