ARTIFICIAL NEURAL NETWORKS(CSEN3011)

UNIT-IV
Attractor Neural Networks

By

Dr.Satyabrata Dash
Assistant Professor
Department of Computer Science and
Engineering, GITAM Deemed to be
University
 Course Educational Objectives( ANN)
1. To understand the architecture, learning algorithm and issues of various neural
networks.
2. Analyze ANN learning, Error correction learning, Memory-based learning,
Competitive learning and Boltzmann learning
3. To adopt gradient - descent techniques in real time applications
4. Provide knowledge on Generalization and function approximation and various
5. architectures of building an ANN
6. Implement and learn the applications of Self-organization Map

2
 SYLLABUS

UNIT 4 Attractor Neural Networks 9 hours, P - 6 hours

• Associative Learning,
• Attractor Associative Memory,
• Linear Associative memory,
• Hopfield Network,
• application of Hopfield Network,
• Brain State in a Box neural Network,
• Simulated Annealing,
• Boltzmann Machine,
• Bidirectional Associative Memory.

3
 SYLLABUS

Textbooks: 1. Neural Networks and Deep Learning - Charu C. Aggarwal, Springer

International Publishing AG, part of Springer Nature 2018 (Chapters 1, 2, 3)
2. Neural Networks A Classroom Approach– Satish Kumar, McGraw Hill Education (India)
Pvt. Ltd, Second Edition. (Chapters 4, 5)

4
 Introduction to Associative Learning

Associative thinking is a mental process that involves linking or associating

one idea, thought, or concept with another, often seemingly unrelated idea or
concept.

It some times due to 'Sense, Feeling, Tone and Intention'

5
1. Associative learning is a type of learning process in psychology where individuals
form associations or connections between Stimuli and Behaviors.
2. Learning occurs when you learn something based on a new stimulus.
3. It is a fundamental concept in the field of behavioral psychology and is often used
to explain how organisms, including humans, acquire new behaviors through their
experiences with the environment.
4. It is expressed as the modification of existing behaviors, or the development of
novel behaviors, that reflects the conscious or unconscious recognition

• A stimulus is something that encourages growth or causes activity. It can

also be something that causes something else to happen, develop, or become
more active.

6
Primary Forms of Associative Learning

7
 Primary Forms of Associative Learning

There are two primary forms of associative learning:

• Classical Conditioning
In classical conditioning, a neutral stimulus (one that does not naturally elicit a
response) is paired with an unconditioned stimulus (one that naturally triggers a
response) to produce a conditioned response.

• Operant Conditioning
In operant conditioning, behaviors are strengthened or weakened based on the
consequences they produce. Behaviors that are followed by desirable consequences
tend to be repeated, while those followed by undesirable consequences are less
likely to be repeated.

8
Associative Memory Network

9
 Associative Memory Network

Associative Memory
1. It is a supervised learning model in Artificial Neural Network.
2. An associate memory network refers to a content addressable memory structure
that associates a relationship between the set of input patterns and output patterns.
3. A content addressable memory structure is a kind of memory structure that enables
the recollection of data based on the intensity of similarity between the input
pattern and the patterns stored in the memory.
4. These types of memories are also called Content-Addressable Memory (CAM).
5. Associative memory makes a parallel search with the stored patterns as data files.
6. Following are the two types of associative memories we can observe
1.Auto Associative Memory
2. Hetero Associative memory

10
 Auto Associative Memory
An auto-associative memory recovers a previously stored pattern that most closely relates
to the current pattern. It is also known as an auto-associative correlator.

Consider x[1], x[2], x[3],….. x[M], be the number of stored pattern vectors, and let
x[m] be the element of these vectors, showing characteristics obtained from the
patterns. The auto-associative memory will result in a pattern vector x[m] when putting
a noisy or incomplete version of x[m].

11
ANN used to represent auto-associative Memory

12
 Hetero Associative memory
Hetero Associative memory
• In a hetero-associate memory, the recovered pattern is generally different from the
input pattern not only in type and format but also in content. It is also known as a
hetero-associative correlator

1. Consider we have a number of key response pairs {a(1), x(1)},

{a(2),x(2)},…..,{a(M), x(M)}. The hetero-associative memory will give a pattern
vector x(m) when a noisy or incomplete version of the a(m) is given.
2. Neural networks are usually used to implement these associative memory models
called neural associative memory (NAM). The linear associate is the easiest artificial
neural associative memory.
3. These models follow distinct neural network architecture to memorize data.
13
ANN used to represent hetero-associative Memory

14
 Linear Associative memory
1. Linear Associative Memory (LAM) is a type of artificial neural network or
information retrieval system used in the field of machine learning and cognitive science.
2. It is a simplified model inspired by the way human memory works, specifically the idea
of association between items or concepts
3. It is a collection of simple processing units which have a quite complex collective
computational capability and behavior.
4. stores and retrieves information based on linear associations between input and
output patterns.
5. It's a simplified form of associative memory that uses a linear transformation to map
inputs to outputs.
6. The linear associator model is a feed-forward type network where produced output is
in the form of single feed-forward computation.

15
Linear Associative memory

16
 Linear Associative memory
1. The model comprises of two layers of processing units, one work as an input layer
while the other work as an output layer.
2. The input is directly associated with the outputs, through a series of weights.
3. The connections carrying weights link each input to every output.
4. The addition of the products of the weights and the input is determined in each neuron
node.
5. All p inputs units are associated to all q output units via associated weight matrix
W = [wij]p * q where wij describes the strength of the unidirectional association of
the ith input unit to the jth output unit.
The connection weight matrix stores the z different associated pattern pairs {(Xk,Yk);
k= 1,2,3,…,z}. Constructing an associative memory is building the connection weight
matrix w such that if an input pattern is presented, the stored pattern associated with the
input pattern is recovered.

17
 Attractor Associative Memory(Attractor Neural Network (ANNs))

1. An Attractor Associative Memory (AAM), also known as a Hopfield network,

is a type of recurrent artificial neural network used for associative memory tasks.
2. It was introduced by John Hopfield in 1982 and is named after him.
3. AAMs are a class of recurrent neural networks that have the ability to store and
retrieve patterns or memories.
4. Such networks can store a high number of different memories and perform
associative retrieval effectively.
5. Attractor neural networks (ANNs) are a type of network that can store many
different memories and perform associative retrieval.
6. These networks have predefined weights that are probed by different types of
input. If the stable state of the network is different during and after the input, it can
serve as a model of associative memory

18
 Attractor Neural Network (ANNs))
1. An attractor neural network, also known as a recurrent neural network (RNN) with
attractor dynamics, is a type of artificial neural network designed to model and simulate
certain properties of biological neural networks, especially those related to memory and
pattern recognition.
2. These networks are characterized by the presence of attractors, which are stable states or
patterns of neural activity that the network tends to converge to under certain conditions.
3. Attractors: Attractors are stable states of neural activity in the network. These can be
fixed-point attractors (where the network settles into a single stable state) or cyclic
attractors (where the network's activity follows a repeating pattern). Attractors
represent memory states or patterns that the network can recall or maintain.
4. Attractor neural networks are often used to model memory processes and pattern
recognition. They can store and retrieve information through the stable attractor states.
When presented with incomplete or noisy input, the network can often complete or
recognize patterns based on its stored attractors.
19
 Application of Attractor Neural Network (ANNs))
Attractor neural networks are used for several reasons in the field of artificial intelligence and
neuroscience due to their unique properties and capabilities
Memory and Attractor neural networks are particularly well-suited for modeling memory
Pattern processes and pattern recognition. They can store and retrieve information
Recognition: through stable attractor states, making them useful for tasks like associative
memory and content-addressable memory retrieval.
Robustness to These networks can tolerate noisy or incomplete inputs and still retrieve the
Noise: closest stored pattern or memory, which is valuable in applications where input
data may be corrupted or incomplete.
Cognitive Attractor networks have been used to model cognitive processes, such as how
Modeling: the brain stores and retrieves information. They provide a simplified but
effective way to understand memory and decision-making mechanisms.
Optimization They have been applied to solve combinatorial optimization problems by
Problems: mapping the problem space to the attractor states. This can lead to efficient
solutions for problems like the traveling salesman problem or graph coloring.
Neuroscience Attractor neural networks are used in neuroscience to gain insights into how
Research: certain brain regions work, especially those associated with memory and
decision-making. These models can help researchers understand the dynamics
of neural systems.
20
Hopfield Network

21
 Hopfield Network

1. A Hopfield network is a particular type of single-layered neuron network.

2. Dr. John J. Hopfield invented it in 1982.
3. These networks were introduced to collect and retrieve memory and store various
patterns.
4. Also, auto-association and optimization of the task can be done using these networks.
5. In this network, each node is fully connected(recurrent) to other nodes.
6. These nodes exist only in two states: ON (1) or OFF (0).
7. These states can be restored based on the input received from other nodes.
8. Unlike other neural networks, the output of the Hopfield network is finite.
9. Also, the input and output sizes must be the same in these networks

22
 Hopfield Network
.A recurrent network is a Hopfield network when
• The neurons have discrete output (for convenience we use bipolar encoding, i.e.,
activation function is the sign function)
• Each neuron has a threshold
• Each pair of neurons is connected by a weighted connection. The weight matrix
is symmetric and has a zero diagonal (no connection from a neuron to itself)
• The Hopfield network consists of associative memory. This memory allows the
system to retrieve the memory using an incomplete portion.
• The network can restore the closest pattern using the data captured in associative
memory.
• This feature of Hopfield networks makes it a good candidate for pattern recognition.

23
 Architecture of Hopfield Network

The architecture of the Hopfield network consists of the following elements:

1. Individual nodes preserve their states until required an update.

2. The node to be updated is selected randomly.
3. Each node is connected to all other nodes except itself.
4. The state of each node is either 0/1 or 1/-1.
5. The Hopfield network structure is symmetric, i.e., Wij = Wji for all i's and j's.

24
Hopfield Network

25
 Architecture of Hopfield Network

In the diagram, each symbol represents:

1. x1, x2, x3 - represents the input.

2. y1, y2, y3 - represents output obtained from each node.

3. Wij - represents the weight associated with the

connection from i to j.

26
 TRAINING Algorithm for Hopfield Network

1. During training of discrete Hopfield network, weights will be updated.

2. As we know that we can have the binary input vectors as well as bipolar input
vectors.
3. Hence, in both the cases, weight updates can be done with the following relation
4. Instead of the usual gradient descent-based training, Hopfield Networks use a form of
Hebbian learning, where synaptic weights are updated based on the correlations
between the input patterns and their desired outputs.

27
 TRAINING Algorithm for Hopfield Network

The training algorithm for a Hopfield Network:

1. Initialize Weights: Start by initializing the synaptic weights of the network. Typically,
you can set the weights to zero or small random values.
2. Training Patterns: Define a set of training patterns that you want the network to learn.
These patterns should be binary, where each element is either +1 or -1.
3. Weight Update (Hebbian Learning): For each pair of patterns (i and j), update the
synaptic weights according to the Hebbian learning rule:
4. Diagonal Elements: Set the diagonal elements of the weight matrix to zero. This
prevents neurons from influencing themselves during recall.
5. Thresholds (Optional): You can add thresholds (biases) to each neuron to control their
activation during recall.
6. Repeat if Necessary: Depending on your application, you may need to repeat steps 3-5
multiple times or until the network reaches a desired level of stability.
28
 TRAINING Algorithm for Hopfield Network

The rule is different for binary and bipolar. For storing a set of 'N' input patterns [ X(n)
where n = 1 to N ], the rule followed is as follows:

Binary Input:

Bipolar Input:

29
 Testing Algorithm for Hopfield Network
Testing Algorithm
1. Initialize the weights using the above training algorithm rules.
2. Follow steps 3 to 7 for each input vector 'Xi'.
3. Assign the value of external input vector 'Xi' to initial activators Yi', for all i = 1 to N.
4. Follow steps 5 to 7 for each initial activator 'Yi'.
5. Calculate the network's net input 'Yin' using the below rule:

Based on the calculated net input and threshold value, calculate the output. There are three
different conditions for the output. These are:

Broadcast the obtained output 'Yi' to all other units. And update the activation vector
and Test the network.
30
 Application of Hopfield Network
Hopfield Network are used for several reasons in the field of artificial intelligence and neuroscience
due to their unique properties and capabilities
Memory and Hopfield Network are particularly well-suited for modeling memory processes
Pattern and pattern recognition. They can store and retrieve information through stable
Recognition: attractor states, making them useful for tasks like associative memory and
content-addressable memory retrieval.
Robustness to These networks can tolerate noisy or incomplete inputs and still retrieve the
Noise: closest stored pattern or memory, which is valuable in applications where input
data may be corrupted or incomplete.
Cognitive Hopfield Network have been used to model cognitive processes, such as how
Modeling: the brain stores and retrieves information. They provide a simplified but
effective way to understand memory and decision-making mechanisms.
Optimization They have been applied to solve combinatorial optimization problems by
Problems: mapping the problem space to the attractor states. This can lead to efficient
solutions for problems like the traveling salesman problem or graph coloring.
Neuroscience Hopfield Network are used in neuroscience to gain insights into how certain
Research: brain regions work, especially those associated with memory and decision-
making. These models can help researchers understand the dynamics of neural
systems.
31
 Brain State in a Box(BSB) neural Network
1. The Brain-State-in-a-Box neural network is a nonlinear auto-associative neural
network and can be extended to hetero-association with two or more layers.
2. It is also similar to Hopfield network.
3. It was proposed by J.A. Anderson, J.W. Silverstein, S.A. Ritz and R.S. Jones in 1977.
4. BSB is a fully connected network with the maximum number of nodes depending upon
the dimensionality n of the input space.
5. All the neurons are updated simultaneously.
6. Neurons take values between -1 to +1.

NN&DL by Dr.Satyabrata Dash 32

 Brain State in a Box(BSB) neural Network

1. State space of a network with N nodes is a N-dimensional hypercube.

(centered in the origin)

2. Continuous values for neurons
3. Training patterns with values in {-1,1}
4. Like the Hopfield model, the BSB model is an energy-minimizing network: certain
corners of the unit hypercube act as attractors
5. BSB: clustering capabilities useful for data representation and concept formation
6. Hopfield: used e.g. for content-addressable memory

NN&DL by Dr.Satyabrata Dash 33

 Brain State in a Box(BSB) neural Network Algorithm

• The positive feedback in the BSB model causes x(0) to increase in Euclidean
norm with an increasing number of iterations until it hits a wall of the box (unit
hypercube) then slides along the wall and eventually ends in a stable corner of
the box, where it cannot get out of the box.
BSB Algorithm
• BSB model is basically a positive feedback system with amplitude limitation.
• BSB algorithm defined by the given equation.

NN&DL by Dr.Satyabrata Dash 34

 Brain State in a Box(BSB) neural Network Algorithm

• With X(0) initial state vector of the model representing an input activation pattern
B >0 small constant called feedback factor ,
• W can be considered by a single layer linear neuron network
and the activation function is pricewise linear

NN&DL by Dr.Satyabrata Dash 35

 Brain State in a Box(BSB) neural Network
1. The BSB model was principally used to model the effects and mechanisms found in
psychology and cognitive science.
2. A possible function of the BSB network is to identify a pattern from a given noisy
version. The BSB network can also be used as a pattern identifier that utilizes a smooth
proximity measure and generates stable decision boundaries.

NN&DL by Dr.Satyabrata Dash 36

Bidirectional Associative Memory

NN&DL by Dr.Satyabrata Dash 37

 Bidirectional Associative Memory

• Bidirectional Associative Memory (BAM) is a supervised learning model in

Artificial Neural Network.
• This is hetero-associative memory, for an input pattern, it returns another pattern
which is potentially of a different size.
• This phenomenon is very similar to the human brain.
• Human memory is necessarily associative.
• It uses a chain of mental associations to recover a lost memory like associations of
faces with names, in exam questions with answers, etc.
• In such memory associations for one type of object with another, a Recurrent
Neural Network (RNN) is needed to receive a pattern of one set of neurons as an
input and generate a related, but different, output pattern of another set of neurons.

NN&DL by Dr.Satyabrata Dash 38

Bidirectional Associative Memory

NN&DL by Dr.Satyabrata Dash 39

Bidirectional Associative Memory

NN&DL by Dr.Satyabrata Dash 40

 Bidirectional Associative Memory

Algorithm:
• Storage (Learning): In this learning step of BAM, weight matrix is calculated
between M pairs of patterns (fundamental memories) are stored in the synaptic
weights of the network following the equation .
•
𝑀

𝑊 = ෍ 𝑋𝑚 ∗ 𝑌𝑚 𝑇
𝑚=1

• . Testing: We have to check that the BAM recalls perfectly Ym for

corresponding Xm and recalls Xm for corresponding Ym. Using,

• All pairs should be recalled accordingly.

NN&DL by Dr.Satyabrata Dash 41

 Bidirectional Associative Memory

• Retrieval: For an unknown vector X (a corrupted or incomplete version of a pattern

from set A or B) to the BAM and retrieve a previously stored association:
X not equal to Xm, m=1,2,3,....m

a. Initialize the BAM: X(0)=X, p=0

b. Calculate the BAM output at iteration p:

c. Update the input vector X(p):

d. Repeat the iteration until convergence, when input and output remain unchanged.

• All pairs should be recalled accordingly.

NN&DL by Dr.Satyabrata Dash 42

 Bidirectional Associative Memory Advantages
Bidirectional Associative Memory (BAM) is a type of recurrent neural network used for
associative memory tasks. It has some advantages, particularly in specific applications,
when compared to other neural network architectures.

Advantages Explanations
Associative BAM is designed specifically for associative memory tasks, allowing
Memory: it to store and retrieve patterns in a bi-directional manner. This means
it can recall patterns from partial or noisy input, making it suitable for
tasks like pattern completion and recognition.
Robustness to BAM can tolerate some level of noise in the input patterns. It can
Noise: often reconstruct a stored pattern even if the input is corrupted or
incomplete. This makes it useful in real-world scenarios where data
may not be perfect.
Simple BAM has a relatively simple and straightforward architecture
Architecture: compared to some other neural networks, like deep feedforward
networks or recurrent neural networks. This simplicity can make it
easier to understand and implement for certain applications.

NN&DL by Dr.Satyabrata Dash 43

 Bidirectional Associative Memory Advantages
Advantages Explanations
Fast Recall: BAM can retrieve stored patterns quickly once they have been
learned. The recall process in BAM is deterministic and does not
require an iterative process like backpropagation, which can be
computationally expensive.
Parallel BAM can process multiple patterns simultaneously because of its
Processing: architecture. This can lead to faster pattern retrieval and make it
suitable for tasks where efficiency is essential.

Stability: BAM networks are stable and tend to converge to a stable state during
recall, which is beneficial in applications requiring consistent and
predictable responses.

Pattern BAM can effectively separate and store multiple patterns, even if
Separation: there is some overlap in their features. This makes it useful for tasks
where different patterns need to be stored and retrieved without
interference.

NN&DL by Dr.Satyabrata Dash 44

Simulated Annealing

NN&DL by Dr.Satyabrata Dash 45

Boltzmann learning

NN&DL by Dr.Satyabrata Dash 46

 Boltzmann learning

1. Boltzmann Learning, specifically referring to Boltzmann Machines (BMs).

2. It is a class of neural networks used for generative modeling and unsupervised learning
3. These are stochastic learning processes having recurrent structure and are the basis of
the early optimization techniques used in ANN.
4. Boltzmann Machine was invented by Geoffrey Hinton and Terry Sejnowski in 1985.
5. Boltzmann learning is statistical in nature, and is derived from the field of
thermodynamics.
6. It is similar to error-correction learning and is used during supervised training.
7. In this algorithm, the state of each individual neuron, in addition to the system output,
are taken into account.
8. In this type of learning an error signal is used to train the system in each iteration

NN&DL by Dr.Satyabrata Dash 47

 Objective of Boltzmann Machine
The main purpose of Boltzmann Machine is to optimize the solution of a problem. It is
the work of Boltzmann Machine to optimize the weights and quantity related to that
particular problem.

Architecture
The following diagram shows the
architecture of Boltzmann machine. It is
clear from the diagram, that it is a two-
dimensional array of units. Here, weights
on interconnections between units are –p
where p > 0. The weights of self-
connections are given by b where b > 0.

NN&DL by Dr.Satyabrata Dash 48

 Boltzmann learning
Training Algorithm
As we know that Boltzmann machines have fixed weights, hence there will be no training
algorithm as we do not need to update the weights in the network. However, to test the
network we have to set the weights as well as to find the consensus function [Math
Processing Error]
.
Boltzmann machine has a set of units Ui and Uj and has bi-directional connections on them.

• We are considering the fixed weight say wij.

• wij ≠ 0 if Ui and Uj are connected.

• There also exists a symmetry in weighted interconnection, i.e. wij = wji.

• wii also exists, i.e. there would be the self-connection between units.

• For any unit Ui, its state ui would be either 1 or 0.

NN&DL by Dr.Satyabrata Dash 49

 Boltzmann learning
The main objective of Boltzmann Machine is to maximize the Consensus Function
which can be given by the following relation

• Now, when the state changes from either 1 to 0 or from 0 to 1, then the change in
consensus can be given by the following relation −
• Here ui is the current state of Ui.

• The variation in coefficient (1 - 2ui) is given by the following relation −

• Generally, unit Ui does not change its state, but if it does then the information would be
residing local to the unit. With that change, there would also be an increase in the
consensus of the network.

• Probability of the network to accept the change in the state of the unit is given by the
following relation −

• Here, T is the controlling parameter. It will decrease as CF reaches the maximum value.

NN&DL by Dr.Satyabrata Dash 50

 Boltzmann learning
Testing Algorithm
Step 1 − Initialize the following to start the training − Weights representing the constraint
of the problem Control Parameter T
Step 2 − Continue steps 3-8, when the stopping condition is not true.

Step 3 − Perform steps 4-7.

Step 4 − Assume that one of the state has changed the weight and choose the integer I, J
as random values between 1 and n.

Step 5 − Calculate the change in consensus as follows −

NN&DL by Dr.Satyabrata Dash 51

 Boltzmann learning
Step 6 − Calculate the probability that this network would accept the change in state
Step 7 − Accept or reject this change as follows −

Case I − if R < AF, accept the change.

Case II − if R ≥ AF, reject the change.

Here, R is the random number between 0 and 1.

Step 8 − Reduce the control parameter [Math Processing Error]

as follows −

T[Math Processing Error] = ⁡

0.95T[Math Processing Error]

Step 9 − Test for the stopping conditions which may be as follows −

Temperature reaches a specified value

There is no change in state for a specified number of iterations

NN&DL by Dr.Satyabrata Dash 52

Thank You

53