UNIT-2

Unsupervised Learning Network- Introduction, Fixed Weight Competitive Nets,

Maxnet, Hamming Network, Kohonen Self-Organizing Feature Maps, Learning Vector
Quantization, Counter Propagation Networks, Adaptive Resonance Theory Networks.
Special NetworksIntroduction to various networks

Fixed Weight Competitive Nets

During training process also the weights remains fixed in these competitive networks. The
idea of competition is used among neurons for enhancement of contrast in their activation
functions. In this, two networks- Maxnet and Hamming networks

Maxnet
Maxnet network was developed by Lippmann in 1987. The Maxner serves as a sub net for
picking the node whose input is larger. All the nodes present in this subnet are fully
interconnected and there exist symmetrical weights in all these weighted interconnections.

Architecture of Maxnet
The architecrure of Maxnet is a fixed symmetrical weights are present over the weighted
interconnections. The weights between the neurons are inhibitory and fixed. The Maxnet with
this structure can be used as a subnet to select a particular node whose net input is the largest.

Testing Algorithm of Maxnet

The Maxnet uses the following activation function:
Testing algorithm
Step 0: Initial weights and initial activations are set. The weight is set as [0 < ε < 1/m], where
"m" is the total number of nodes. Let
Xj(0) = input the node Xj

and

Step 1: Perform Steps 2-4, when stopping condition is false.

Step 2: Update the activations of each node. For j = 1 to m,

Step 3: Save the activations obtained for use in the next iteration. For j = 1 to m,

Step 4: Finally, test the stopping condition for convergence of the network. The following is
the stopping condition: If more than one node has a nonzero activation, continue; else stop.

Hamming Network
The Hamming network is a two-layer feedforward neural network for classification of binary
bipolar n-tuple input vectors using minimum Hamming distance denoted as DH(Lippmann,
1987). The first layer is the input layer for the n-tuple input vectors. The second layer (also
called the memory layer) stores p memory patterns. A p-class Hamming network has p output
neurons in this layer. The strongest response of a neuron is indicative of the minimum
Hamming distance between the stored pattern and the input vector.

Hamming Distance
Hamming distance of two vectors, x and y of dimension n
x.y = a - d
where: a is number of bits in aggreement in x & y(No.of Similaritie bits in x & y), and d is
number of bits different in x and y(No.of Dissimilaritie bits in x & y).
The value "a - d" is the Hamming distance existing between two vectors. Since, the total
number of components is n, we have,
n=a+d
i.e., d = n - a
On simplification, we get
x.y = a - (n - a)
x.y = 2a - n
2a = x.y + n
a = 12x.y + 12n
From the above equation, it is clearly understood that the weights can be set to one-half the
exemplar vector and bias can be set initially to n/2

Testing Algorithm of Hamming Network

Step 0: Initialize the weights. For i = 1 to n and j = 1 to m,

Initialize the bias for storing the "m" exemplar vectors. For j = 1 to m,

Step 1: Perform Steps 2-4 for each input vector x.

Step 2: Calculate the net input to each unit Yj, i.e.,

Step 3: Initialize the activations for Maxnet, i.e.,

Step 4: Maxnet is found to iterate for finding the exemplar that best matches the
input patterns.

Kohonen Self-Organizing Feature Maps

Self-Organizing Feature Maps(SOM) was developed by Dr. Teuvo Kohonen in
1982. Kohonen Self-Organizing feature map (KSOM) refers to a neural network,
which is trained using competitive learning. Basic competitive learning implies
that the competition process takes place before the cycle of learning. The
competition process suggests that some criteria select a winning processing
element. After the winning processing element is selected, its weight vector is
adjusted according to the used learning law.
Feature mapping is a process which converts the patterns of arbitrary
dimensionality into a response of one or two dimensions array of neurons. The
network performing such a mapping is called feature map. The reason for reducing
the higher dimensionality, the ability to preserve the neighbor topology.

Training Algorithm
Step 0: Initialize the weights with Random values and the learning rate
Step 1: Perform Steps 2-8 when stopping condition is false.
Step 2: Perform Steps 3-5 for each input vector x.
Step 3: Compute the square of the Euclidean distance, i.e., for each j = i to m,

Step 4: Find the winning unit index J, so that D(J) is minimum.

Step 5: For all units j within a specific neighborhood of J and for all i, calculate the new
weights:

step 6: Update the learning rare a using the formula (t is timestamp)

Step 7: Reduce radius of topological neighborhood at specified time intervals.

Step 8: Test for stopping condition of the network.

Learning Vector Quantization

In 1980, Finnish Professor Kohonen discovered that some areas of the brain develop
structures with different areas, each of them with a high sensitive for a specific input pattern.
It is based on competition among neural units based on a principle called winner-takes-all.
Learning Vector Quantization (LVQ) is a prototype-based supervised classification
algorithm. A prototype is an early sample, model, or release of a product built to test a
concept or process. One or more prototypes are used to represent each class in the dataset.
New (unknown) data points are then assigned the class of the prototype that is nearest to
them. In order for "nearest" to make sense, a distance measure has to be defined. There is no
limitation on how many prototypes can be used per class, the only requirement being that
there is at least one prototype for each class. LVQ is a special case of an artificial neural
network and it applies a winner-take-all Hebbian learning-based approach. With a small
difference, it is similar to Self-Organizing Maps (SOM) algorithm. SOM and LVQ were
invented by Teuvo Kohonen.
LVQ system is represented by prototypes W=(W1....,Wn). In winner-take-all training
algorithms, the winner is moved closer if it correctly classifies the data point or moved away
if it classifies the data point incorrectly. An advantage of LVQ is that it creates prototypes
that are easy to interpret for experts in the respective application domain

Training Algorithm
Step 0: Initialize the reference vectors. This can be done using the following steps. From the
given set of training vectors, take the first "m" (number of clusters) training vectors and use
them as weight vectors, the remaining vectors can be used for training.Assign the initial
weights and classifications randomly. K-means clustering method.Set initial learning rate α
Step l: Perform Steps 2-6 if the stopping condition is false.
Step 2: Perform Steps 3-4 for each training input vector x
Step 3: Calculate the Euclidean distance; for i = 1 to n, j = 1 to m,

Find the winning unit index J, when D(J) is minimum

Step 4: Update the weights on the winning unit, Wj using the following conditions.

Step 5: Reduce the learning rate α

Step 6: Test for the stopping condition of the training process. (The stopping conditions may
be fixed number of epochs or if learning rare has reduced to a negligible value.)

Counterpropagation network
Counterpropagation network (CPN) were proposed by Hecht Nielsen in 1987.They are
multilayer network based on the combinations of the input, output, and clustering layers. The
application of counterpropagation net are data compression, function approximation and
pattern association. The ccounterpropagation network is basically constructed from an instar-
outstar model. This model is three layer neural network that performs input-output data
mapping, producing an output vector y in response to input vector x, on the basis of
competitive learning. The three layer in an instar-outstar model are the input layer, the
hidden(competitive) layer and the output layer.
There are two stages involved in the training process of a counterpropagation net. The input
vector are clustered in the first stage. In the second stage of training, the weights from the
cluster layer units to the output units are tuned to obtain the desired response.
There are two types of counterpropagation network:

1. Full counterpropagation network

2. Forward-only counterpropagation network

Full counterpropagation network

Full CPN efficiently represents a large number of vector pair x:y by adaptively constructing a
look-up-table. The full CPN works best if the inverse function exists and is continuous. The
vector x and y propagate through the network in a counterflow manner to yield output vector
x* and y*.

Architecture of Full Counterpropagation Network

The four major components of the instar-outstar model are the input layer, the instar, the
competitive layer and the outstar. For each node in the input layer there is an input value xi.
All the instar are grouped into a layer called the competitive layer. Each of the instar
responds maximally to a group of input vectors in a different region of space. An outstar
model is found to have all the nodes in the output layer and a single node in the competitive
layer. The outstar looks like the fan-out of a node.

Training Algorithm for Full Counterpropagation Network:

Step 0: Set the initial weights and the initial learning rare.
Step 1: Perform Steps 2-7 if stopping condition is false for phase-I training.
Step 2: For each of the training input vector pair x: y presented, perform Steps 3-5.
Step 3: Make the X-input layer activations to vector X. Make the Y-inpur layer activations to
vector Y.
Step 4: Find the winning cluster unit. If dot product method is used, find rhe
cluster unit Zj with target net input: for j = 1 to p.

If Euclidean distance method is used, find the cluster unit Zj whose squared
distance from input vectors is the smallest
If there occurs a tie in case of selection of winner unit, the unit with the smallest
index is the winner. Take the winner unit index as J.

Step 5: Update the weights over the calculated winner unit Z j

Step 6: Reduce the learning rates α and β

Step 7: Test stopping condition for phase-I training.

Step 8: Perform Steps 9-15 when stopping condition is false for phase-II training.
Step 9: Perform Steps 10-13 for each training input pair x:y. Here α and β are small constant
values.
Step 10: Make the X-input layer activations to vector x. Make the Y-input layer activations
to vector y.
Step 11: Find the winning cluster unit (use formulas from Step 4). Take the winner unit index
as J.
Step 12: Update the weights entering into unit ZJ

Step 15: Test stopping condition for phase-II training.

Forward-only Counterpropagation network:
A simplified version of full CPN is the forward-only CPN. Forward-only CPN uses only the
x vector to form the cluster on the Kohonen units during phase I training. In case of forward-
only CPN, first input vectors are presented to the input units. First, the weights between the
input layer and cluster layer are trained. Then the weights between the cluster layer and
output layer are trained. This is a specific competitive network, with target known.

Architecture of forward-only CPN

It consists of three layers: input layer, cluster layer and output layer. Its architecture
resembles the back-propagation network, but in CPN there exists interconnections between
the units in the cluster layer.

Training Algorithm for Forward-only Counterpropagation network:

Step 0: Initial the weights and learning rare.
Step 1: Perform Steps 2-7 if stopping condition is false for phase-I training.
Step 2: Perform Steps 3-5 for each of uaining input X
Step 3: Set the X-input layer activations to vector X.
Step 4: Compute the winning cluster unit (J). If dot product method is used, find the cluster
unit zj with the largest net input.

Step 8: Perform Steps 9-15 when stopping condition is false for phase-II training.
Step 9: Perform Steps 10-13 for each training input Pair x:y..
Step 10: Set X-input layer activations to vector X. Sec Y-outpur layer activations to vector
Y.
Step 11: Find the winning cluster unit (use formulas from Step 4). Take the winner unit index
as J.
Step 12: Update the weights entering into unit ZJ,
Adaptive Resonance Theory
The Adaptive Resonance Theory (ART) was incorporated as a hypothesis for human cognitive data
handling. The hypothesis has prompted neural models for pattern recognition and unsupervised
learning. ART system has been utilized to clarify different types of cognitive and brain data.

The Adaptive Resonance Theory addresses the stability-plasticity(stability can be defined as the
nature of memorizing the learning and plasticity refers to the fact that they are flexible to gain new
information) dilemma of a system that asks how learning can proceed in response to huge input
patterns and simultaneously not to lose the stability for irrelevant patterns. Other than that, the
stability-elasticity dilemma is concerned about how a system can adapt new data while keeping what
was learned before. For such a task, a feedback mechanism is included among the ART neural
network layers. In this neural network, the data in the form of processing elements output reflects
back and ahead among layers. If an appropriate pattern is build-up, the resonance is reached, then
adaption can occur during this period.

It can be defined as the formal analysis of how to overcome the learning instability accomplished by a
competitive learning model, let to the presentation of an expended hypothesis, called adaptive
resonance theory (ART). This formal investigation indicated that a specific type of top-down learned
feedback and matching mechanism could significantly overcome the instability issue. It was
understood that top-down attentional mechanisms, which had prior been found through an
investigation of connections among cognitive and reinforcement mechanisms, had similar
characteristics as these code-stabilizing mechanisms. In other words, once it was perceived how to
solve the instability issue formally, it also turned out to be certain that one did not need to develop any
quantitatively new mechanism to do so. One only needed to make sure to incorporate previously
discovered attentional mechanisms. These additional mechanisms empower code learning to self-
stabilize in response to an essentially arbitrary input system. Grossberg presented the basic principles
of the adaptive resonance theory. A category of ART called ART1 has been described as an
arrangement of ordinary differential equations by carpenter and Grossberg. These theorems can
predict both the order of search as the function of the learning history of the system and the input
patterns.

ART1 is an unsupervised learning model primarily designed for recognizing binary patterns. It
comprises an attentional subsystem, an orienting subsystem, a vigilance parameter, and a reset
module, as given in the figure given below. The vigilance parameter has a huge effect on the system.
High vigilance produces higher detailed memories. The ART1 attentional comprises of two
competitive networks, comparison field layer L1 and the recognition field layer L2, two control gains,
Gain1 and Gain2, and two short-term memory (STM) stages S1 and S2. Long term memory (LTM)
follows somewhere in the range of S1 and S2 multiply the signal in these pathways.
 Gains control empowers L1 and L2 to recognize the current stages of the running cycle. STM
reset wave prevents active L2 cells when mismatches between bottom-up and top-down signals
happen at L1. The comparison layer gets the binary external input passing it to the recognition layer
liable for coordinating it to a classification category. This outcome is given back to the comparison
layer to find out when the category coordinates the input vector. If there is a match, then a new input
vector is read, and the cycle begins once again. If there is a mismatch, then the orienting system is in
charge of preventing the previous category from getting a new category match in the recognition
layer. The given two gains control the activity of the recognition and the comparison layer,
respectively. The reset wave specifically and enduringly prevents active L2 cell until the current is
stopped. The offset of the input pattern ends its processing L1 and triggers the offset of Gain2. Gain2
offset causes consistent decay of STM at L2 and thereby prepares L2 to encode the next input pattern
without bais.

ART1 Implementation process:

ART1 is a self-organizing neural network having input and output neurons mutually couple
using bottom-up and top-down adaptive weights that perform recognition. To start our
methodology, the system is first trained as per the adaptive resonance theory by inputting
reference pattern data under the type of 5*5 matrix into the neurons for clustering within the
output neurons. Next, the maximum number of nodes in L2 is defined following by the
vigilance parameter. The inputted pattern enrolled itself as short term memory activity over a
field of nodes L1. Combining and separating pathways from L1 to coding field L2, each
weighted by an adaptive long-term memory track, transform into a net signal vector T.
Internal competitive dynamics at L2 further transform T, creating a compressed code or
content addressable memory. With strong competition, activation is concentrated at the L2
node that gets the maximal L1 → L2 signal. The primary objective of this work is divided
into four phases as follows Comparision, recognition, search, and learning.

Advantage of adaptive learning theory(ART):

It can be coordinated and utilized with different techniques to give more precise outcomes.

It doesn't ensure stability in forming clusters.

It can be used in different fields such as face recognition, embedded system, and robotics,
target recognition, medical diagnosis, signature verification, etc.

t shows stability and is not disturbed by a wide range of inputs provided to inputs.

It has got benefits over competitive learning. The competitive learning cant include new
clusters when considered necessary.

Application of ART:
ART stands for Adaptive Resonance Theory. ART neural networks used for fast, stable
learning and prediction have been applied in different areas. The application incorporates
target recognition, face recognition, medical diagnosis, signature verification, mobile control
robot.

Target recognition:

Fuzzy ARTMAP neural network can be used for automatic classification of targets depend on
their radar range profiles. Tests on synthetic data show the fuzzy ARTMAP can result in
substantial savings in memory requirements when related to k nearest neighbor(kNN)
classifiers. The utilization of multiwavelength profiles mainly improves the performance of
both kinds of classifiers.

Medical diagnosis:

Medical databases present huge numbers of challenges found in general information

management settings where speed, use, efficiency, and accuracy are the prime concerns. A
direct objective of improved computer-assisted medicine is to help to deliver intensive care in
situations that may be less than ideal. Working with these issues has stimulated several ART
architecture developments, including ARTMAP-IC.

Signature verification:

Automatic signature verification is a well known and active area of research with various
applications such as bank check confirmation, ATM access, etc. the training of the network is
finished using ART1 that uses global features as input vector and the verification and
recognition phase uses a two-step process. In the initial step, the input vector is coordinated
with the stored reference vector, which was used as a training set, and in the second step,
cluster formation takes place.

Mobile control robot:

Nowadays, we perceive a wide range of robotic devices. It is still a field of research in their program
part, called artificial intelligence. The human brain is an interesting subject as a model for such an
intelligent system. Inspired by the structure of the human brain, an artificial neural emerges. Similar
to the brain, the artificial neural network contains numerous simple computational units, neurons that
are interconnected mutually to allow the transfer of the signal from the neurons to neurons. Artificial
neural networks are used to solve different issues with good outcomes compared to other decision
algorithms.