!r..

~
"~--
.: (

'I
Principles of S.oft Computing

(2nd Edition)

Dr. S. N. Sivanandam
Formerly Professor and Head
Department of Electrical and Electronics Engineering and
Department of Computer Science and Engineering,
PSG College of Technology,
Coimbatore

Dr. S. N. Deepa
Assistant Professor
Department of Electrical and Electronics Engineering,
Anna University of Technology, Coimbatore
Coimbatore

WILEY
 r,.
~I

About the Authors

Dr. S. N. Sivanandam completed his BE (Electrical and Electronics Engineering) in 1964 from Government
College of Technology, Coimbatore, and MSc (Engineering) in Power System in 1966 from PSG College of
Technology, Cc;:.imbarore. He acquired PhD in Control Systems in 1982 from Madras Universicy. He received
Best Teacher Award in the year 2001 and Dbak.shina Murthy Award for Teaching Excellence from PSG
College ofTechnology. He received The CITATION for best reaching and technical contribution in the Year
2002, Government College ofTechnology, Coimbatore. He has a toral reaching experience (UG and PG) of41
years. The toral number of undergraduate and postgraduate projects guided by him for both Computer Science
and Engineering and Electrical and Elecrionics Engineering is around 600. He has worked as Professor and
Head Computer Science and Engineering Department, PSG College of Technology, Coimbatore. He has
been identified as an outstanding person in the field of Compurer Science and Engineering in MARQUlS
'Who's Who', October 2003 issue, USA. He has also been identified as an outstanding person in the field of
Computational Science and Engineering in 'Who's Who', December 2005 issue, Saxe~Coburg Publications,
UK He has been placed as a VIP member in the continental WHO's WHO Registry ofNatiori;u Business
Leaders, Inc., NY, August 24, 2006.
A widely published author, he has guided and coguided 30 PhD research works and at present 9 PhD
research scholars are working under him. The rotal number of technical publications in International/National
Journals/Conferences is around 700. He has also received Certificate of Merit 2005-2006 for his paper from
The Institution ofEngineers (India). He has chaired 7 International Conferences and 30 National Conferences.
He is a member of various professional bodies like IE (India), ISTE, CSI, ACS and SSI. He is a technical
advisor for various reputed industries and engineering institutions. His research areas include Modeling and
Simulation, Neural Networks, Fuzzy Systems and Genetic Algorithm, Pattern Recognition, Multidimensional
system analysis, Linear and Nonlinear control system, Signal and Image prSJ_~-~sing, Control System, Power
system, Numerical methods, Parallel Computing, Data Mining and Database Security.
Dr. S. N. Deepa has completed her BE Degree from Government College of Technology, Coimbarore,
1999, ME Degree from PSG College of Technology, Coimbatore, 2004, and Ph.D. degree in Electrical
Engineering from Anna University, Chennai, in the year 2008. She is currently Assistant Professor, Dept. of
Electrical and Electronics Engineering, Anna University ofTechnology, Coimbatore. She was a gold medalist
in her BE. Degree Program. She has received G.D. Memorial Award in the year 1997 and Best Outgoing
Student Award from PSG College of Technology, 2004. Her ME Thesis won National Award from the Indian
Society of Technical Education and L&T, 2004. She has published 7 books and 32 papers in International
and National Journals. Her research areas include Neural Network, Fuzzy LOgic, Genetic Algorithm, Linear
and Nonlinear Control Systems, Digital Control, Adaptive and Optimal Control.
 Contents

Preface v
About the Authors ............... . viii
1. Introduction
Learning Objectives
1.1 Neural Networks
1.1.1 Artificial Neural Network: Definicion ..................................................... 2
1.1.2 Advantages ofNeural Networks ........................................................... 2
1.2 Application Scope of Neural Networks ............................................................. 3
1.3 Fuzzy Logic ........................... , ............................................................... 5
1.4 Genetic Algorithm ................................................................................... 6
1.5 Hybrid Systems ...................................................................................... 6
1.5.1 Neuro Fuzzy Hybrid Systems ................._............................................. 6
1.5.2 Neuro Genetic Hybrid Systems ............................................................ 7
1.5.3 Fuzzy Genetic Hybrid Systems .................. 7
1.6 Soft Computing 8
1.7 Summary'~ .................... 9

2/\Art;ficial Neural Network: An Introduction ...... II

J \.-- Learning Objectives ......................... II
2.1 Fundamental Concept ........... . II
2.1.1 Artificial Neural Network ...................................................... . II
2.1.2 Biological Neural Network 12
2.1.3 Brain vs. Computer - Comparison Between Biological Neuron and Artificial
Neuron (Brain vs. Computer) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2 Evolution of Neural Networks ..................................................................... 16
2.3 Basic Modds of Artificial Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.1 Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.2 Learning ..................... . ......... 20
2.3.2.1 Supervised Learning .............................................. . ...... 20
2.3.2.2 Unsupervised Learning ....................................................... 21
2.3.2.3 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .... 21
2.3.3 Activation Functions ...................................................................... 22
2.4 Important Terminologies of A.NNs ............................................................... 24
U! ~~--- ...................................... M
2.4.2 Bias .. .. .. .. .. . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. .. .. . .. ..................... 24
2.4.3 Threshold ................................................................................. 26
2.4.4 Learning Rare ............................................................................. 27
2.4.5 Momentum Factor ........................................................................ 27
2.4.6 Vigilance Parameter .................................................................... 27
2.4.7 Notations .................................................................................. 27
2.5 McCulloch-Pitrs Neuron...... ------------------------------ 27
 -~
::_>;,

X Contents

2.5.1 Theory .................................... . . ..... 27

2.5.2 Architecture .... . 28
2.6 Linear Separability .. . . ............. 29
2. 7 Hebb Network
2.7.1 Theory
:::: ;;
2.7.2 Flowchan of Training Algoriclun . . 31
2.7.3 Training Algorithm ............. . 31
2.8 Swnmary .............................. 33
2.9 Solved Problems
2.10 Review Questions ............ ::::::::::::.::.:::::::::::::::::::::::::H
2.11 Exercise Problems
2.12 Projects .. .. 47
3. Supervised Learning Network ........................................................................ 49

.r-;_1 ~:~~(~~j~~~~~.:::: :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :~

3.2 rerceptron Nenvorks ................................................................................ 49
3.2.1 Theory ..................................................................................... 49
3.2.2 Perceptron Learning Rule ................................................................. 5 I
3.2.3 Aichitecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.2.4 Flowchart for Training Process ........................................................ 52
3.2.5 Perceptron Training Algorithm for Single Output Classes ... , ......................... 52
3.2.6 Perceprron Training Algoriilim for Multiple Output Classes ........................... 54
3.2.7 Perceptron Necwork Testing Algorithm .................... , .......................... 55
3.3 Adaptive Linear Neuron (Adaline) .............................................................. 57
3.3.1 Theory ................................................................ : ............... 57
3.3.2 Deha Rule for Single Output Unit ..................... , ............................. 57
3.3.3 Architecture .......................................................................... 57
3.3.4 Flowchart for Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ............ , 57
3.3.5 Training Algorithm ............................... , .................................. 58
3.3.6 Testing Algorithm ......... ................... , ....................................... 60
3.4 Multiple Adaptive Linear Neurons .. .. . . . . . . .. .. . ..................................... 60
3.4.1 Theory . . . . . . . . . . . . . ..... ..... ... . ....... .. ...... .. . . . . . . .. .... . 60
3.4.2 Architecture . . . . . . . . . . . . . . . . . . . ..................................................... 60
3.4.3 Flowchart ofTraining Process . . . .. .. .. .. .. . . . . . . . .. .. .. .. . .. .. .. .. .. . .. ........... 60
3.4.4 Training Algorithm . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ............. 61
3.5 Back-Propagation Network .................................................................... 64
3.5.1 Theory ................................................................................ 64
3.5.2 Architecture ........................................................................... 65
3.5.3 Flowchart for Training Process ....................................................... 66
3.5.4 Training Algorithm ................................. :.................................... 66
3.5.5 Learning Factors ofB;ack-Propa"gation Nclwork ...................................... 70
3.5.5.1 Initial Weights ............................................................... 70
3.5.5.2 Leamfng Ratea ........................................................... 71
3.5.5.3 Momentum Factor . . . . . . . . . . . . . . . . ......................................... 71
3~.4 Generalization .............................................................. 72
 Contents xi

~ ...... 72
3.5.5.5 Number of Training Data ........... .
3.5.5.6 Number of Hidden Layer Nodes . . ...... 72
3.5.6 TesringAlgoriilim of Back-Propagation Network ...... . 72
3.6 Radial Basis Function Network .......... i-.. / .... ;. 73
3.6.1 Theory 73
3.6.2 Architecture ..................... '.. . ................................................... 74
3.6.3 Flowchart for Training Process ... 74
3.6.4 Training Algorithm .............. . ' 74
3.7 Ttme Delay Neural Necwork ........ . 76
3.8 Funcnonal Link Networks ......... ~- ........... .. ......................... 77
3.9 Tree Neural Networks .................................................... , , ....................... 78
3.10 Wavelet Neural Networks ......................................................................... 79
3.11 Summary .................................................. ...... BO
3.12 Solved Problems ...................................... .. -~ 81
3.13 Review Questions 94
3.14 Exercise P;oblems 95
3.15 Projecrs ...................... .. 96

AAssoci=~i~~je~i~;~~~~.::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::.::::::::: ~;
4.1 Introduction ........................................................................................ 97
4.2 Training Algorithms for Pattern Association .... , ................ , . , . . . . . . . . . . . . . . . . . . . . . . . . .... 98
4.2.1 Hebb Rule .............................................................................. 98
4.2.2 Outer Producrs Rule . . . . . . . . . . .. . . .............. 100
4.3 Autoassociative Memory Network . .................................................... 101
4.3.1 Theory .............. 101
4.3.2 Architecture .101
4.3.3 FlowchartforTrainingProcess ...................................................... 101
4.3.4 Training Algorithm ........................ , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......... 103
4.3.5 TesringAlgorithm .................................................................. 103
4.4 Heteroassociative Memory Network .. .. .. . . . .. . .. . .. . .. .. .. . . . . .. . . . . . . .. . . . . . .. . .. .. .. . . I 04
4.4.1 Theory ... .... .. ... .. ........ ......... .. ... . . . . . . . ..... ............ . ....... 104
4.4.2 Architecture ........ . ......................................................... 104
4.4.3 TestingAlgorithm ........... .................. ................. . ................ 104
4.5 Bidirectional Associative Memory (BAM) .. .. .. . .. .. . .. .. .. .. .. . .. .. .. .. .. .. .. . .. . .. . 105
4.5.1 Theory ...................... ............ ........... .... ........................ 105
4.5.2 Architecture ... .......... . . .......... .. ....... ................. ............... 105
4.5.3 Discrete Bidirectional Associative Memory ......................................... 105
4.5.3.1 Determination ofWeights ............................................... 106
4.5.3.2 Activation Functions for BAl:vl .. .. .. .. . .. .. . .. .......................... 107
4.5.3.3 Testing Algorithm for Discrete BAl:vl .. . .. . .. .. .. .. . .. .. .. .. .. .. . .. ...... 107
4.5.4 Continuous BAM ........ ...... .. . .. ................... ........ .. ... ............... 108
4.5.5 Analysis of Hamming Di.~tance, Energy Function and Storage Capacity ............ 109
-\.6 Hopfield Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.6.1 Discrete Hopfield Nerwork ......... ................ . ............................. 110
4. 6.1.1 Architecture of Discrete Hop field Net .. .. . .. . .. . . .. . .. .. .. .. . .. .. . .. . 111
 ~
"fi
.
Contents
xii

4.6.1.2 Training Algorithm of Discrete Hopfield Net .............................. 111

4.6.1.3 . Testing Algorithm of Discrete Hopfield Net ............................... 112
4.6.1.4 Analysis of Energy Function and Storage Capacity on Discrete
Hopfield Ne< ................................................................. 113
4.6.2 Continuo~ Hopfield Nerwork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.6.2.1 Hardware Model of Continuous Hopfield Network ....................... 114
4.6.2.2 Analysis of Energy Function of Continuous Hopfield Network .......... 116
4.7 Iterative Auroassociacive Memory Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
4.7.1 Linear Autoassociacive Memory (LAM) ................................................ 118
4.7.2 Braininilie-Box Network .............................................................. 118
4.7.2.1 Training Algorithm for Brainin-ilieBox Model ........................... 119
4.7.3 Autoassociator with Threshold Unit .................................................... 119
4.7.3.1 TesringAlgoridtm ............................................................ 120
4.8 Temporal Associative Memory Network ........................................................ 120
4.9 Summary .......................................................................................... 121
4.10 Solved Problems ................................................................................... 121
4.11 Review Questions ................................................................................. 143
4.12 Exercise Problems ................................................................................. 144
4.13 Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5. Unsupervised Learning Networks .................................................................. 147
Learning Objectives ............................................................................... 147
5.1 Introduction ....................................................................................... 147
5.2 FIXed Weight Competitive Nets .................................................................. 148
5.2.1 Maxner ................................................................................... 148
5.2.1.1 Architecture ofMaxnet ...................................................... 148
5.2.1.2 Testing/Application Algorithm of Max net ....................... , .......... 149
5.2.2 Mexican HarNer ........................... : ............................................ 150
5.2.2.1 Architecture ................................................................... 150
5.2.2.2 Flowchart ................................................................... 150
5.2.2.3 Algoridtm ..................................................................... 152
5.2.3 Hamming Network .._.................................................................. 153
5.2.3.1 Architecture ................................................................... 154

J 5.2.3.2 Testing Algorithm ............................................................ 154

Kohonen Self-Organizing Feature Maps ....................................................... 155

. . ! ;:;:; !:':.:c~~~.................. ....... ........................ ....... .. .......... :;~

5.3.3 Flowchart ................................................................................. 158
5.3.4 TrainingAlgoriilim ...................................................................... 160
/ \ 5.3.5 Kohonen Self.Qrganizing Momr Map ............................................... 160
J5.4; Learning Vector Quantization ................................................................. 161
5.4.1 Theo<y .................................................................................. 161
5.4.2 Architecture ............................................................................. 161
5.4.3 Flowchart ................................................................................ 162
5.4.4 TrainingAlgoriilim ................................................................... 162
5.4.5 Variants .................................... 164
~
i'
Contents xill

5.4.5.1 LVQ2 ......................................................................... 164

5.4.5.2 LVQ2.1 ...................................................................... 165
5.4.5.3 LVQ3 ................ : ........................................................ 165
5.5 Counterpropagation Networks ............... :.. ................................................. 165
5.5.1 Theo<y ................................ :................................................... 165
5.5.2 Full Counterpropagation Net ........................................................... 166
5.5.2.1 Architecture ................................................................... 167
5.5.2.2 Flowchart ..................................................................... 169
5.5.2.3 TtainingAlgoritl!m ........................................................... 172
5.5.2.4 Testing (Application) Algoridtm ............................................ 173
5.5.3 Forward-Only Counterpropagation Net ............................................... 174
5.5.3.1 Architecture ................................................................... 174
5.5.3.2 Flowchart ..................................................................... 175
5.5.3.3 TtainingAlgoridtm ........................................................... 175
5.5.3.4 TestingAlgoridtm ............................................................ 178
5.6 Adaptive Resonance Theory Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
5.6.1 Theo<y ................................................................................... 179
5.6.1.1 FundarnemalArchitecrure ................................................... 179
5.6.1.2 Fundamental Operating Principle ..... ...... 180
5.6.1.3 Fundarnema!Algoridtm ......................... . 181
5.6.2 Adaptive Resonance Theory 1 ....................... .. . 181
5.6.2.1 Architecture ...............,-. ........................... . .1U
5.6.2.2 Flowchart of Training Pro~s ..... 1M
5.6.2.3 Training Algoriilim . ... 1M
5.6.3 Adaptive Resonance Theory 2 . .............. 1~
5.6.3.1 Architecture ................................................................. 188
5.6.3.2 Algori1hm ................................................................... 188
5.6.3.3 Flowchart ..................................................................... 192
5.6.3.4 Training Algorithm ........................................................... 195
5.6.3.5 Sample Values of Parameter ................................................. 196
5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
5.8 Solved Problems ................................................................................... 197
5.9 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ......... 226
5.10 Exercise Problems ................................................................................. 227
5.11 Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
6. Special Networks ...................................................................................... 231
Learning Objectives ............................................................................... 231
6.1 Introduction ....................................................................................... 231
6.2 Simulated Annealing Network ................................................................... 231
6.3 Boltzmann Machine .............................................................................. 233
6.3.1 Architecture .............................................................................. 234
6.3.2 Algoridtm ................................................................................ 234
6.3.2.1 Setting dte Weigh" of dte Network ......................................... 234
6.3.2.2 Testing Algorithm ........................................................... 235
6.4 Gaussian Machine . . . . . . . . . . . . .. . . . . . . . ............................ 236
 Contents
xiv

6.5 Cauchy Machine .................................................................................. 237

6.6 Probabilistic NeUral Net .......................................................................... 237
6.7 Cascade Correlation Nerwork .................................................................... 238
6.8 Cogniuon Nerwork ............................................................................... 240
6.9 Neocogniuon Network ........................................................................... 241
6.10 Cellular Neural Nerwork ......................................................................... 242
6.11 Logicon Projection Ne[Work Model ............................................................. 243
6.12 Spatio~Temporal Connectionist Neural Network ............................................... 243
6.13 Optical Neural Networks ......................................................................... 245
6.13.1 Electro~Optical Multipliers ............................................................. 245
6.13.2 Holographic Correlators ................................................................ 246
6.14 Neuroprocessor Chips .............. , ............................................................. 247
6.15 Summary .......................................................................................... 249
6.16 Review Questions ................................................................................. 249
7. Iurroduction to Fuzzy Logic, Classical Sets and Fuzzy Sets ................................... 251
Learning Objectives ................................................................................ 251
7.1 Introduction to Fuzzy Logic ...................................................................... 251
7.2 Classical Sets (Crisp Sers) .................................... , .................................. 255
7.2.1 Operations on Classical Sets .................................... , ....................... 256
7.2.l.l Union ......................................................................... 257
7.2.1.2 Intersection ................................................................. 257
7.2.1.3 Complement ............................................................... 257
7.2.1.4 Difference (Subtraction) ................................................... 258
7.2.2 Properties of Classical Sets ........ .. ............................................. 258
7 .2.3 Function Mapping of Classical Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..... 259
7.3 Fuzzy Sets ..................................................................................... 260
7.3.1 Fuzzy Set Operations .................................................................. 261
7.3.1.1 Union ..................... .................. . ........................... 261
7.3.1.2 Intersection ................................................................ 261
7.3.1.3 Complement ............................................................. 262
7.3.1.4 More operations on Fuzzy Sets .......... , ................................ 262
7.3.2 Properdes of Fuzzy Sets ............................................................. 263
7.4 Summary........ ....... ........... . ..................................................... 264
7.5 Solved Problems ................................................................................. 264
7.6 Review Quesrions ............... ................................. ... ......... . ................. 270
7.7 Exercise Problems ............................................................................. 271
8. Classical Relations and Fuzzy Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Learning Objectives .......................................................................... 273
8.1 Inuoduction ................................................................................... 273
8.2 Cartesian Product of Relation .. , . , ............................. , ... , , ..... , ...................... 273
8.3 Classical Relation ......................_.................................................. ~------- 274
8.3.1 Cardinality of Classical Relation ....... ...................... . ..................... 276
8.3.2 Operations on Classical Relations ..................................... / .............. 276
8.3.3 Properties-of Crisp Relations ......................................................... 277
8.3.4 Composition of Classical Relations .................................................... 277
 XV
Contents

8.4 fll2Lf Relations

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
8.4.1
Cardinality of Fuzzy Relations ............ : ............................................. 281
8.4.2 Operations on Fuzzy Relations _. ........................................................ 281
8.4.3 Properties of Fuzzy Relations ............................................................ 282
.8.4.4 Fuzzy Composition ................ . ................................................... 282
8.5 Tolerance and Equivalence Relations . ~ .......................................................... 283
8.5.1 Cl:w;ical Equivalence Relation .......................................................... 284
8.5.2 Classical Tolerance Relation ............................................................. 285
8.5.3 Furiy Equivalence Rdacion ............................................................. 285
8.5.4 FuzZy Tolerance Relation; ............................................................... 286
8.6 Noninteractive Fuzzy Sets .................................................................. ." ..... 286
8.7 Summary .......................................................................................... 286
8.8 Solved Problems ................................................................................... 286
8.9 Review Questions ............................................. -- -- .. ---- ....... :.. 292
8.10 Exercise Problems ........................ -- .. ........................... 293

9. Membership Functions ............... .. 295

Learning Objectives ...... . 295
9.1 Introduction ................................ . 295
9.2 Features of the Membership Functions .............................................. .-........... 295
9.3 Fuzzificdtion ....................................................................................... 298
9.4 Methods of Membership Value Assignments .................................................... 298
9.4.1 Intuition ...................................................... -........................... 299
9.4.2 Inference ............................................................................... 299
9.4.3 Rank Ordering.. . . . .. .. ... . . ........................................................ 301
9.4.4 Angular Fuzzy Se~ .................................................................... 301
9.4.5 Neural Networks ....................................................................... 302
9.4.6 Genetic Algorithms ................................................................ 304
9.4.7 Induction Reasoning .................................................... 304
Summary ................................ . ,........................ 305
9.5
9.6 Solved Problems .. .. .. ............... . 305
Review Questions ........ 309
9.7
9.8 Exercise Problems .. ..... 309
10. Defuzrification ......................................................................................... 311
Learning Objectives ............................................................................ 311
10.1 Introduc-tion ..................................................................................... 311
10.2 Lambda-Curs for Fuzzy Sets (Alpha-Cuts) ..................................................... 311
10.3 Lambda~Cuts for Fuzzy Relations ................................................................ 313
10.4 Defuzzification Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. .. .. . 313
l 0.4.1 Max~Membership Principle . . . . . . . . . . . . . . ............................................. 315
10.4.2 Cemroid Me<hod ....................................................................... 315
10.4.3 WeightedAverageMe<hod .............................................................. 316
10.4.4 Mean-Max Membership ................................................................. 317
10.4.5 Center of Sums .......................................................................... 317
10.4.6 CenrerofLargestAr"ea ................................................................. 317
10.4. 7 First of Maxima (Last of Maxima) ................. 318
 ~

XVI Contents

10.5 Summary ............... . 320

10.6 Solved Problems 320
10.7 Review Questions ..................................................... 327
10.8 Exercise Problems .... :.......... 327
11. Fuzzy Arithmetic and Fuzzy Measures ............................................................ 329
Learning Objectives ............................................................................... 329
11.1 Introduction ....................................................................................... 329
11.2 Fuzzy Arithmetic .................................................................................. 329
11.2.1 Interval Analysis of Uncenain Values ................................................ , .. 329
11.2.2 Fuzzy Numbers .......................................................................... 332
11.2.3 Fuzzy Ordering .................... , ,.................................... ,............... 333
11.2.4 Fuzzy Vectors ............................................................................ 335
11.3 Extension Principle ............................................................................... 336
11.4 F=yMeasures ................................................................ : ................... 337
11.4.1 Belief and Plausibilicy Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
11.4.2 Ptebability Measures .................................................................... 340
11.4.3 Possibility and Necessity Measures ..................................................... 340
11.5 Measures of FU7.Ziness ............................................................................. 342
11.6 Fuzzy Integrals ......................................... , .......................................... 342
11.7 Summary .......................................................................................... 343
11.8 Solved Ptoblems ........................... , ....................................................... 343
11.9 Review Questions ............................................................................... 345
11.10ExerciseProblems ............................................................................... 346
12. Fuzzy Rule Base and Approximate Reasoning .. .. .. .. .. .. .. .. .. .. . .. ...................... 347
Learning Objectives ............................................................................. 347
12.1 Introduction ....................................................................................... 347
12.2 TruthValuesandTablesinFuzzyLogic ....................................................... 347
12.3 Fuzzy PropOsitions ................................................................................ 348
12.4 FormationofRules ............................................................................. 349
12.5 Decomposition of Rules (Compound Rules) ................................................. 350
12.6 AggregationofFuzzyRules ................................................................... 352
12.7 Fuzzy Reasoning (Approximate Reasoning) ..................................................... 352
12.7.1 Categorical Reasoning ................................................................... 353
12.7.2 Qualitative Reasoning ................................................................... 354
12.7.3 Syllogistic Reasoning .................................................................... 354
12.7.4 Dispositional Reasoning ............................................................... 354
12.8 F=y Inference Systems (FIS) ................................................................. 355
12.8.1 Construction and Working Principle ofFIS ................ , ..... , .................... 355
12.8.2 Methods ofFIS .......................................................................... 355
12.8.2.1 Mamdani F1S ................................................................. 356
12.8.2.2 Takagi-Sugeno Fuzzy Model (TS Method) ................................ 357
12.8.2.3 Comparison between Mamdani and Sugeno Method ..................... 358
12.9 Overview of Fuzzy Expert System .......... , ................................ , .......... , .. , ..... 359
12.10 Summary ................................................................................. , ........ 360
12.11 Review Questions .......................................... , ...................................... 360
~

Contents XVII

12.12 Exercise Problems ................................................ 361

13. Fuzzy Decision Making ...................... , ....................................................... 363
Learning Objectives ...................... .'..... : ".' ................. , ............................. 363
13.1 Introduction ....................................................................................... 363
13.2 Individual Decision Making ............ , .... :~ .................................................. 364
13.3 Mu1ciperson Decision Making ......................... , .. , ...................................... 364
13.4 Mu1tiobjective Decision Making ..... , ................................ , .......................... 365
13.5 Multiattribute Decision Making ................................................................. 366
13.6 Fuzzy Bayesian Decision Making .. ~ ..... , ....................................................... 368
13.7 Summary .......................................................................................... 371
13.8 Review Questions ................................................................................. 371
13.9 Exercise Problems .............. , ............................................................. 371
14. Fuzzy Logic Control Systems ................................................ , ...................... 373
Learning Objectives ............................................................................... 373
14.1 Introduction ....................................................................................... 373
14.2 Control System Design ........................................................................... 374
14:3 Architecture and Operation ofFLC System ..................................................... 375
14.4 FLCSystemMode~ .............................................................................. 377
14.5 Application of FLC Systems ...................................................................... 377
14.6 Summary .......................................................................................... 383
14.7 Review Questions ................................................................................. 383
14.8 Exercise Problems ........................ , .. , ..................... , ............................... 383
15. Genetic Algorithm .................................................................................... 385
Learning Objectives ...................................... , ........................................ 385
15.1 Inuoduction ....................................................................................... 385
15.1.1 What are Genetic Algorithms? .......................................................... 386
15.1.2 Why Generic Algorithms? ............................................................. 386
15.2 Biological Background ......... , .......... , ................... , ................................... 386
15.2. 1 The Cell .................................................................................. 386
15.2.2 Chromosomes .............................................. , ........................... 386
15.2.3 Genecics .................................................................................. 387
15.2.4 Reptoduction .......................................................................... 388
15.2.5 Natural Seleccion ...................................... 390
15.3 Traditional Optimization and Search Techniques . 390
15.3.1 Gradient~ Based Local Optimization Method .......... . ........ 390
15.3.2 Random Search ... 392
15.3.3 Stochastic Hill Climbing .. .. 392
15.3.4 Simulated Annealing ..... . .. ... 392
15.3.5 Symbolic Artificial Intelligence ... . . 393
15.4 Genetic Algorithm and Search Space ......... . 394
15.4.1 Search Space 394
15.4.2 Genetic Algorithms World ............................... .. 395
15.4.3 Evolution and Optimization ....... , ............. . 395
15.4.4 Evolution and Genetic Algorithms ......................... .. 396
xviii Contents

15.5 Generic Algorithm vs. Traditional Algorithms .................................................. 397

15.6 Basic Terminologies in Genetic Algorithm ...................................................... 398
15.6.1 Individuals ............................................................................... 398
15.6.2 Gone> ..................................................................................... 399
15.6.3 Fitn"" ...... .-............................................................................ 399
15.6.4 Populations ............................................................................... 400
15.7 Simplo GA ......................................................................................... 401
15.8 General Genetic Algorithm ...................................................................... 402
15.9 Operators in Generic Algorithm ............ ..................................................... 404
15.9.1 Encoding .............................................................................. 405
15.9.1.1 Binary Encoding ............................................................. 405
15.9.1.2 Ocral Encoding ............................................................... 405
15.9.1.3 Hc=dooimal Encoding ...................................................... 406
15.9.1.4 Permutation Encoding {Real Nwnber Coding) ............................ 406
15.9.1.5 Valuo Encoding ............................................................... 406
15.9.1.6 Tree Encoding ................................................................ 407
15.9.2 Selection .................................................................................. 407
15.9.2.1 Roulette Wheel Selection .................................................... 408
15.9.2.2 Random Selection ............................................................ '408
15.9.2.3 Rank Sdocrion ............................................................... 408
15.9.2.4 TournamenrSelecrion ....................................................... 409
15.9.2.5 Boltzmann Selection ................................................. : ...... 409
15.9.2.6 Stochastic Universal Sampling . . . . .. . .. .. . . .. .. .. .. .. .. . .. . .. . .. . . ...... 410
15.9.3 Crossover {Recombination) .................................................. _. ........ 410
15.9.3.1 Single~PoimCrossover ....................................................... 411
15.9.3.2 Two-Point Crossover ......................................................... 411
15.9.3.3 Multipoint Crossover {N-Point Crossover) ............................... 412
15.9.3.4 Uniform Crossover ........................................................ 412
15.9.3.5 Three-Parent Crossover ................................................. 412
15.9.3.6 CrossoverwithReducedSurrogate ..................................... 413
15.9.3.7 Shuffle Crossover ........................................................... 413
15.9.3.8 Precedence Preservative Crossover ......................................... 413
15.9.3.9 Ordered Crossover ...................................................... 413
15.9.3.10 Partially Matched Crossover ............................................... 414
15.9.3.11 Crossover Probability ...................................................... 415
15.9.4 Mutation ........................ .................................................... 415
15.9.4.1 Flipping ..................................................................... 415
15.9.4.2 Interchanging......... . .................................................. 415
15.9.4.3 Revorsing ..................................................................... 416
15.9.4.4 Mutation Probability ...................................................... 416
15.10 Stopping Condition for Generic Algorithm Flow .............................................. 416
15.10.1 Be>dndividlla! ........................................................................ 417
15.10.2 Worst individual ......................................................................... 417
15.10.3 Sum of Fitness ........................................................................... 417
15.10.4 Median Fitness ......................................................................... 417
15.11 Constraints in Genetic Algorithm ............................................................ 417

L
Contents
xix

15.12 Problem Solving Using Genetic Algorithm :............................................... , ..... 4-18

15.12.1 Maximizing a Function ................... ." . .-............................................ 418
15.13 The Schema Theorem .....................,....................................................... 422
15.13.1 The Optimal Allocation of Trials' ... _:... ; ............................................... 424
15.13.2 Implicit Parallelism ................. :: .': .. : ............................................... 425
15.14Ciassification of Generic Algorithm .... , .... :~ ...................................... f: 426
15.i4.1 Messy Generic Algorithms .............................................................. 426
15.14.2 Adaptive Genetic Algorithms -........................................................... 427
15.14.2.1 Adaptive Probabilities of Crossover and Mutation ......................... 427
15.14.2.2 De>ign of Adapciep, andpm ............................................... 428
15.14.2.3 Practical Considerations. and Choice ofValues for kt, ~. k3 and k4 ...... 429
15.14.3 Hybrid Genocic Algorirhms ............................................................. 430
15.14.4 Parallel Gonecic Algorithm .............................................................. 432
15.14.4.1 Global Paralldizacion ........................................................ 433
15.14.4.2 Classifir:ation of Parallel GAs ................................................ 434
15.14.4.3 Coarso-Grainod PGAs- The Island Model ........ ; ....................... 440
15.14.5 Independent Sampling Generic Algorithm (ISGA) .................................... 441
15.14.5.1 Comparison ofiSGAwith PGA ............................................ 442
15.14.5.2 Components ofiSGAs ....................................................... 442
15.14.6 Real~Coded Generic Algorithms ......................................................... 444
15.14.6.1 Crossover Operators for Real~Coded GAs .................................. 444
15.14.6.2Mutation Operators for Real~Coded GAs .................................. 445
15.15HollandClassifierSysrems ....................................................................... 445
15.15.1 The Production System ................................................................. 445
15.15.2 The Bucket Brigade Algorithm ......................................................... 446
15.15.3 Rule Generation ......................................................................... 448
15.16 Genetic Programming ............................. , .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........ 449
15.16.1 WorkingofGeneticProgramming ................................................... 450
15.16.2 Characteristics of Generic Programming ............................................... 453
15.16.2.1 Human~Competitive ........................................................ 453
15.16.2.2 High-Rerum ........................... , .................................. 453
15.16.2.3 Routine ...................................................................... 454
15.16.2.4 Machine Intelligence ........................ , .............................. 454
15.16.3 Data Representation ................................................................... 455
15.16.3.1 Crossing Programs ........................................................ 458
15.16.3.2 Mutating Programs ........................................................ 460
15.16.3.3 The Fitness Function ........................................................ 460
15.17 Advantages and Limitations of Genetic Algorithm ............................................. 461
15.18 Applications of Generic Algorithm ............................................................. 462
15.19Summa')' ....................................................................................... 463
15.20 Review Questions ............................................................................... 464
15.21 Exercise Problems .......... . ....................... 464
16. Hybrid Soft Computing Techniques ..................... . 465
Learning Objectives . . . . . . . . . . . . . . . . . . . ................................... . ........ 465
16.1 Introduction .......................... . 465
XX Contents

16.2 Neuro-Fuzzy Hybrid Systems .................................................................... 466

16.2.1 Comparison of Fuzzy Systems with Neural Networks ............. , .. , ............... 466
16.2.2 Characteristics ofNeuro~Fuzzy Hybrids .......................... ,......... , ... ,...... 467
16.2.3 Classifications ofNeuro,F=y Hybrid Systems ....................................... 468
16.2.3.1 Cooperative Neural Fuzzy 5}'5tems .......................................... 468
16.2.3.2 General Neuro-Fuzzy Hybrid Systems (General NFHS) .................. 468
16.2.4 Adaptive Neuro,Fuzzy Inference Sysrem (ANFIS) in MATLAB ..................... 470
16.2.4.1 FIS Structure and Parameter Adjustment ................................... 470
16.2.4.2 Constntints of ANFIS ........................................................ 471
16.2.4.3 The ANFIS Editor GUI ..................................................... 471
16.2.4.4 Data Formalities and the ANFIS Editor GUI .............................. 472
16.2.4.5 More on ANFIS Editor GUI ................................................ 472
16.3 Genetic Neuro~Hybrid Systems .............................................. , .... , ... , .. , ...... 476
16.3.1 Properties of Genetic Neuro-Hybrid Systems ................................... , ...... 476
16.3.2 Genetic Algorithm Based Back-Propagation Network (BPN) ........................ 476
16.3.2.1 Coding ........................................................................ 477
16.3.2.2 Weight Extraction ............................................................ 477
16.3.2.3 Fitness Function .............................................................. 478
16.3.2.4 R<production ofOffspting .................................................. 479
16.3.2.5 Convergence .................................................................. 479
16.3.3 Advantages ofNeuro-Genecic Hybrids ................................................. 479
16.4 Genetic Fuzzy Hybrid and Fuzzy Genetic Hybrid Systems .................................... 479
16.4.1 Genetic F=y Rule Based Systems (GFRBSs) ......................................... 480
16.4.1.1 GeneticTuningProcess ................................................... 481
16.4.1.2 Genetic Learning of Rule Bases ........................................... 482
16.4.1.3 Genetic Learning of Knowledge Base ........................... , ........... 483
16.4.2 Advantages of Generic Fuzzy Hybrids ................................................ 483
16.5 Simplified F=y ARTMAl' . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..... 483
16.5.1 Supervised ARTMAP System . .. .. . .. .. .. .. .. .. . .. .. . . .. .. .. . . . .. .. .. . . . .. . . . . . ....... 484
16.5.2 Comparison of ARTMAP with BPN ................................................... 484
16.6 Summary ........................................................................................ 485
16.7 Solved Problems using MATLAB ........... , . , , . ,............ , , , , , .............................. 485
16.8 Review Questions .. , , , ..................... , , , ............. , , ,.................................... 509
16.9 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...... 509
17. Applications of Soft Computing .................... . 511
Learning Objectives . , . , ........ , , , , . . . ..... , ...... , , , , .... 511
17.1 Introduction ............................................. . . ... 511
17.2 A Fusion Approach of Multispectral Images with SAR (SynthericAperrure Radar)
Image for Flood Area Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
17.2.1 Image Fusion .......................................................................... 513
17.2.2 Neural Network Classification .......................................................... 513
17.2.3 Methodology and Results ............................................................... 514
17.2.3.1 Method ....................................................................... 514
17.2.3.2 Results ............................................ ........ ..... . ....... 514
17.3 Optimization ofTrav~ling Salesman Problem using Genetic Algorithm Approach ........... 515
 xxi
Contents

17.3.1 Genetic Algorithms ...............:: ...................................................... 516

17.3.2 Schemata' 517
17.3.3 Problem Representation ................................................................. 517
17.3.4 Reproductive Algorithms---~-... :,. ..._................................................. 517
17.3.5 MutacioriMethods ............... ; .. :_..'................................................. 518
17.3.6 Results ' 518
17.4 Genetic Algorithm-Based Internet SearCh Technique ........................................... 519
17.4.1 Genetic Algorithms and lmemet ....................................................... 521
17.4.2 First Issue: Representation ofGenomes ................................................ 521
17.4.2.1 String Represen[acion ........................................................ 521
17.4.2.2 AirayofSuingRepresentation .............................................. 522
17.4.2.3 Numerical Representation ................................................... 522
17.4.3 Second Issue: Definicion of the Crossover Operator ................................... 523
17.4.3.1 Classical Crossover ........................................................... 523
17.4.3.2 Parent Crossover .............................................................. 523
17.4.3.3 Link Crossover ................................................................ 523
17.4.3.4 Overlapping Links ........................................................... 523
17.4.3.5 Link Pre-Evaluation .......................................................... 524
17.4.4 Third Issue: Selection of the Degree of Crossover ..................................... 524
17.4.4.1 Limired.Crossover ............................................................ 524
17.4.4.2 Unlimited Crossover ......................................................... 524
17.4.5 Fourth Issue: Definicion of the Mutation Operator ................................... 525
17.4.5.1 Generational Mutation ...................................................... 526
17.4.5.2 Selective Mutation ........................................................... 526
17.4.6 Fifth Issue: Definition of the Fimess Function ......................................... 527
17.4.6.1 Simple Keyword Evaluation ................................................. 527
17.4.6.2 Jac=d's Score ................................................................ 527
17.4.6.3 Link Evaluation .............................................................. 529
17.4.7 Sixth Issue: Generation of rhe Output Set ............................................. 529
17.4.7.1 ImeracriveGeneration ..................................................... 529
17.4.7.2 Post~Generation ........... .. ..................... 529
17.5 Soft Computing Based Hybrid Fuzzy Controllers ............................ -- .............. 529
17.5.1 Neuro~FuzzySysrem ..................................................................... 530
17.5.2 Real-1ime Adaptive Control of a Direct Drive Motor ................................ 530
17.5.3 GA-Fuzzy Systems for Control of Flexible Robers .................................... 530
17.5.3.1 Application to Flexible Robot Control .................................. 531
17.5.4 GP-FuzzyHierarchical Behavior Control ............................................ 532
17.5.5 GP-Fuzzy Approach ..................................................................... 533
17.6 Soft Computing Based Rocket Engine Control ................................................. 534
17.6.1 Bayesian Belief Networks .............................................................. 535
17.6.2 Fuzzy Logic Control ..................................................................... 536
17.6.3 Software Engineering in Marshall's Flight Software Group ........................... 537
17.6.4 Experimental Appararus and Facility Turbine Technologies SR-30 Engine .......... 537
17.6.5 System Modifications .................................................................... 538
17.6.6 Fuel-Flow Rate Measurement Sysrem .................................................. 538
17.6.7 Exit Conditions Monitoring ................ 538
xxii Contents

17.7 Summary ............ ' 539

17.8 Review QuestiOns ........................ . 539
17.9 Exercise Problems ........................ , ......... . 539
18. Soft Computing Techniques Using C and C++ ............................................... 541
Learning Objectives .~ ......................... , .... , . , ............................................ 541
18.1 Introduction ....................................................................................... 541
18.2 Neural Ne[Work lmplem~macion ................................................................ 541
18.2.1 Perception Network ..................................................................... 541
18.2.2 Adaline Ne[Work ......................................................................... 543
18.2.3 Mad.aline Ne[Work for XOR Function ................................................. 545
18.2.4 Back Propagation Network for XOR Function using Bipolar Inputs and
Binary Targets ............................................................................ 548
18.2.5 Kohonen Self-Organizing Fearure Map ................................................ 551
18.2.6 ART 1 Network with Nine Input Units and Two Cluster Units ..................... 552
18.2.7 ART 1 Network ro Cluster Four Vectors ............................................... 554 '
18.2.8 Full Counterpropagation Network ..................................................... 556
18.3 Fu:z.zy Logic Implementation ..................................................................... 559
18.3.1 lmplemem the Various Primitive Operations of Classical Sets ........................ 559
18.3.2 To Verify Various laws Associated with Classical Sets ................................. 561
18.3.3 To Perform Various Primitive Operations on Fuzzy Sets with
Dynamic Components ................................................................. 566
18.3.4 To Verify the Various laws Associated with Fuzzy Set ................................ 570
18.3.5 To Perform Cartesian Product Over Two Given Fuzzy Sets ......................... 574
18.3.6 To Perform Max-Min Composition ofTwo Matrices Obtained from
Canesian Product .................................. . ... 575
18.3.7 To Perform Max-Product Composition of Two Matrices Obtained from
Canesian Product .............................................. ,..................... 579
18.4 Genetic Algorithm Implementation ............. , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582
18.4.1 To Maximize F(xJ>X2) = 4x, + 3"2 ................................................... 583
18.4.2 To Minimize a Function F(x) = J ........ .. .. .. ....... ............... .... ......... 587
18.4.3 Traveling Salesman Problem (TSP) .................................................... 593
18.4.4 Prisoner's Dilemma .................................................................. 598
18.4.5 Quadratic Equation Solving ....... ................. .. . . . . . . . ................. 602
18.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....... , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..... GOG
18.6 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . G06
19. MATLAB Environment for Soft Computing Techniques ...... . . ......... 607
Learning Objectives .................. . 607
19.1 Introduction .......................... .. 607
19.2 Getting Started with MATLAB
19.2.1 Matrices and Vectors
...... . .. ............. .... ~~~ ::
19.3 Introduction to Simulink ................. .-....................................................... G10
19.4 MATLABNeuralNeworkToolbox ............................................................. 612
19.4.1 Creating a ~usrom Neural Network ................................................... 612
19.4.2 Commands in Neural NeMorkToolbox .............................................. GI4
19.4.3 Neural Ne[Work Graphical User Interface Toolbox .................................. 619

I
L
 xxlii
Contenls

19.5 Fuzzy Logic MATLAB Toolbox ............': ..................................................... 624

19.5.1 Commands in Fu:z.zy Logic Toolbox . ~. ,., .............................................. 625
19.5.2 Simulink Blocks in Fuzzy Logic Toolbox ............................................... 626
19.5.3 Fuzzy!.ogicGUIToolbox ...... L........................... : .............................. 628
19.6 GeneticAlgorithmMATLABToolbox ,; .. -..;:... : ................................................ 631
19.G.1 MATIAB Genetic Algorithm Commands ............................................. 632
19.6.2 Generic Algorithm Graphical User Interface ........................................... 635
19.7 Neural Newark MATLAB Source Codes ....................................................... 639
19.8 Fuzzy LogicMATLAB Source Codes ............................................................ 669
19.9 Genetic Algorithm MATLAB Souq:e Codes .................................................... 681
19.10Summary ........................................................................................ >690.
19.11 Exercise Problems ................................................................................. 690
Bibliography .... : ......................................................... . 691
Sample Question Paper 1 .............................................. . 707
709
Sample Question Paper 2
................712
Sample Question Paper 3 .... .
Sample Question Paper 4 .... )....... . 714
..................... 716
Sample Question Paper 5
Index ............................................................... . 718

L
 0~~ ..... ~
-c::'

hltroduction 1
Learning Objectives - - - - - - - - - - - - - - - - - - - ,
Scope of soft computing. An overview of fuzzy logic.
Various components under soft computing. A note on genetic algorithm.
Description on artificial neural networks The theory of hybrid systems.
with its advantages and applications.

11.1 Neural-Networks
A neural necwork is a processing device, either an algorithm or an actual hardware, whose design was
inspired by the design and functioning of animal brains and components thereof. The computing world
has a lot to gain from neural necworks, also known as artif~eial neural networks or neural net. The neu-
ral necworks have the abili to lear le which makes them very flexible and powerfu--rFDr"'
neural networks, there is no need to devise an algorithm to erfo~2Pec1 c {as -;-rt-r.rrir,ttlere it no
need to understand the internal mec amsms o at task. These networks are also well suited"llr'reir-
time systents because_,.Ofth~G"'f~t"~~PonSe- ana-co~pmarional times which are because of rheir parallel
architecmre.
Before discussing artiftcial neural net\Vorks, let us understand how the human brain works. The human
brain is an amazing processor. Its exact workings are still a mystery. The most basic element of the
human brain is a specific type of cell, known as neuron, which doesn't regenerate. Because neurons aren't
slowly replaced, it is assumed that they provide us with our abilities to remember, think and apply pre-
vious experiences to our every action. The human brain comprises about 100 billion neurons. Each
neuron can connect with up to 200,000 orher neurons, although 1,000-10,000 interconnections arc;
typical.
The -power of the human mind comes from the sheer numbers of neurons and their multiple
interconnections. It also comes from generic programming and learning. There are over 100 different
classes of neurons. The individual neurons are complicated. They have a myriad of parts, subsystems
and control mechanisms. They convey informacion via a host of electrochemical pathways. Together
these neurons and their conneccions form a process which is not binary, not stable, and not syn-
chronous. In short, it is nothing like the currently available elecuonic computers, or even arcificial neUral
networks.
2 lnlroduclion

1.1.1 Artificial Neural Network: Definition

An artificial neural network (ANN) may be defined as an infonnationprocessing model that is inspired by
the way biological nervous systems, such as the brain, process information. This model rriis ro replicate only
the most basic functions of rh~ brain. The key element of ANN is the novel structure of irs information
processing system. An ANN is composed of a large number of highly interconnected prOcessing elements
(neurons) wo_rking in unison to solve specific problems.
Anificial neural networks, like people, learn by example. An ANN is configured for a specific application,
such as pattern recognition or data classification through a learning process. In biological systems, learning
involves adjustments to the synaptic connections that exist between the neurons. ANNs undergo a similar
change that occurs when the concept on which they are built leaves the academic environment and is thrown
into the harsher world of users who simply wa~t to get a job done on computers accurately all the time.
Many neural networks now being designed are statistically quite accurate, but they still leave their users with
a bad raste as they falter when it comes to solving-problems accurately. They might be 85-90% accurate.
t
Unfortunately, few applications tolerate that level of error.

I 1.1.2 Advantages of Neural Networks

Neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, could
be used to extract patterns and detect trends that are too complexro be noticed by either humans or other
computer techniques. A trained neural network could be thought of as an "expert" in a particular cat-
egory of information it has been given m an.Jyze. This expert could be used to provide projections in
new situations of interest and answer "what if' questions. Other advantages of worlcing with an ANN
include:

l. Adaptive learning: An ANN is endowed with the ability m learn how to do taSks based on the data given
for training or initial experience.
2. Selforganizlltion: An ANN can create irs own organization or representation of the information it receives
during learning tiine.
3. Real-time operation: ANN computations may be carried out in parallel. Special hardware devices are being
designed and manufactured to rake advantage of this capability of ANNs.
4. Fault tolerattce via reduntMnt iufonnation coding. Partial destruction of a neural network leads to the
corrcseonding degradation of performance. However, so~ [email protected] .be reJained even
after major ~e.~ dam~e.
.---
Currently, neural ne[\vorks can't function as a user interface which translates spoken words into instructions
for a machine, but someday they would have rhis skilL Then VCRs, home security systems, CD players, and
word processors would simply be activated by voice. Touch screen and voice editing would replace the word
processors of today. Besides, spreadsheets and databases would be imparted such level of usability that would
be pleasing co everyone. But for now, neural networks are only entering the marketplace in niche areas where
their statistical accuracy is valuable.
Many of these niches indeed involve applications where answers provided by the software programs are not
accurate but vague. Loan approval is one such area. Financial institutions make more money if they succeed in
having the lowest bad loan rate. For these instirurions, insralling systems that are "90% accurate" in selecting
the genuine loan applicants might be an improvement over their current selection pro~ess. Indeed, some banks
have proved that the failure rate on loans approved by neural networks is lower than those approved by tkir
1.2 Application Scope of Neural Networks 3

Computer science
Artificial intel~igence Mathematics
(approximation theory,
optimization)

Physics
Dynamical systems
Statistical physics

Economics/finance Engineering
(time series, data mining) Image/signal processing
Control theory robotics

Figure 1~ 1 The multi-disciplinary point of view of neural nerworks.

best traditional methods. Also, some credit card companies are using neural networks in their application
screening process. -
' I h1s newest method of looking into the future by analyzing past experiences has generated irs own unique
set of problems. One such problem is to provide a reason behind a computergenerated answer, say, as to
why a particular loan application was denied. To explain how a network learned and why it recommends a
particular decision has been difficult. The inner workings of neural networks are "black boxes." Some people
have even called the use of neural networks "voodoo engineering." To justifY the decisionmaking process,
several neural network tool makers have provided programs that explain which input through which node
dominates the decision-making process. From this information, experts in the application may be able to infer
which data plays a major role in decision making and its imponance.
Apart from filling the niche areas, neural nerwork's work is also progressing in orher more promising
application areas. The next section of this chapter goes through some of these areas and briefly details
the current work. The objective is to make the reader aware of various possibilities where neural networks
might offer solutions, such as language processing, character recognition, image compression, pattern
recognition, etc.
Neural networks can be viewed from a multi-disciplinary poim of view as shown in Figure 1-l. /
._-"'"

I 1.2 Application Scope of Neural Networks

The neural networks have good scope of being used in the following areas:

I. Air traffic control could be automated with the location, altitude, direction and speed of each radar blip
taken as input to the nerwork. The output would be the air traffic controller's instruction in response to
each blip.
2. Animal behavior, predator/prey relationships and population cycles may be suitable for analysis by neural
networks.
3. Appraisal and valuation of property, buildings, automobiles, machinery, etc. should be an easy task for a
neural network.
4 Introduction

4. Bet#ng on horse races, stock markets, sporting events, etc. could be based on neural network
predictions.
5. Criminal sentencing could be predicted using a large sample of crime details as input and the resulting
semences as output.
6. Compkr physical and chemical processes that may involve the interaction of numerous (possibly unknown)
mathematical formulas could be modeled heuristically using a neural network.
7. Data mining, cleaning and validation could be achieved by determining which records suspiciously diverge
from the pattern of their peers.
8. Direct mail advertisers could use neural network analysis of their databases to decide which customers
should be targeted, and avoid wa.iring money on unlikely targets.
9. Echo pauerns from sonar, radar, seismic and magnetic instrumems could be used to predict meir targets.
10. Econometric modeling based on neural networks should be more realistic than older models based on
classical statistics.
11. Employee hiring could be optimized if the neural nerworks were able to predict which job applicant would
show the best job performance.
12. Expert consultants could package their intuitive expertise imo a neural network ro automate their
services.
13. Fraud detection regarding credit cards, insurance or aXes could be automated using a neural network
analysis of past incidents.
14. Handwriting and typewriting could be recognized by imposing a grid over the writing, then each square
of the grid becomes an input to the neural necwork. This is called "Optical Character Recognition."
15. Lake water levels could be predicted based upon precipitation patterns and river/dam flows.
16. Machinery control could be automated by capturing me actions of experienced machine operators into a
neural network.
17. Medical diagnosis is an ideal application for neural networks.
18. Medical research relies heavily on classical statistics to analyze research data. Perhaps a neural network
should be included in me researcher's tool kit.
19. Music composition has been tried using neural networks. The network is trained to recognize patterns in
the pirch and tempo of certain music, and rhen the network writes irs own music.
20. Photos ttnd fingerprints could be recognized by imposing a fine grid over the photo. Each square of the
grid becomes an input to me neural network.
21. Rmpes ttnd chemicalfonnulations could be optimized based on the predicted outcome of a formula change.
22. Retail inventories could be optimized by predicting demand based on past pauerns.
23. River water levels could be predicted based on upstream reports, and rime and location of each report.
24. Scheduling ofbuses, airplanes and elevators could be optimized by predicting demand.
25. Staffscheduling requiren1:ents for restaurants, retail stores, police stations, banks, etc., could be predicted
based on the customer flow, day of week, paydays, holidays, weather, season, ere.
26. Strategies for games, business and war can be captured by analyzing the expert player's response ro given
stimuli. For example, a football coach must decide whether to kick, piss or ru'n on the last down. The
inputs for cltis decision include score, time, field location, yards w first down, etc.
1.3 Fuzzy Logic 5

27. Traffic flows could be predicted so rhar signal tiiTling could be optimized. The neural network could
recognize "a weekday morning "ru~h hour during a schOol holiday" or "a typi~ winter Sunday morning."
-28. Voice recognition could be obtained by analyzing )he audio oscilloscope panern, much like a smck market
~
1
graph.
29. Weather prediction may be possible. Inputs would)ndude weather reports from surrounding areas.
Outpur(s) would be the future weather in specific areas based on the input information. Effects such as
ocean currents and jet streams could be included.

Today, ANN represents a major extension to computation. Different types of neural networks are available
for various applications. They perform operario1ls akin to the human brain though to a limited o:tent. A rapid
increase is expected in our understanding of me ANNs leading to the improved network paradigms and a
host of ap~lication opporruniries.

11.3 Fuzzy Logic

The concept of fuzzy logic (FL) was conceived by Lotfi Zadeh, a Professor at the UniVersity of California ar
Berkeley. An organized method for dealing wim imprecise darn is_ called fuzzy logic. The data are considered
as fuzzy sets.
Professor Zadeh presented FL nor as a control methodology but as a way_ of processing data by
allowing partial set membership rather than crisp set membership or nonmembership. This approach
to set theory was nor applied to conuol systems until the l970s due to insufficient computer capabil-
ity. Also, earlier me systems were designed only to accept precise and accurate data. However. in certain
Sysrems it is not possible to get the accurate data. Therefore, Professor Zadeh reasoned _mar for process-
ing need nor always require precise and numerical information input; processing can be performed even
with imprecise inputs. Suitable feedback controllers may be designed to accept noisy, imprecise input,
and they would be much more effective and perhaps easier to implement. The processing with impre-
cise inputs led to the growth of Zadeh's FL. Unfortunately, US manufacturers have nor been so quick to
embrace this technology while the Europeans and Japanese have been aggressively building real products
around it.
Fuzzy logic is a superset of convemional (or Boolean) logic and contains similarities and differences with
Boolean logic. FL is similar to Boolean logic in that Boolean logic results are returned by FL operations
when all fuzzy memberships are restricted ro 0 and 1. FL differs from Boolean logic in that it is permissive
of natural language queries and is more like human thinking; it is based on degrees of truth. For exam-
ple, traditional sets include or do nor include an individual element; there is no other case rhan true or
false. However, fuzzy sw allow partial membership. FL is basicallY a multivalued logic that allows inter-
mediate values to be defined between conventional evaluations such as yes/no, tntelfolse, bln.cklwhite, ere.
Notions like rather warm or pretty cold can be formulated mathematically -and processed with the com-
puter. In this way, an attempt is made ro apply a more human-like way of thinking in the programming of
compmers.
Fuzzy logic is a problem-solving control syst.em methodology that lends itself ro implementation in systems
ranging from simple, small, embedded microcontrollers to large, networked, multichannel PC or workstation-
based data ~cquisition and control systems. It can be implemented in hardware, software or a combination of
bot;h. FL Provides a simple way to arrive at a definite conclusion based upon vague, ambiguous, imprecise,
noisy, or missing input information. FLs approach to control problems mimics how a person would make
decisions, only much faster.
6 .Introduction

11.4 Genetic Algorithm

Genetic algorithm (GA) is ieminiscent of sexual reproduction in which the genes of rwo parents combine
to form those of their children. When it is applied ro problem solving, the basic premise is that we can
create an initial population of individuals represencing possible solutions to a problem we are trying ro solve.
Each of iliese individuals has certain characteristics that make them more or less fit as members of the
population. The more fir members will have a higher probability of mating and producing offspring that have
a significam chance of retaining the desirable characteristics of their parents than rhe less fit members. This
method is very effective at finding optimal or near-optimal solurions m a wide variety of problems because it
does nm impose many limirarions required by traditional methods. It is an elegant generate-and-test strategy
dm can identify and exploit regu\ariries in the environment; and results in solutions that are globally optimal
or nearly so.
Genetic algorithms are adaptive computational procedures modeled on the mechanics of natural generic
systems. They express their ability by efficiently exploiting the historical informacion to speculate on new
offspring with expected improved performance. GAs are executed iteratively on a set of coded solutions,
called population, with three basic operators: selection/reproduction, crossover and mutation. They use
only the payoff (objective function) information and probabilistic transition rules for moving to the next
iteration. They are different from most of the normal optimization and search procedures in the following
four ways:
1. GAs work with the coding of the parameter set, not with the parameter themselves;
2. GAs work simultaneously with multiple poims, not a'single point;
3. GAs search via sampling (a blind search) using only the payoff information;
4. GAs search using stochastic operators, not deterministic ~les.
Since a GA works simulraneously on a set of coded solutions, it has very little chance to get stuck at
local optima when used as optimization technique. Again, it does not need any son of auxiliary information,
like derivative of the optimizing function. Moreover, rhe resolution of rhe possible search space is increased
by operating on coded (possible) solutions and not on the solutions themselves. Further, this search space
need not be continuous. Recently, GAs are finding widespread applications in solving problems requiring
efficient and effecrive search, in business, scientific and engineering circles like synthesis of neural netwmk
architecrures, traveling salesman problem, graph coloring, scheduling, numerical optimization, and pattern
recognition and image processing.

I 1.5 Hybrid Systems

Hybrid systems can be classified into three different systems: Neuro fuzzy hybrid system; neuron generic
hybrid system; fuzzy genetic hybrid systems. These are discussed in detail in the following sections.

I 1.5.1 Neuro Fuzzy Hybrid Systems

A neuro fuzzy hybrid system is a fuuy system that uses a learning algorithm derived from or inspired by neural
nerwork theory w determine its parameters (fuzzy sets and fuzzy rules) by processi~g data samples.
In other words, a neuro fuzzy hybrid system refers to die combination of fuzzy set theory and neural
neWorks having advantages of both which are listed below.
1. It can handJe any kind of informacion (numeric, linguistic, logical, etc.).
1.5 Hybrid Systems 7

2. h can manage imprecise, partial, vague or imperfect information.

3. It can resolve conflicts by collaboration and aggregation.
4. Ir has self-learning, self-organizing and self-tunjf)-g.p.p~bilities.
5. It doesn'r need prior knowledge of relationshipS.ofdata.
6. It can mimic human decision-making process., .-
7. It makes computation fast by using fuzzy nurriber operations.
Neuro fuzzy hybrid systems combine the advantages of fuzzy systems, which deal with explicit knowledge
that can be explained and understood, and noural networks, which deal with implicit.knowledge that can be
acquired by learning. Neural nerwork learning provides a good way to adjust the knowledge of the expert (i.e.,
artificial intelligence system) and automatically generate additional fuzzy rules and membership functions
to meet certain specifications. It helps reduce design time and costs. On the other hand, FL enhances the
generalization capability of a neural nerwork system by providing more reliable output when extrapolation is
needed beyond the limirs of the training data.

I 1.5.2 Neuro Genetic Hybrid Systems

Genetic algorithms {GAs) have been increasingly applied in ANN design in several ways: topology opti-
mization, genetic training algorithms and control parameter optimization. In topology optimization, GA
is used to select a topology (number of hidden layers, number of hidden nodes, interconnection parrern)
for the ANN which in turn is trained using some training scheme, most commonly back propagation.
In genetic training algorithms, the learning of an ANN is formu1ated as a weight optimization prob-
lem, usually using the inverse mean squared error as a fitness measure. Many of the control parameters
such as learning rate, momentum rate, tolerance level, etc., can also be optimized using GAs. In addi-
tion, GAs have been used in many other innovative ways, w create new indicators based on existing ones,
select good indicators, evolve optimal trading systems and complement other techniques such as fuzzy
logic.

I 1.5.3 Fuzzy Genetic Hybrid Systems

The optimization abilities of GAs are used to develop the best set of rules to be used by a fuzzy inference
engine, and to optimize the choice of membership functions. A particular use of GAs is in fuzzy classifi-
cation systems, where an object is classified on the basis of the linguistic values of the object attributes.
The most difficult part of building a system like this is to find the appropriate ser of fuzzy rules. The
most obvious approach is to obtain knowledge from experts and translate this into a set of fuzzy rules. But
this approach is time consuming. Besides, experts may not be able to put their knowledge into an appro
priate form of words. A second approach is to obtain the fuzzy rules through machine learning, whereby
the knowledge is auromatically extracted or deduced from sample cases. A fuzzy GA is a directed random
search over all {discrete) fuzzy subsets of an interval and has features which make it applicable for solving
this problem. It is capable of creating the classification rules for a fuu.y system where objects are classi-
fied by linguistic terms. Coding the rules genetically enabies the system to deal with mulcivalue FL and is
more efficient as it is consistent with numeric ood.ing of fuzzy examples. The training data and randomly
generated rules are combined to create the initial population, giving a better starring point for reproduc-
tion. Finally, a fitness function measures the strength of the rules, balancing the quality and diversity of ilie
population.
a Introduction

11.6 Soft Computing

The two major problem-solving technologies include:

1. hard computing;
2. soft computing.

Hard computing deals wirh. precise models where accurate solmions are achieved quickly. On the other
hand, soft computing deals with approximate models and gives solution to complex problems. The t<.yO
problem-solving technologies are shown in Figure 12.
Soft computing is a relatively new concept, the rerm really entering general circulation in 1994. The term
"soft computing" was introduced by Professor Lorfi Zadeh with the objective of exploiting the tolerance
for imprecision, uncenaincy and partial truth tO achieve tractability, robustness, low solution cost and better
rapport with realicy. The ultimate goal is m be able to emulate fie human mind as closely as possible. Soft
compuring involves parmership of several fields, the mosr imponam being neural nerworks, G~ and FL. Also
included is the field of probabilistic reasoning, employed for its uncertaincy control techniques. However, this
field is nor examined here.
Soft computing uses a combination of GAs, neural nerworks and FL. A hybrid technique, in fact, would
inherit all the advantages, but won't have the less desirable features of single soft computing componems. It
has to possess a good learning capacicy, a better learning time than that of pure GAs and less sensitivity to
the problem of local extremes than neural nerworks. In addition, it has m generate a fuzzy knowledge base,
which has a linguistic representation and a very low degree of computational complexity.
An imponam thing about the constituents of soft computing is that they are complementary, not camper~
itive, offering their own advantages and techniques to pannerships to allow solutions to otherwise unsolvable
problems. The constituents of soft computing are examined in turn, following which existing applications of
partnerships are described.
"Negotiation is the communication process of a group of agents in order to reach a mutually accepted
agreement on some matter." This definition is typical of the research being done into negotiation and co~
ordination in relation to software agents. It is an obvious necessity that when multiple agents interact, they
will be required to co-ordinate their efforts and attempt to son our any conflicts of resources or interest.
It is important to appreciate rhar agents are owned and conrrolled by people in order to complete tasks on
their behalf. An exampl! of a possible multiple-agent-based negotiation scenario is the competition between

HARD COMPUTI~JG SOFT COMPUTING

l Precise models
J
I
t t
Symbolic Traditional
logic numerical Approximate
reaSDiling modeling and reasoning
~ditio~ search

Figure 12 Problem-solving technologies.

1.7 Summary
9

long~disrance phone call providers. When the consumer picks up the phone and dials, an agent will com-
municate on the consumer's behalf with all the available nerwork providers. Each provider will make an
offer that the consumer agent can accept of reje~. _'A realistic goal would be to select the lowest avail~
able- price for .the call. However, given the first rOurid_.df offers, network providers may wish to modifY
their offer to make it more competitive. The new offer is then submitted to the consumer agenr and the
process continues until a conclusion is reached. One advantage of this process is that the provider can
dynamicaUy alter its pricing strategy to account for changes in demand and competidon, therefore max~
imizing revenue. The consumer will obviously benefit from the constant competition berween providers.
Best of all, the process is emirely amonompus as the agents embody and act on the beliefs and con-
straints of the parties they represent. Further changes can be made to the protocol so that providers
can bid low without being in danger of making a loss. For example, if the consumer chooses to go
with the lowest bid but pays the second lowest price, this will rake away the incentive to underbid or
overbid.
Much of the negotiation theory is based around human behavior models and, as a result, it is oft:en trans-
lated using Distributed Artificial Intelligence techniques. The problems associated with machine negotiation
are as difficult to solve as rhey are wirh human negotiation and involve issues such as privacy, security and
deception.

11.1 Summary

The computing world has a lot to gain from neural networks whose ability to learn by example makes them
very flexible and powerful. In case of neural nerworks, there is no need to devise an algorithm to perform a
specific task, i.e., there is no need to understand the imernal mechanisms of that rask. Neural networks are
also well suited for real-time systems because of their fast response and computational times, which are due
to their parallel architecture.
Neural nerworks also contribute to other areas of research such as neurology and psychology. They are
regularly used tO model parts of living organisms and to investigate the internal mechanisms of the brain.
Perhaps the most exciting aspect of neural nerworks is the possibility that someday "conscious" networks
n:aighr be produced. Today, many scientists believe that consciousness is a "mechanical" property and that
"conscious" neural nerworks are a realistic possibility.
Fuzzy logic was conceived as a better method for sorting and handling data but has proven to be an excellent
choice for many control system applications since it mimics human comrollogic. It can be built inro anything
from small, hand-held producrs to large, computerized process control systems. It uses an imprecise but very
descriptive language to deal with input data more like a human operator. It is robust and often works when
first implemented with little or no tuning.
When applied to optimize ANNs for forecasting and classification problems, GAs can be used to search
for the right combination of inpur data, the most suitable forecast hori7:0n, the optimal or near-optimal
network interconnection patterns and weights among the neurons, and the conuol parameters (learning ~te,
momentum rate, tolerance level, etc.) based on the uaining data used and the pre~set criteria. Like ANNs,
GAs do not always guarantee you a perfect solution, but in many cases, you can arrive at an acceprable solution
without die rime and expense of an exhaustive search.
Soft computing is a relatively new concept, the term really entering general circulation in 1994, coined by
Professor Lotfi Zadeh of the University of California, Berkeley, USA, it encompasses several fields of comput-
ing. The three that have been examined in this chapter are neural nerworks, FL and GAs. Neural networks are
important for their ability to adapt and learn, FL for its exploitation of partial truth and imprecision, and GAs
 -=1

10 Introduction I
--1

for their application to optimization. The field of probabilistic reasoning is also sometimes included under the
soft computing umbrella foi- its control of randomness and uncertainty. The importance of soft computing
I
lies in using these methodologies in partnership - they all offer their own benefits which are generally nor
competitive and can therefore, work together. As a result; several hybrid systems were looked at - systems in
which such partnerships exist.
 or' . ~c~~t} en}
---==
Artificial Neural Network:
An Introduction 2
learning Objectives
The fundamema.ls of artificial neural net~ Various terminologies and notations used
work. throughout the text.
The evolmion of neural networks. The basic fundamental neuron model -
Comparison between biological neuron and McCulloch-Pins neuron and Hebb network.
:inificial neuron. The concept of linear separability to form
Basic models of artificial neural networks. decision boundary regions.
The different types of connections of neural
nern'orks, learning and activation functions
are included.

;:;,\1 bjyO
)-(o,~ c.
ri ..~I',"
'~ .
J
I 2.1 Fundamental Concept
Neural networks are those information processing systems, which are constructed and implememed to model
the human brain. The main objective of the neural network research is to develop a computational device
for modeling the brain to perform various computational tasks at a faster rate .than the traditional systems .
.-..., Artificial neural ne.~qrks perfOFm various tasks such as parr~nmarchjng and~"dassificarion. oprimizauon
~on, approximatiOn, vector uamizatio d data..clus.te..di!fThese_r__'!_5~~~2'!2'..J~~for rraditiOiiif'
Computers, w ,c are er 1 gomll~putational raskrlndrp;;ise !-rithmeric operatic~. Therefore,
for implementation of artificial n~~speed digital corrlpurers are used, which makes the
simulation of neural processes feasible.

I 2.1.1 Artificial Neural Network

& already stated in Chapter 1, an artificial neural nerwork (ANN) is an efficient information processing
system which resembles in characteristics with a biological neural nerwork. ANNs possess large number of
highly interconnected processing elements called notUs or units or neurom, which usually operate in parallel
and are configured in regular architectures. Each neuron is connected wirh the oilier by a connection link. Each
connection link is associated with weights which contain info!11ation about the_iapu.t signal. This information
is used by rhe neuron n;t to solve a .Particular pr.cl>lem. ANNs' collective behavior is characterized by their
ability to learn, recall and' generaUa uaining p:erns or data similar to that of a human brain. They have the
ro
capability model networkS of ongma:l nellfOIIS as-found in the brain. Thus, rhe ANN processing elements
are called neurons or artificial neuro'f\, , , l"-
( \- ,.'"
c' ,, ,
'-- .r ,-
\ ' \
12 ~ Artificial Neural Network: An Introduction

x, X,
...

~@-r
Figure 21 Architecture of a simple anificial neuron net.

Input '(x) :- ~- i ' (v)l----~mx

~--.J

Figure 22 Neural ner of pure linear equation.

It should be noted that each neuron has an imernal stare of its own. This imernal stare is called ilie
activation or activity kv~l of neuron, which is the function of the. inputs the neuron receives. The activation
signal of a neuron is transmitted to other neurons. Remembe(i neuron can send only one signal at a rime,
which can be transmirred to several ocher neurons.
To depict rhe basic operation of a neural net, consider a set of neurons, say X1 and Xz, transmitting signals
to a110ilier neuron, Y. Here X, and X2 are input neurons, which transmit signals, andY is the output neuron,
which receives signals. Input neurons X, and Xz are connected to the output neuron Y, over a weighted
interconnection links (W, and W2) as shown in Figure 21.
For the above simple rleuron net architecture, the net input has to be calculated in the following way:

]in= +XIWI +.xz102

where Xi and X2 ,gL~vations of the input neurons X, and X2, i.e., the output of input signals. The
output y of the output neuron Y can be o[)i"alneaOy applymg act1vanon~er the ner input, i.e., the function
of the net input:

J = f(y;,)
Output= Function (net input calculated)

The function robe applied over the l]t input is call:a;;dti:n fo'!!!f_on. There are various activation functions,
which will be discussed in the forthcoming sect10 _ . e a ave calculation of the net input is similar tq the
calculation of output of a pure linear straight line equation (y = mx). The neural net of a pure linear cqu3.tion
is as shown in Figure 22.
Here, m oblain the output y, the slope m is directly multiplied with the input signal. This is a linear
equation. Thus, when slope and input are linearly varied, the output is also linearly varied, as shown in
Figure 23. This shows that the weight involved in dte ANN is equivalent to the slope of the linear straight
line.

I 2.1.2 Biological Neural Network

It iswdlknown that dte human brain consists of a huge number of neurons, approximatdy 10 11 , with numer
ous interconnections. A schematic diagram of a biological neuron is s_hown in Figure 2-4.
 2.1 Fundamental Concept 13

Slope= m

t
y
,

X----->-

Figure 23 Graph for y = mx.

Synapse

Nucleus --+--0

7 /
-c_....-- . v,
DEindrites 1: t 1 1

Figure 2-4 Schcmacic diagram of a biological neuron.

~
The biological neuron depicted in Figure 2-4 consists of dtree main pans:
1. Soma or cell body- where the cell nucleus is located.
2. Dendrites- where the nerve is connected ro the cell body.
3. Axon- which carries ~e impu!_s~=-;t the neuron.
Dendrites are tree-like networks made of nerve fiber connected to the cell body. An axon is a single, long
conneC[ion extending from the cell body and carrying signals from the neuron. The end of dte axon splits into
fineruands. It is found that each strand terminates into a small~ed JY1111pse. Ir is duo
na se that e neuron introduces its si nals to euro , T e receiving ends o e a ses
~}be nrarhr neurons can be un both on the dendrites and on y. ere are approximatdy
f :,}_er neuron in me numan Drain.
~es are passed between the synapse and the dendrites. This type ofsignal uansmission involves
a. chemical process in which specific transmitter substances are rdeased from the sending side of the junccio
This results in increase or decrease in th~ inside the bOdy of the receiving cell. If the dectric
potential reaches a threshold then the receiving cell fires and a pulse or action potential of fixed strength and
duration is sent oulihro'iigh the axon to the apcic junctions of the other ceUs. After firing, a cd1 has to wait
for a period of time called th efore it can fire again. The synapses are said to be inhibitory if
they let passing impulses hind the receiving cell or txdtawry if they let passing impulses cause
the firing of the receiving cell.
----""
 .J

14

Inputs
Artificial Neural Network: An Introduction

j
~ Weights
x, ~
";

Processing
w,
element
~/
X,

Figure 25 Mathematical model of artificial neuron.

Table 21 Terminology relarioii:ShrpS b~tw-een

biological and artificial neurons
Biological neuron Anificial neuron
Cell Neuron
Dendrites Weights or inrerconnecrions
Soma Nee inpur
Axon Outpm

Figure 2~5 shows a mathematical represenracion of the above~discussed chemical processing raking place
in an artificial neuron.
In chis model, the net input is elucidated as

Yin = Xt WJ + XzW2 + + x,wn = L" x;w;

i=l

where i represents the ith processing elemem. The activation function is applied over it ro calculate the
output. The r-reighc represents the strength of synapse connecting the input and the output neurons. ft pos
irive weight corresponds to an excitatory synapse, and a negative weight corresponds to an inhibitory
synapse.
The terms associated with the biological neuron and their counterparts in artificial neuron are prescmed '!.
in Table 2-l.

2.1.3 Brain vs. Computer - Comparison Between Biolbgical Neuron and

Artificial Neur9n (Brain vs. Computer)

A comparison could be made between biological and artificial neurons on the basis of the following criteria:
1. Speed T~e of rxecurion in the ANN is of& .. wannsergnds whereas in the ci.se of biolog-
ical neuron ir is of a few millisecondS. Hence, the artificial neuron modeled using a com purer is more
faster. -

I
l
2. f Fundamental Concept 15

2. jJ'ocessing: Basically, the biological neuron can perform massive paralld operations simulraneously. The
artificial neuron can also perform several parallel operations simultaneouSlY, but, ih general, the artificial
neuron ne[INork process is faster than that of the brain. .
3. Size and complexity: The total number of neUrons in the brain is about lOll and the total number of
interconnections is about 1015 Hence, it can be rioted that the complexity of the brain is comparatively
higher, i.e. the computational work takes places not"Cmly in the brain cell body, but also in axon, synapse,
ere. On the other hand, the size and complOciry ofan ANN is based on the chosen application and
the ne[INork designer. The size and complexity of a biological neuron is more than iliac Of an arcificial
neurorr.-----
4. Storage capacity (mnno,Y}: The biologica.l. neuron stores the information in its imerconnections or in
synapse strength but in an artificial neuron it is smred in its contiguous memory locations. In an artltlcial
neuron, the continuous loading of new information may sometimes overload the memory locations. As a
result, some of the addresses containing older memory locations may be destroyed. But in case of the brain,
new information can be added in the interconnections by adjusting the strength without descroying the
older infonnacRm. A disadvantage related to brain is that sometimes its memory niay fail to recollect the.
stored information whereas in an artificial neuron, once the information is stored in its me~ locations,

-
it can be retrieved. Owing to these facts, rhe adaptability is more toward an artificial neuron.
5. Tokrance: The biola ical neuron assesses fault tolerant capability whereas the artificial neuron has no
fault tolerance. Th distributed natu of the biological neurons enables to store and retrieve information
even when the interconnections m em get disconnected. Thus biological neurons nc fault toleF.lm. But in
case of artificial neurons, the mformauon gets corrupted if the network interconnections are disconnected.
Biological neurons can accept redundancies, which is not possible in artificial neurons. Even when some
ceHs die, the human nervous system appears to be performing with the same efficiency.
6. Control mechanism: In an artificial neuron modeled using a computer, there is a control unit present in
Central Processing Unit, which can transfe..! and control precise scalar values from unit to unit, bur there
is no such control unit for monitoring in the brain. The srrengdl of a neuron in the brain depends on the
active chemicals present and whether neuron connections are strong or weak as a result ~mre layer
rather t~ synapses. However, rhe ANN possesses simpler interconnections and is freefrom
chemical actions similar to those raking place in brain (biological neuron). Thus, the control mechanism
of an arri6cial neuron is very simple compared to that of a biological neuron. --

So, we have gone through a comparison between ANNs and biological neural ne[INorks. In shan, we can
say that an ANN possesses the following characteristic.s:

1. It is a neurally implemented mathem~

2. Ther~lilgfi(y'"interconnected processing elements called nwrom in an ANN.
3. The interconnections with their weighted linkages hold the informative knowledge.
4. The input signals arrive at the processing elelnents through connections and connecting weights.
5. The processing elements of the ANN have the ability to learn, recall and generalize from the given data
by suitable assignment or adjustment of weights.
6. The computational power can be demonstrated only by the collective behavior of neurons, and it should
be noted that no single neuron carries specific information.

The above-mentioned characteristic.s make the ANNs as connectionist models, parallel distributed processing

--
models, self-organizing systems, neuro-computing systems and neuro-morphic systems.
 :w

16 Artificial Neural Network: An Introduction "

I 2.2 Evolution of Neural Networks

The evolution of neural nenvorks has been facilitated by the rapid developmenr ofarchitectUres and algorithms
that are currently being used. The history of the developmenr of neural networks along with the names of
their designers is outlined Tab!~ 2~2.
In the later years, the discovery of the neural net resulted in the implementation of optical neural nets,
Boltzmann machine, spatiotemporal nets, pulsed neural networks and support vector machines.

Table 22 Evolution of neural networks

Year Newal Designer Description
necwork
1943 McCulloch md McCulloch and The arran gemem of neurons in this case is a combination of logic
Pitts neuron Pins functions. Unique feature of this neuron is the concept of
threshold.
1949 Hebb network Hebb It is based upon the fact that if two neurons are found to be active
simulraneously then the strength of the connection bmveen them
should be increased.
1958, Percepuon F<>nk Here the weighrs on the connection path can be adjusted.
1959. Rosenblau,
1962, Block, Minsky
1988 and Papert
1960 Adaline Widrow and Here the weights are adjusted ro reduce the difference between the
Hoff net input to the output unit and the desired output. The result
here is very negligible. Mean squared error is obtained.
1972 Kohonen Kohonen The concept behind this network is that the inputs are clustered
self-organizing together to obtain a fired ourput neuron. The clustering is
feature map performed by winner-take all policy.
1982, Hopfield John Hopfidd This neural network is based on fixed weights. These nets can also
1984, network and Tank act as associative memory nets.
1985,
1986,
1987
1986 Back- Rumelhart, This network is multi-layer wirh error being propagated backwards
propagation Hinton and from the output unirs ro the hidden unirs.
network WiUiams
1988 Counter- Grossberg This network is similar ro rhe Kohonen network; here the learning
propagation occurs for all units in a panicular layer, and there exists no
network competition among these units.
1987- Adaptive Carpenter and The ART network is designed for both binary inputs and analog
1990 Resonance Grossberg valued inpur.s. Here the input pauems can be presented in any
Theory <ARn order.
1988 Radial basis Broomhead and This resembles a back propagation network bur the activation
&merion Lowe function used is a Gaussian function.
network
1988 Neo cogniuon Fukushima This network is essential for character recognition. The deficiency
occurred in cogniuon network (1975) was corrected by this
network.
:w:

" 17
2.3 Basic Models of Artificial Neural Network

~are specified by the three basic. entities namely:

1. the model's synaptic interconnectionS;
2. the training or learning rules adopted for upda~ng arid adjusting the connection weights;
3. their activation functions.

I 2.3.1 Connections
The neurons should be visualized for their arrangements in layers. An ANN consists of a set of highly inter-
connected processi elements (neurons) such that each processing element output is found robe connected
throughc.. e1g ts to the other processing elements or to itself, delay lead and lag-free_.conn'eccions are allowed.
Hence, the arrange!llents of these orocessing elements and-dl'e" g:ametFy o'f-tJiciC'interconnectipns are essential
for an ANN. The point where the connection ongmates and terminates should De noted, :ind the function
o ea~ processing element in an ANN should be specifie4.
Bes1 es e pie neuron shown in Figure??, there exist several other cypes of neural network connections.
/fie arrangement of neuron:2form layers and the connection panem formed wi~in and between layers is
~led the network architecture. here exist five basic types of neuron connection architectUres. They are:
1. single-layer feed-forwar network;
2. multilayer feed-forward network;
3. single node with itS own feedback;
4. single-layer recurrent network;
5. mulrilayer recurrent network.
Figures 2-6-2-10 depict the five types of neural network architectures. Basically, neural nets are classified
into single-layer or multilayer neural ners. A layer is formed by taking a processing element and combining it
wirh other processing elements. Practically, a layer implies a stage, going stage by stage, i.e., the input srageand
the output stage are linked with each other. These linked interconnections lead to the formation of various
netw-ork architecrures. When a layer of the processing nodes is formed, the inputs can be connected to these

I
lnpul Output

.lI layer layer

'
I Output '
neurons

I
I
I
Figure 26 Single~layer feed-forward network.

I
18/ Artificial Neural Network: An Introduction j
Input I
layer

Output
neurous

Figure 27 Multilayer feed-forward network.

Output
Input
--------..
~
-

Feedback
-
(A) (B)

Figure 28 (A) Single node wirh own feedback. {B) Comperirive ners.

nodes with various weighrs, resulting in&MJ.n)rnp;u~~~eope~ Thus, a single-laye1 feed-forward

netw rk is formed.
A mu t1 erfeed-forward network (Figure 2-?) is formed by the interconnection of several layers. The
input layer is that which receives the input and this layer has no function except buffering the input si nal.
The output layer generates the output of the network. Any layer that is formed between e input and output
layers is called hidden layer. This hidde-n layer is internal to the network and has no direct contact with the
external environment. It should be noted that there may be zero to several hidden layers in an ANN. More the
number of the hidden layers, more is Ute com lexi f Ute network This may, however, provide an efficient
output response. In case of out ut from one layer is connected to d
evlill' node in the next layer.
A n'etw.Qrk is said m be a feed~forward nerwork if no neuron in the output layer is an input to a node in
the same layer or in the preceding layer. On the other hand, when ou uts can be directed back as inputs to
same or pr..t:eding layer nodes then it results in me formation e back networ. .
If the feedback of clte om put of clte processing elements ts recred back at input tO the processing
elements in the same layer r.fen ic is tailed ilueral feedbi:Uk. Recurrent networks are feedback networks
with d(\'ied loop. Figure 2~8(A) shows a simple recurrent neural network having a single neuron with
2.3 Basic Models of Artificial Neural Network 19

. ~~! ...

0> ~"

0 wnm ..

Figure 29 Singlelayer recurrent network.

Input layer

Y, )---'-----

0 "" Vn

. .. <:.:..::,/.......
,

"\\ -::::

0 "'" v~

0. . ~n2
Figure 210 Multilayer recurrent ne[Work.

feedback to itself. Figure 2~9 shows a single layer network with a feedback connection in which a processing
element's output can be directed back ro the processing element itself or to clte other processing element or
to both.
The architecture of a competitive layer is shown in Figure 2~8(8), the competitive interconneccions having
fixed weights of -e. This net is called Maxnet, and will be discussed in the unsupervised learning network
category. Apart from the network architectures discussed so far, there also exists another type of archirec~
rure with lateral feedback, which is called the oncenter--off-surround or latmzl inhibition strUCture. In this
~ ----
 20 Artificial Neural Network: An Introduction
,
r
<
'
__o;,L
0

,,
1.
;~'' ? rl
1

__ ~.\~;-.\
'h-'"'""==::':::C:=:=:'::
Flgure2-11~on.&~r~
c< \,. ,'
-~-'
o'
c~ \'
..;'s~?ucture, each processing neuron receives two differem classes of inputs- "excitatory" input &om nearby ~
processing elements and "inhibitory" inputs from more disramly_lggted..pro@~ elements. This cype of
inter~ is shown in Figure"2:-1T:----------- ----~
In Figure 2-11, the connections with open circles are excitatory connections and the links with solid con-
nective circles are inhibitory connections. From Figure 2-10, it can be noted that a processing element output
can be directed back w the nodes in a preceding layer, forming a multilayer recunmt network. Nso, in these
networks, a processing dement output can be directed back to rhe processing element itself and to other pro-
cessing elemenrs in the same layer. Thus, the various network architecrures as discussed from Figures 2~6-211
can be suitably used for giving effective solution ro a problem by using ANN.

I 2.3,2 Learning

The main property of an ANN is its capability to learn. Learning or training is a process by means of which a
neural network adapts itself to a stimulus by making$rop~~rer adjustm~ resulting in the production
of desired response. Broadly, there are nvo kinds o{b;ning in ANNs:
1. Parameter learning: h updates the connecting weights in a neural net.
2. Strncttm learning: It focuses on the change in network structure (which includes the number of processing
elemems as well as rheir connection types).
The above two types oflearn.ing can be performed simultaneously or separately. Apart from these two categories
of learning, the learning in an ANN can be generally classified imo three categories as: supervised learning;
unsupervised learning; reinforcement learning. Let us discuss rhese learning types in detail.

2-_3,2, 1 Supervised Learning

The learning here is performed with the help of a teacher. Let us take the example of the learning process
of a small child. The child doesn't know how to readlwrite. He/she is being taught by the parenrs at home
and by the reacher in school. The children are trained and molded to recognize rhe alphabets, numerals, etc.
Their each and every action is supervised by a teacher. Acrually, a child works on the basis of the output that
he/She has to produce. All these real-time events involve supervised learning methodology. Similarly, in ANNs
following the supervised learning, each input vector re uires a cor din rar et vector, which represents
the desired output. The input vecror along with the target vector is called trainin
informed precisely about what should be emitted as output. The block 1a
working of a supervised learning network.
During training. the input vector is presented to the network, which results in an output vecror. This
outpur vector is the actual output vecwr. Then the actual output vector is compared with the desired (target)
output vector. If there exists a difference berween the two output vectors then an error signal is generated by
, 2.3 Basic Models of Artificial Neural Network

X -+
Neural
network y
21

(lnpu :) w (Actual output)

Error
<
Error
(0-Y) b
signal
signals (Desi ad output)
generator

Figure 2-12 Supervised learning.

ence, the

2.3,2,2 Unsupervised Learning

The learning here is performed without the help of a teacher. Consider the learning process of a tadpole, it
learns by itself, that is, a child fish learns to swim by itself, it is not taught by its mother. Thus, its learning
process is independent and is nor supervised by a teacher. In ANNs following unsupervised learning, the
\ input vectors of simil~pe are grouped without th use of training da.ta t specify ~ch
'~ group looks or to which group a number beloogf n e training process, efietwork receives rhe input
~-~paii:erns and organizes these patterns to form clusters. When a new input panern is applied, the neural
" network gives an output response i dicar.ing..ili_~c which the input pattern belongs. If for an input,
a pattern class cannot be found the a new class is generated The block 1agram of unsupervised learning is
shown in Figure 2~13.
From Figure 213 it is clear that there is no feedback from the environment to inform what the outputs
should be or whether the outputs are correct. In this case, the network must itself discover patterns~~
lariries, features or categories from the input data and relations for the input data over (heOUtj:lut. While
discovering all these features, the network undergoes change m Its parameters. I h1s process IS called self
organizing in which exact clusters will be formed by discovering similarities and dissimilarities among the
objects.

2.3.2.3 Reinforcement Learning

This learning process is similar ro supervised learning. In the case of supervised learning, the correct rarget
output values are known for each input pattern. But, in some cases, less information might be available.

~
X y
(lnpu al output)

Figure 2-13 Unsupervised learning.

22 Artificial Neural Network: An Introduction

Neural
X network y
(lnpu t) w (Actual output)

Error Error
signals signal A
generator (Relnlforcement
siignal)

Figure 2~14 Reinforcement learning.

For example, the necwork might be told chat its actual output is only "50% correct" or so. Thus, here only
critic information is available, nor the exacr information. The learning based on this crjrjc jofnrmarion is
called reinforCfment kaming and the feedback sent is called reinforcement sb
The block diagram of reinforcement leammg IS shown in Figure 2-14. The reinforcement learning is a
form of su ervis the necwork receives some feedback from its environment. However, the
feedback obtained here is only evaluative and not mstrucr1ve. e extern rem orcemenr signals are processed
in the critic signal generator, andilie obtained ;rnc signals are sent to the ANN for adjustment of weights
properly so as to get better critic feedback in furure. The reinforcement learning is also called learning with a
critic as opposed ro learning with a teacher, which indicates supervised learning.
So, now you've a fair understanding of the three generalized learning rules used in the training process of
ANNs.

I 2.3.3 Activation Functions

To better understand the role. of the activation function, let us assume a person is performing some work.
To make the work more efficient and to obrain exact output, some force or activation may be given. This
aaivation helps in achieving the exaa ourpur. In a similar \vay, the aaivation function is applied over the net
inpu~eulate.the output of an ANN.
The information processing of a processing element can be viewed as consisting of two major parts: input
and output. An integration fun~tion (say[) is associated with the input of a processing element. This function
serves to combine activation, information or evidence from an external source or other processing elements
into a net mpm ro the processing element. I he nofllmear actlvatlon-fi:iiicfion IS usei:l to ensure that a neuron's
response is ~nded - diat 1s, the acrual response of the neuron is conditioned or dampened as a reru.h-of
large or small activating stimuli and is thus controllabl_s.
Certain nonlinear fllncnons are used to aCh.eve dle advantages of a multilayer network from a single-layer
nerwork. When a signal is fed thro~ a multilayer network with linear activation functions, che output
obtained remains same as that could be obtained using a single~layer network. Due to this reason, nohlinear
functions are widely used in multilayef networks compared ro linear functions.
There are several activation functions. Let us discuss a few in chis section: rF\
1. Identity fimction: It is a linear function and can be defined as 'I. \Y ':I '
(.

f(x) = x foe all x \.r.'

' \,,' . '-'
-~

c~-
2.3 Basic Models of Artificial Neural Network
23

The output here remains the same as input. The input layer uses the idemity activation function.
2. Binary step function: This function can be defined as

f(x) = { 1 if x) e
0 1fx<e

where 8 represents the lhreshold value. This function is most widely used in single-layer nets to convert
the net input to an output that is a binary (1 or 0).
3. Bipolar step fimction: This function can be defined as

'f(x)=\ .1 ifx)8
-1 tf x< (}

where 8 represents the dueshold value. This function is also used in single-layer nets to convert the nee
input to an output that is bipolar(+ 1 or -1).
4. Sigmoidal fonctions-. The sigmoidal functions are widely used in back-propagation nets because of the
relationship between the value of the functions ar a point and the value of the derivative at that ~nt
which reduces the computational blJ!den d~ng.
Sigm01dil funcnons are of two types: -
Binmy sigmoid fonction: It is also rermed as logistic sigmoid function or unipolar sigmoid function.
It can be defined as
I
f(x) = 1 + ,-'-'

where A is the steepness parameter. The derivative of rhis funcrion is

c--------... """\
/ J'(x) =J.f(x)[l- f(x)] \

Here the range of che sigmoid funct~~iS"fr~~ Qr~ 1~ -- - ___ ..

Bipo!dr sigmoid fimction: This function is defined as

2 1-e-Ax
f ( x )1= ---1=--
+ e-Ax l + e-Ax
where A is thesteef'n~~rand the sigmoid function range is between -1 and+ 1. The derivative
ofthisiilliC:~.: I ..

A
J'(x) = [1 +f(x)][l - f(x)]
2
The bipolar sigmoidal function is closely related ro hyperbolic rangenr &merion, which is written as
et-e-x 1-e-b:
h(x)=--=--
r+e-x 1 +e-2x
The derivative of the hyperbolic tangent function is

h'(x) =[I +h(x)][l- h(x)]

24 Artificial Neural Network: An Introduction

If the nerwork uses a binary data, it is better to conven it to bipolar form and use ilie bipolar sigmoidal
acnvauon funcnon or hyperbolic tangent function.

5. Ramp function: The ~p funaion is defined as

if X> 1
f(x) = U if Q.:::: X .:5: 1
if x< 0

The graphical representations of all the activation functions are Shown in Figure 2-I5(A)-(F).

I 2.4 Important Terminologies of ANNs

This section introduces you ro the various terminologies related with ANNs.

I 2.4.1 Weights

In the architecrure ofan ANN, each neuron is connected ro other neurons by means ofdirected communication
links, and each communication link is associated with weights. The weighrs contain information about e
if'!pur ~nal. This information is used by the net ro solve a problem. The we1ghr can ented in
-rem1sOf matrix. T4e weight matrix can alSO bt c:rlled connectzon matrix. To form a mathematical notation, it
is assumed that there are "n" processingelemenrs in~ each processing element has exaaly "m"
adaptive weighr.s. Thus, rhe weight matrix W is defined by

wT\ \w'' WJ2 WJm

\
'',.
-,,
WT W22 \~,, "\

W=
2

I=
""' IU)_m

'"'
w~j LWn] 7Vn2 1Unm

where w; = [wil, w;2 ... , w;m]T, i = 1,2, ... , n, is the weight vector of processing dement and Wij is the
weight from processing element":" (source node) to processing element "j' (destination node).
If the weight matrix W contains all the adaptive elements of an ANN, then the set of aH W matrices
will determine dte set of all possible information processing configurations for this ANN. The ANN can be
realized by finding an appropriate matrix W Hence, the weights encode long-term memory (LTM) and rhe
activation states of neurons encode short-term memory (STM) in a neural network.

I 2.4-2 Bias

The hi the necwork has its impact in calculating the net input. The bias is included by adding
a component .ro 1 to the input vector us, the input vector ecomes

X= (l,XJ, ... ,X;, ... ,Xn)

2.4 Important Tenninologies of ANNs 25

1 ,l(x)

f(x)'
0 X

X
(A) (B)

I(!C)

+1f-----

0 X
\
-1

(C) (D)

l(x),

I(!C)

+1

+1 X
(E) (F)

Figure 2-15 Depicrion of activation functions: (A) identity function; (B) binary step function; (C) bipolar step
function; (D) binary sigmoidal function; (E) bipolar sigmoidal function; (F) ramp function.

The bias is considered. like another weight, dtat is&= b}

Consider a simple network shown in Figure 2-16
with bias. From Figure 2-16, the net input to dte ourput neuron Yj is calculated as

"
Jinj = Lx;Wij = XOWOj +X] W]j + XlWJ.j + + X 11 Wnj

i=O
"
=wo1+ Lx;wif
i=l

"
Ji"j = bj + Ex;wij
i=l
26 Artificial Neural Network: An Introduction

bj
w,J

X~ w11 :"(
w,l

x,
Figure 216 Simple net with bias.

c(Bias)

(Weight) ~
Input J@ m ]; Y ) y.=mx+c

Figure 217 Block diagram for straight line.

The activation function discussed in Section 2.3.3 is applied over chis nee input to calculate the ouqmt. The
bias can also be explain~d as follows: Consider an equation of straight line,

y= mx+c
where xis the input, m is rhe weight, cis !he bias andy is rhe output. The equation of the suaight line can
also be represemed as a block diagram shown in Figure 2~17. Thus, b}as plays a major role in dererrnj_njng
the ouq~ut of rhe nerwork.
The bias can be of two types: positive bias and negaiive bias. The positive bias helps in increasing ~et
input of the network and rhe negative bias helps in decreasing the n_~_r)!!.R-1.!-.~ o(Jli!!_p.et\licid{. I hus, as a result
of the bias effect, the output of rhe network can be varied. ---

I 2.4.3 Threshold

Thr~ldis a set yalue based upon which the final outp_~t-~f ~e network may be calculated. The threshold
vafue is used in me activation function. X co.mparrson is made between the Cil:co.lared:net>input and the
threshold to obtain the ne ork outpuc. For each and every apPlicauon;mere1S'a-dlle5hoidlimit. Consider a
direct current DC) motor. If its maximum spee~then lhe threshold based on the speed is 1500
rpm. If lhe motor is run on a speed higher than its set threshold,-it-m~amage motor coils. Similarly, in neural
networks, based on the threshold value, the activation functions ar-;;-cres.iie(l"al:td the ourp_uc is calculated. The
activation function using lhreshold can be defined as -----

/(net)={_: if net "?-8

ifnet<8

where e ~ the fixed threshold value.

.L
 2.5 McCu!loch-Pitts Neuron 27
-r ~r
I
~
2.4.4 Learning Rate .__f o\ ,~'
' '"'
'
The learning rate is denoted by "a." It is used to ,co-9-uol the amounfofweighr adillStmegr ar each step of
~- The learning rate, ranging from 0 -to 1, 9'erer.ffi_iri.es the rate of learning at each time step.

I 2.4.5 Momentum Factor

Convergence is made faster if a momenrum factor is added to the weight updacion erocess. This is generally
done in the back propagation network. If momentum has to be used, the weights from one or more previous
uaining patterns must be saved. Momenru.nl helps the net in reasonably large we1ght adjustments until the
correct1ons are in lhe same general direction for several patterns.

I 2.4.6 Vigilance Parameter

I 2.4. 7 Notations

The-notations mentioned in this section have been used in this textbook for explaining each network.
x;: Activation of unit Xi, inp_uc signal.
y;: Activation of unit Yj, Jj = f(J;nj)
Wij: Weight on connection from unit X; ro unit Yj.
bj: Bias acting on unitj. Bias has a constant activation of 1.
W: Weight matrix, W = {wij}
Yinj= Net input to unit Yj given by Yinj = bj + L;XiWij
l!x\1: Norm of magnitude vector X.
Bj: Threshold for activation of neuron Yj-
S: Training input vector, S = (s 1 , , s;, ... , s11)
T: Training ourput vector, T = (tJ, ... , fj, .. , t 71 )
X: Input vector, X= (XI> , Xi> , x11)
D..wij: Change in weights given by 8.wij = Wij(new) - Wij(old)
a: Learning rate; it controls the amount of weight adjustment at each step of training.

I 2.5 McCulloch-Pitts Neuron

I 2.5.1 Theory

The McCulloch-Pitts neuron was the earliest neural network discovered in 1943. It is usually called as M-P
neuron. The M-P neurons are connected by directed weighted paths. It should be noted that the activation of
aM-P neuron is binary, that is, at any time step the neuron maY fire or may por 6re The weights associated
wilh the communication links may be excitatocy (weight is positive) or inhibioocy (weight is negative). All ilie

L
 /

28 Artificial Neural Network: An Introduction

excitatory connected weights entering into a particular neuron will have same weights. The threshold plays
a major role in M-P neuron: There is a fiXed threshold for each neuron, and if ilie net input to the neuron
is greater than the.threshold then ilie neuron fires. Also, it should be noted that any nonzero inhibitory
input would prevent the neuro,n from firing. The M-P neurons are most widely used in the case of logic
functiOn~.------------

I 2.5.2 Architecture

A simple M-P neuron is shown in Figure 2-18. As already discussed, the M-P neuron has both excitatory and
inhibitory connections. It is excitatory with weight (w > 0) or inhibitory with weight -p(p < 0). In Figure
2-18, inpms &om Xi ro Xn possess excitatory weighted connections and inputs from Xn+ 1 m Xn+m possess
inhibitory weighted interconnections. Since the firing of ilie output neuron is based upon the threshold, the
activation function here is defined as

f(y;,)=(l ify;,;?:-0
0 ify;n<8

For inhibition to be absolute, the threshold with the activation function should satisfy the following condition:

() > nw- p

The output wiH fire if it receives sa6:~~citatory in~~~~ut no inhibitory inputs, where
kw:>:O>(k-l)w
----
The M-P neuron has no particular training algorithm. An analysis has to be performed m determine the
values of the weights and the ili,reshold. Here the weights of the neuron are set along with the threshold to
make the neuron "perform a simple logic functiofk-Xhe-M J?. neurons are used as buildigs ~ocks on...which
we can model any funcrion or phenomenon, which can be represented as a logic furfction.

x,
~

~
'J
~
X,
-X,
-
~' 'y

xm,
-p:;??
~
Xm

Figure 218 McCulloch- Pins neuron model.

2.6 Linear Separabilily 29

I 2.6 Linear Separability

~ fu'l'N does not give an exact solution for a nonlinea;-. problem. However, it provides possible approximate
solutions nonlinear problems. Linear separability, is _ifie ~ritept wherein the separatiOn of the input space
into regions is ase on w e er e network respoilse isJositive or negative.
A decision line is drawn tO separate positive and negative responses. The decision line may also be called as
the decision-making line or decision-support line or linear-separable line. The necessity of the linear separability
concept was felt to classify the patterns based upon their output responses. Generally the net input @cU'Iau:a-
to t1te output Unu IS given as

"
Yin = b + z:x,w;
i=l

For example, if 4hlpolar srep acnvanoijfunction is used over the calculated ner input (y;,) then the value of
the funct:ion fs" 1 for a positive net input and -1 for a negative net input. Also, it is clear that there exists a
boundary between the regions where y;, > 0 andy;, < 0. This region may be called as decision boundary and
can be determined by the relation

"
b+ Lx;w;=O
l~l

On the basis of the number of input units in the network, the above equation may represenr a line, a plane
or a hyperplane. The linear separability of the nerwork is based on the decision-boundary line. If there exist
weights (with bias) for which the training input vectors having positive (correct:) response,+ l,lie on one side
of the decision boundary and all the other vectors having negative (incorrect) response, -1, lie on rhe other
side of the decision boundary. then we can conclude the/PrObleffi.Js "linearly separable."
Consider a single-layer network as shown in Figure 2-~ias irlduded. The net input for the ne[Work
shown in Figure 2-l9 is given as

y;,=h+xtwl +X21V2
The sepaming line for wh-ich the boundary lies between the values XJ and X'2 so that the net gives a positive
response on one side and negative response on other side, is given as
b+xtw1 +X2Ui2 = 0

x, X, w,
w,

Figure 219 A single-layer neural net.

30 Artificial Neural Network: An Introduction

If weight WJ. is not equal to 0 then we get

WI b
X2 = --Xl--
w, w,
Thus, the requirement for the'positive response of the net is

0t~l W\ + "2"'2 > '!)

During training process, lhe values of Wi> W2 and bare determined so that the net will produce a positive
(correct) response for the training data. if on the other hand, threshold value is being used, then the condmon-
for obtaining the positive response from ourpur unit is

Net input received> ()(threshOld) I

Yir~-> 8
XtW\ + XZW2 > (}

The separating line equation will then be

XtWJ +X2W2 =()

W\ 8
"'=--XI+- (with w, 'f' 0)
w, w,
During training process, the values of WJ and W2 have to be determined, so that the net will have a correct
response to the training data. For this correct response, the line passes close rhrough the origin. In certain
situations, even for correct response, the separating line does not pass through the origin.
Consider a network having positive response in the first quadram and negative response in all other
quadrants (AND function) with either binary or bipolar data, then the decision line is drawn separating the
positive response region from rhe negative response region. This is depicred in Figure 2-20.
Thus, based on the conditions discussed above, the equation of this decision line may be obtained.
Also, in all the networks rhat we would be discussing, the representation of data plays a major role.

X,

+
(Positive response region)

(Negalive response region)

-x, x,

Decision
line

-x,
Figure 220 Decision boundary line.

i
l
 2.7 Hebb Network 31

However, the dara representation mode has to be decide_d - whether it would be in binary form or in
bipolar form. It may be noted that the bipolar reoresenta'tion is bener than the
Using bipolar data

I
--
ues are represeru;d
vice-versa.

2. 7.1 Theory
can be represented by

1 2.7 H~bb Network (e-n (,j 19,., ":_ w1p--tl u,.,; t-)
<..J

I

For a neural net, the Hebb learning rule is a simple one. Let us understand it. Donald Hebb stated in 1949
that in the brain, the learning is performed by th c ange m e syna nc ebb explained it: "When an
axon of cell A is near enough to excite cdl B, an y or permanently takes pia~ it, some
growth process or merahgljc cheag;e rakes place in one or both the cells such that Ns efficiency, as one of the
cellS hrmg B. is increased.,
According to the Hebb rule, the weight vector is found to increase proportionately to the product of the
input and the learning signal. Here the learning signal is equal tO the neuron's output. In Hebb learning,
if two interconnected neurons are 'on' simu)taneously then the weights associated w1ih these neurons can
be increased by ilie modification made in their synapnc gap (strength). The weight update in Hebb rule is
given by

w;(new) = w;(old) + x;y

The Hebb rule is more suited for ~ data than binary data. If binary data is used, ilie above weight
updation formula cannot distinguish two conditions namely;
1. A training pair in which an input unir is "on" and target value is "off."
2. A training pair in which both ilie input unit and the target value are "off."
Thus, iliere are limitations in Hebb rule application over binary data. Hence, the represemation using bipolar
data is advanrageous.

I 2. 7.2 Flowchart of Training Algorithm

The training algorithm is used for rhe calculation and -~diustmem of weights. The flowchart for the training
algorithm ofHebb ne[Work is given in Figure 2-21. The notations used in the flowchart have already been
discussed in Section 2.4.7.
In Figure 2-21, s: t refers to each rraining input and target output pair. Till iliere exists a pair of training
input and target output, the training process takes place; elSe, IE tS stopped.

I 2. 7.3 Training Algorithm

The training algorithm ofHebb network is given below:

I Step 0: First initialize ilie weights. Basically in this network iliey may be se~ro zero, i.e., w; = 0 fori= 1 \
to n where "n" may be the total number of input neurons. '
Step 1: Steps 2-4 have to b~ performed for each input training vector and mger output pair, s: r.

i
l
 32 Artificial Neural Network: An Introduction

For
No
each
s: t

Yes

Activate input units

XI= Sl

Activate output units

y=t

Weight update
w1(new)= w1(old) +X1Y

Bias update
b(new)=b(old)+y

('
\ l8 '
~
tI " , , Figure 2~21 Flowchm ofHebb training algorithm.
,. S~~ 2: Input units acrivations are ser. Generally, the activation function of input layer is idemiry funcr.ion:
0- s; fori- tiiiJ
'c Step 3:., Output umts activations are set: y 1= t. i
Step 4: Weight adjustments and bias adjtdtments are performed:

wz{new) = w;(old} + x;y

b(new) = b(old) + y
2.9 Solved Problems 33

The above five steps complete the algorithmic process. In S~ep 4, rhe weight updarion formula can also be
given in vector form as

w(newl'= u,(old) +xy

Here the change in weight can be expressed as

D.w = xy
As a result,

w(new) = w(old) + l>.w

The Hebb rule can be used for pattern association, pattern categorization, parcem classification and over a
range of other areas.

I 2.8 Summary
In this chapter we have discussed dte basics of an ANN and its growth. A detailed comparison between
biological neuron and artificial neuron has been included to enable the reader understand dte basic difference
between them. An ANN is constructed with few basic building blocks. The building blocks are based on
dte models of artificial neurons and dte topology of few basic structures. Concepts of supervised learning,
unsupervised learning and reinforcement learning are briefly included in this chapter. Various activation
functions and different types oflayered connections are also considered here. The basic terminologies of ANN
are discussed with their typical values. A brief description on McCulloch-Pius neuron model is provided.
The concept of linear separability is discussed and illustrated with suitable examples. Derails are provided for
the effective training of a Hebb network.

I 2.9 Solved Problems

I. For the network shown in Figure I, calculate the weights are

net input to the output neuron.
[xi, x,, XJI = [0.3, 0.5, 0.6]
0.3 [wJ,w,,w,] = [0.2,0.1,-0.3]
X~
~ The net input can be calculated as
0.5 0.1
@ y Yin =X] WJ + X'2WZ + X3W3
__/"
= 0.3 X 0.2+0.5 X 0.1 + 0.6 X (-0.3)
-0.3 = 0,06 + 0.05-0,18 = -O.D7

2. Calculate the ner input for the network shown in

Figure 1 Neural net.
Figure 2 with bias included in the network.

Solution: The given neural net consists of three input Solution: The given net consistS of two input
neurons and one output neuron. The inputs and neurons, a bias and an output neuron. The inputs are
 Artificial Neural Network: An Introduction
34

The net input ro the omput neuron is

0.3
"
y;, = b+ Lx;w;
y i::l

(n = 3, because only
0.7 3 input neurons are given]
= b + XJ.Wt + X'2W2 + X3W3
= 0.35 + 0.8 X 0.1 + 0.6 X OJ
Figure 2 Simple neural net.
+ 0.4 X (-0.2)
[x1, X2l = [0.2, 0.6] and the weigh" are [w 1, w,] = = 0.35 + 0.08 + 0.18 - 0.08 = 0.53
[0.3, 0.7]. Since the bias is included b = 0.45 and
bias input xo is equal to 1, the net input is calcu- (i) For binary sigmoidal activation function,
lated as
1 1
y=f(y;.) = 1 + e_,m
- = l+e-053
= 0.625
Yin= b+xJWI +X2W2

= 0.45 + 0.2 X 0.3 + 0.6 X 0.7 (ii) For bipolar sigmoidal activation function,
= 0.45 + 0.06 + 0.42 = 0.93
- 2_ - 1 =
. - __ 2 - 1
y-f(y,.,)- 1 +0' 1 +e 0.53
Therefore y;, = 0.93 is the ner input.
= 0.259
3. Obtain rhe output of the neuron Y for the net-
work shown in Figure 3 using activation func- 4. Implement AND function using McCulloch-Fitts
tions as: (i) binary sigmoidal and (ii) bipolar neuron (cake binary daa).
sigmoidal. Solution: Consider the truth table for AND function
(Table 1).
1.0
Table 1
0.1 0.35 Xi X2 y
1 1
0.6 x,l o.3 ;r y 1 0 0
0 1 0
0 0 0
-0.2
0.4
x, In McCulloch-Pires neuron, only analysis is being
performed. Hence, assume che weights be WI = 1
Figure 3 Neural ner. and w1 = 1. The network architecture is shown in
Figure 4. Wiili chese assumed weights, che nee input
is calculated for foul inputs: For inputs
Solution: The given nerwork has three input neu-
rons with bias and one output neuron. These form (1,1), y;n=xiwt+X2wz=l x 1+1 xI =2
a single-layer network. The inpulS are given as (l,O), Yi11 =XJWJ +X2Wz = 1 X 1 +0 X 1= 1
[xi>X2X3] = [0.8,0.6,0.4] and the weigh<S are
[w 1, w,, w3] = [0.1, 0.3, -0.2] with bias b = 0.35
(Q, 1), Ji = XJ Wj +X2W2 = 1+ 1 X 1 = 1
0 X
(0,0), )'in =XIWl +X2W2 = 0 X 1 +OX 1 = 0
(irs input is always 1).
 35
2.9 Solved Problems

Table2
Xj X2
- y_
w1=1
0 0 0 @
y' ;: ~
0

0
~
~- The given function gives an ourputonlywhenxi = 1
andX2 ;:; 0. The weights have to bedecidedonlyafi:er
the analysis. The net Qn be represented as shown in
Figure 4 Neural net. < Figure 5. , ..X , 0 \ 0
tt>l>"':u I 'n
For an AND function, the output is high if both the
inputs are ~igh. For this condition, the net input is
w151
calculated as 2. Hence, based on ch.is net input, the
threshold is set, i.e. if the threshold value is greater
than or equal m 2 then the neuron fires, else it does y
nor fire. So the threshold value is set equal to2((J"= 2).
This can also be obained by
w2521

-\ "{ e?- nw- p

, .
~
/'} ,., ....... Figure 5 Neural net (weights fixed after analysis).
Here, ~ = 2, w = 1 (excitatory weights) and p = 0
(no inhibitory weights). Substituting these values in Case 1: Assume thac both weights W! and 'W'z. are
the above~rnencioned equation we get excitatory, i.e.,

WJ=W2=1
8~2xl-0=>8~2
Then for the four inputs calculace che net input using
Thus, the output of neuron Y can be written as .
' y;,=XIW] +l.11V1
... ]\
l ify,.?-2 "'; For inputs
y = f(y;,) = 0 if y;, < 2 j \ ..
\ (1, 1), Yin= 1 X 1 +l X1= 2
/ \""
0 (1, 0), Yin= 1 X 1+0 X I= 1
where "2" represents che threshold value. (0, 1), Yiu = 0 X 1+1 X 1= 1
..--- 5. lmplemem ANDNOT function using (0, 0), Yitl = 0 X 1+0 X 1= 0
McCulloch-Pirrs neuron (use binary data
representation). From the calculated net inputs, it is not possible co
fire ilie neuron for input (1, 0) only. Hence, t~ese J-.
Solution: In the case of ANDNOT funcrion, the weights are norsUirable. IJI-il'b'l
1\(Jp / . /
response is true if the first input is true and the Assume one weight as excitato\Y and the qther as --\-- rr
second input is fa1se. For all ocher input variations, inhibitory, i.e., ,.. ' l,t-) ...tit'
rhe response is fa1se. The truth cable for AND NOT
WI =1, wz=-1
function is given in Table 2.
 36 Artificial Neural Network: An Introduction

Now calculate the net input. For the inputs A single-layer net is not sufficient to represent the
function. An intermediate layer is necessary.
(1,1), y;, = 1 X 1 + 1 X -1 = 0 ~
(1,0), y;,=1x1+0x. -1=1'
(0,1), J;, = 0 X 1 + l X -1 = -1
(0, 0), Yin= 0 X 1 +0 X -1 = 0 ~ y)-.-y

From the calculated net inputs, now it is possible

co fire the neuron for input (1, 0) only by fixing a
threshold of 1, i.e.,()~ 1 for Y unit. Thus,
Figure 6 Neural net for XOR function (ilie
tl!i=:=l; 1112=-1; 6?:.1 weights
Nou: The value ~f() is caJ'?llared using the following: shown are obtained after analysis).
) .,
8?:. nw-:-p First function (zJ = XJ.Xi"): The rrut:h table for
function ZJ is shown in Table 4.
(}?:. 2 x 1- 1 ~.,[for "p" inhibitory only
~. ~'7 magnitude consitk'red] Table4
9?:.1 J~r,J X]
~. S\) "'- Zi

Thus, the output of neuron Y can be written as 0 0 0

0 1 0
1 ify;,::01 0 1
1
y=f(y;,)= 0 ify;,< 1 1 0

"6. lmplementXORfunction using McCulloch-Pitts The net representation is given as

neuron (consider binary data). Case 1: Assume both weighrs as excitatory, i.e.,

Solution: The trmh table for XOR function is given wu = 1021 = 1

in Table 3.
Calculate the net inpms. For inputs,
Table3
X]
"'- y (0, 0), Zj,0 = 0 X 1+ 0 X I= 0
0 0 0 (Q, 1), ZJin = 0 X 1+ l X 1= l
0 1 1 (1, 0), Z!i, = 1 X 1+ 0 X 1= 1
1 0 1 (1,1), ZJin = 1 X 1+1 X 1= 2
1 1 0
Hence, it is not possible to obtain function z1
In this case, ilie output is "ON" foronlyoddnumber using these weighlS.
ofl's. For rhe rest it is "OFF." XOR function cannot Case 2: Assume one weight as excitatory and the
be ;presented by simple and single logic function; it oilier as inhibitory, i.e.,
is represented as

\Lf::~
WB=l; U/21=-l

y=z, +za
,, 1
~

x, z,)(z,..,=x,w,,+JC2w2d
where
-1
Z! = Xlii (function 1) X,
Z2 = XJx:z (funccion 2) "'
y = zi(OR)z, (function 3) Figure 7 Neural net for Z 1.

,!
 2.9 Solved Problems 37
,, x, Case 2: Assume one weight as excitatory and the
-1 ot~er as inhibitory, i.e.,

w12=-1; wzz=l
Now calculate the net inputs. For the inputs
Figure 8 Neural ner for Z2.
(0, 0), Z2in :::: 0 X -1 +0 X 1= 0
Calculate the net inputs. For inputs 21'1-3.2-fr
(Q, 0), Zlin = 0 X 1+0 X -1 = Q "
(Q, 1), ZJin = QX 1 + 1 X -1 = -1 (1, 1), Z2j11 = 1 X

(1, 0), Z\in = 1 X } + 0 X -1 :=: 1

Thus, based on this
(1, 1), Ziin = 1 X 1 + 1 X -1 =0 possible to get the requi

On the basis of this calculated net input, it is

possible to get the required output. Hence, W22 =1
w11

WZI
e~ 1
== 1
= -1
for the zl neuron
f.{; r '',I SIV
---
8~1

Third function (;o. = ZJ OR zz): The truth rable

for this function is shown in Table 6.

~--------
Second function =
(zz XIX2.): The truth table for Table&
function Z2 is shown in Table 5. y zz
(
~ v Xi "'- Zi

TableS I) 0 0 0 0 0
0 1 1 0 1
Xi "'- zz 1 0 1 1 0
0 0 0 I 1 0 0 0
0 I 1
0 0 Here the net input is calculated using
1 0
~]in :::
The net representation is given as follows:
~e 1: Assume both weights as excitatory, i.e.,

w12 = wn = 1
- V] :::
Z] V]

VZ
+ Z2VZ
Case 1: Assume both weights as excitatory, i.e.,

= 1
)

Now calculate the net inputs. For the inputs Now calculate the net inp~t. For inputs

(O,O),Z2in=Ox 1+0x 1=0 (O,O),y;,=Ox 1+0x 1=0

(0, 1), ZJ..1; 1 =0 X 1 + 1 X 1 = 1 (0, 1), y;, = 0 X 1+ 1 X 1= 1
(l,Q),Z2in = 1 X 1 +0 X 1:::1 (1, 0), Ji, = 1 X 1 + 0 X 1= 1
(1, 1), zz;, = 1 X 1 + 1 X 1= 2 (1,1), y;, = 0 X 1+ 0 X 1= 0

Hence, it is not possible to obtain function zz (because for X] = 1 and X2 = l, ZJ = 0 and

using these weights. Z2 = 0)

l
38 Artificial Neural Network: An lntn:iduction

z, z,
,
(-1,1) /7 (1,1)
+ / +
y
-~x,, y,)
.(-1,0)
x,

Figure 9 Nemal ner for Y(Z1 ORZ,). (>,, y,) +

(-1, -1) (0,-1) (1, -1)

e
Swing a threshold -of 2::. 1' Vj == 1'2 = I, which
Function decision
boundary
implies that the net is recognized. Therefore, the
analysis is made for XOR function using M-P Figure 10 Graph for 'OR' function.
neurons. Thus for XOR function, the weights are
obtained as Using this value the equation for the line is given as

wu = Zll22 = 1 (excitatory) y = mx+c= (-1)x-l = -x-1

WJ2 = W21 = -1 (inhibitory) Here the quadrants are nm x andy but XJ and xz, so
VJ = Vz = 1 (excirarory) the above equation becomes

7. Using the linear separability concept, obtain the \ xz =-xi -1 (2.1)

response for OR function (rake bipolar inputs and
This can be wrinen as
bipolar targets).
-WI b
Solution: Table 7 is the truth table for OR function xz= --XI-'-- (2.2)
with bipolar inputs and targets. wz wz
Comparing Eqs. (2.1) and (2.2), we get
Table7
Xi X2 y Wi b
I 1 w2 =I;
-I 1 '"'
-I I I Therefore, WJ = l, wz = 1 and b =
1. Calculating
-I -I -I the net input and output of OR function on the basis
of these weights and bias, we get emries in Table 8.
The uurh table inpurs and corresponding outputs
TableS
have been plotted in Figure 10. If output is 1, it is
.------;-:=::;:=~----:.cl
denoted as"+" else"-." Assuming rbe ~res
as ( l, 0) 3.nd (0, ll; (x,, Yl) and (.xz,yz), the slope
~

1
X2
1
~[Y=b+~}D
"m" of the straight line can be obtained as I -1 1 1 1
-1 I I 1 1
)'2-yi -1-0 -1 -1 -1 -1 -1
m=--=--=-=-1
X2-X] 0+1 1
~
Thus, the output of neuron Y can be written as y,]
We now calculate c:

'= Ji- '"-"i = 0- (~1)(-1) = -1

y=f(;;;,) = 11OifJin<1
if y;,) I

I
_
 2.9 Solved Problems 39

where the threshold is taken as "I" (e = 1) based final (new) weights obtained by presenting the
on the calculated net input. Hence, using the linear first input paaern, i.e.,
separability concept, the response is obtained fo.r
"OR" function. [wi w, b] = [1 l 1]

8. Design a Hebb net to implement logical AND The weight change here is
function (use bipolar inputs and targets).
t:..w 1 =x1y= 1 X -1 = -1
Solution: The training data for the AND function is l>w, =xzy= -I X -I= I
given in Table 9.
l>b=y=-1
Table9
The new weights here are
Inputs Target
Xi X2 b y w1(new) = w1(old) + 6.w1 =I -1 = 0

1 1 1 1 w, (new) = w,(old) + l>w, = 1 + 1 = 2

1 -1 1 -1 b(new) = b(old) + l>b = 1- 1 = 0
-I 1 1 -1
-1 -I 1 -1 Similarly, by presenting the third and fourth
input patterns, the new weights can be calculated.
The neMork is trained using the Hebb network train- Table 10 shows the values of weights for all inputs.
ing algorithm discussed in Section 2.7 .3.lnitially the .
weights and bias are set to zero, i.e., .- ~-- ~~J--- Table 10
Inputs Weight changes Weights
0'2_~\ "'0 Xj X2, b y D.w, D.wz t:..b w1 wz b
(0 0 0)
First input [xi xz b] = [1 1 1] and target = 1 I 1
I I 1 1 1 I I
[i.e., y = 1]: Setting the initial weights as old I -1 2 0
I -1 I -I -I 0
weights and applying the Hebb rule, we get -1 -1 I I -1
-1 I I -1 1
-1 -1 I -1 1 1 -1 2 2 -2
w;(new) = w;(old) + x;y
w 1(new) = w1 (old) +Xi]= 0 + I x l= 1 The sepaming line equation is given by
w,(new) = w,(oid) + xzy = 0 + I x I = f
b(new) = b(old) +y = 0 + 1 = I
-WJ b
xz= - - x , - -
ruz wz
The weights calculated above arc the final weights
that are obtained after presenting the first input. For all inputs, use the final weights obtained
These weights are used as rhe initial weights when for each input to obtain the separating line.
the second input pattern is presented. The weight For the first input [1 I 1), the separating line is
change here is t:..w; = x;y. Hence weight changes given by
relating to the first input are
-1 1
XZ = - X i - - ::::} XZ = -XJ - 1
t:..w1 = XJJ = l x 1 = I 1 1
"'"" =w= 1 x 1 = 1 Similarly, for the second input [ 1 -1 1], the
l>b=y=l separating line is

Second input [x, X2, b] = [1 - 1 1] and -0

=
y -1: The initial or old weights here are the XZ = -x,
2
--02 => xz = 0

_L_
 T

40 Artificial Neural Network: An Introduction

X, Forthethirdinput[-lll],itis
(-1, 1)
I (1,1)
X2. =
-1
-x,
I
+-I ; ;:;} X2 = -x, + 1
' I

- x,
Finally, for the fourth input [ -1 - 1 1], the
separating line is

X2. = zxl
-2 2
+ 2 ::::} X2. = -.X"J +1
(1, -1)

r~
The graphs for each of these separating lines
obtained are shown in Figure 11. In this figure
(A) First Input "+" mark is used for output "1" and"-" mark
is used for output "-1." From Figure 11, it can
X, be noticed rhat. for the first input, the decision
boundary differentiates only the first and fourth
(-1,1)
I (1,1)

+
inputs, and nor all negative responses are separated
from positive responSes. When rhe second input
pattern is presented, the decision boundary sep
ar.ues (1, I) from (I, -I) and (-I. -I) and nor
~
(-1, I). But the boundary line is same for the both
- x,
third and fourth training pairs. And, the decision
boundary line obtained from these input training
pairs separates the positive response region from
the negative response region. H~
(-1, -1) (1, -1)
obtained from this are the final weigh!_afld-are

(B) Second input

-
given a:s

WI =2; tuz=2;
--

b=-2

X, The nerwork can be represented as shown in

(-1.~1
Figure 12.
(1,1)

'

~
-2

x1 x 2 _
1 y

(-1. -1) (1, -1) 2

(C) Third and fourth inputs

Figure 12 Hebb net for AND function.
Figure 11 Decision boundary for AND
function using Hebb rule for 9. Design a Hebb net to implement OR function
each training pair. (consider bipolar inputs and targets).

II
T;:

2.9 Solved Problems 41

Solution: The training pair for the OR function is ilie output response is "-1" lies on ilie other side -of
given in Table 11.
the bOun J, ~ . '-!. N;
Table 11 , I W] = 2; W, = 2; b = 2 ~}' v.J<" Q
Inputs Target ~~ nerwork can be represented as shown in L ...

b
-
y
Figure 14. J '>
"'
X]
X,. v' .y.:,J-"
I I I (i ~.w

-I
-I I I (-1,1) (1, 1) J,,.,;;:rl>
-I -1
I I
I
I
-I + + <ii ~',..
Initially the weights and bias are set to zero, i.e.,

Wj =w2=h=O x,

The nerwork is trained and the final weights are out

lined using the Hebb training algorithm discussed
in Section 2.7.3. The weighrs are considered as final (-1, -1) '
(1. -1)
weights if the boundary line obtained from these
weights separates the positive response region and .112"'-x, -1
negative response region.
By presenting all the input patterns, the weights
are calculated. Table 12 shows the weights calculated Figure 13 Decision boundary for OR function.
for all the inputS.

Table 12 2

Inputs Weight changes Weights

x1
x1 2 y
Xj Xz b J l).wl t.,wz t..b WI W1. b
(0 0 0) 2

-I -1 I 2 0 2
-I -1 I I I I 3 Figure 14 Hebb net for OR function.
-1 -( -1 I I -1 2 2 2
10. Use the Hebb rule method to implement XOR
function {take bipolar inputs and targets).
Using the final weights, the boundary line equation
can be obtained. The separating line equation is Solution: The training patterns for an XOR function
are shown in Table 13.
-wl b -2 2
X,= --X] - - = - X I - - =-X\ - 1 Table 13
wz wz2 2
__Inpu~ Target
The decision region for this net is shown in Figure 13.
It is observed in Figure 13 that straight li~e X']. =
b y
"'I
X]

-xl -1 separates the pattern space into rwo regions. I -I

Theinputparrerns [(I, I), (1, -I), (-1, I)] for which I -I I I
the output response is "1" lie on one side of the -I I I I
' boundary, and the input pattern (-1, -1) foi which -1 -1 I -I

I
42 Artificial Neural Network: An lnlraduc\ion
'";'.
_,__.

Here, a single-layer network with two input neurons, The XOR function can be made linearly separable by
one bias and one output neuron is considered. In solving it in a manner as discussedjn Problem 6. This
this case also, the initial weights are assumed to be method of solving will result in rwo decision bound-
zero: ary lines for separating positive and negative regions
ofXOR function.
WJ ==Wz='b=O 11. Using the Hebb rule, find the weights required to
perfotm the following classifications of the give it
By using the Hebb training algorithm, the network is input patterns shown in Figure 16. The pattern
uained and the final weights are calculated as shown is shown as 3 x 3 matrix form in the squares. The
in the following Table 14.
"+" sytpbols represent the value" 1" and empty
squares indicate "-1." Consider "I" belongs to
Table 14 the members of class {so has target value 1) and
lnputs Weight changes Weights "0" does not belong to the members of class
--- (so has target value -1).
Xj
"' b y l\w1 6.W]. !::J.b WI Wz b
(0 0 0)

ill ili
1 I -1 -1 -1 -1 -1 -1 -1 + +
1 -1 1 1 1 -1 1 0 -2 0 + +
-1 1 1 1 -1 1 I -1 -1 1
-1-11-1 1 1 -1 0 0 0 + + +

'I' o
The final weights obtained after presenting aH the Figure 16 Data for input patterns.
inpm pauerns do nm give correct output for all pat-
terns. Figure 15 shows that the input patterns are Solution: The training input patterns for the given
linearly non-separable. The graph shown in Figure 15 net (Figure 16) are indicated in Table 15.
indicates that the four input pairs that are present can-
not be divided by a single line m separate them into Table 15
two regions. Thus XORfi.mcrion is a case of a panern Pattern Inputs Target
classification problem, which is not linearly separable.
XI X2 X3 :Gj xs X6X7xaX9b y
1 1 -1 1 -1 1 1 I 1
X, 0 1 1 1 1 -1 1 1 I 1 1 -1
')i!-/
(-1, 1) (1,1)
\ ;,,_ Here a single-layer ne[Work with nine input netUons,
P one bias and one output neuron is formed. Set rhe
+
' initial weights and bias to zero, i.e.,
IN::::\
x, booodmy u'") W] ::=W2=W3=W<i=Ws
=wG =w-, =wa =llJ9 = b= 0
+
(-1, -1) (1, -1) Case 1: Presenting first input panern (I), we calculate
change in weights:
X,
f:..w;=x,y, i= 1 to9
Figure 15 Decision boundary for XOR function.
f:..w 1 = XIJ = 1 X 1= }
2.9 Solved Problems 43

[J,'"2 = X2J = 1 X 1 = 1 w,(new) = w,(old) + x,y = 1 + 1 x -1 = 0

l:J.w3 = X3Y = 1 X 1= 1 w4(n.W) = w,(old) + X4J = -1 + 1 x -1 = -2
l:J.w4 =xv= -1 x 1 = -1 u:s(new) = ws(old) +xsy =I+ -1 x -1 = 2
l:J.w5 = XSJ = 1 X 1 = l wG(new) = WG(old) +xGy = -1 + 1 X -1 = -2
l:J.wG =XGJ= -1 X l = -1 107(new) = 107(old) +XJy = 1 + 1 x -1 = 0
!J,107 = XJY = 1 X 1 = 1 W,(new) = w,(oJd) + XSJ = 1 + 1 X -1 = 0
l:J.wa=xsy=1xl=l
W<J(new) = w,(old) + x9y = 1 + 1 x -1 = 0
IJ,W<j =X<)y= 1 X 1= 1
b(new) = b(old) + y = 1 + 1 x -1 = 0
M=y= 1
The final weights after presenting rhc second input
pattern are given as
We now cakulate the new weights using the formula
W(newJ=[OOO -22 -20000]
w;(new) = wi(old) + l:J.wi The weights obtained are indicated in the Hebb net
shown in Figure 17,
Setting the old weights as the initial weights here,
we obrairt 12. Find the weights required to perform the follow-
ing classifications of given input patterns using
the Hebb rule. The inpurs are "1" where''+"
WJ (new) = WJ (old) + l:J.w1 = 0 + 1 = 1
symbol is present and" -1 ''where"," is presem.
'"2(new) = '"2(old) + !J,'"2 = 0 + 1 = 1 "L" pattern belongs to the class (target value+ 1)
w,(new) = w3(o!d) + IJ,w3 = 0 + 1 = 1 and "U" pattern does not belong to the class
(target value -1).
Similarly, calculating for other weights we get Solution: The training input patterns for Figure 18
are given in Table 16.
W4(new) = -1, ws(new) = l, WG(new):::: -1,
WJ(new) = 1, wa(new) = 1, llJ9(new) = 1, Table 16
b(new) = 1 Pattern Inputs Target
X3 X4 X) XG
X] X2. '-7XSX9 b J
The weights after presenting first input pattern are L 1-1-11-1-1

W(new) = [1 1 1 -1 1 -I 1 I 1 1] u -1 1 I -1 -1

Case 2: Now we present the second input pattern

A single-layer ne[Work with nine input neurons, one
(0). The initial weights used here are the final weights
bias and one output neuron is formed. Set the initial
obtained after presenting the fim input pa~ern. Here,
weights and bias tO zero, i.e.,
the weights are calculated as shown below (y = -1
wiclHheinitialweighrsbeing[1ll-11-ll1I1]). W]::=W2=W3=W<j:=W5
=wG=UJ?=wa=U19=b=O
w;(new) = w;(old) + l:J.x; [l:J.w; = x;y] The weights are calculated using
w,(new) = WJ(old) + XiJ =I+ 1 X -1 = 0
w;(new) = w;(old) + x;y
""(new)= '"2(old) + x,y = 1 + 1 x -1 = 0

I
44 Artificial Neural Network: An Introduction

x,

X,

,,

....

y
X,

"'
,,
,,

x9 1 (x9

Figure 17 Hebb ner for the data matrix shown in Figure 16.

' ' '

+ +
'

+ + + +
'
e u
Figure 18 Input clara for given parrerns.
2.9 Solved Problems v
45

The calculated weights are given in Table 17.

Table 17
Tatge' Weights

"' "'-
X3 X4 X5
Inpuu

X6X7XSX9b ----
j WJ
(0
Uf2 Ul3 W4 W5
0 0 0 0
W6 W'J
0 0
Wg

0
U19
0 0)
b

-1 -1 1 -1 -1
-1 -1 1 -1 -1 1 1 1 1, '
-I 0 0 -2 0 0 -2 0 0 0 0
-I I I -I

The obtained weights are indicated in rhe Hebb net

The final weights after preseming the rwo input
~hown in Figure 19.
patterns are
WlnewJ~[OO -200-200001

x,

,,
,,

....
y
,,

"'
,,

"" x,

-- ,(X,
Figure 19 flebb ne< of Figure 18.
46 Artificial Neural Network An Introduction

I 2.10 Review Questions

l. Define an artificial neural network. 15. What is the necessity of activation function?
2. Srate ilie properties of the processing element of 16. List the commonly used accivation functions.
an artificial neural network. 17. What is me impact of weight in an anifidal
3. How many signals can be sent by a neuron at a neural network?
particular rime instant? 18. What is the mher name for weight?
4. Draw a simple artificial neuron and discuss dte 19. Define bias and threshold.
calculation of net input.
20. What is a learning rate parameter?
5. What is the influence of a linear equation over
21. How does a momentum factor make faster
the net input calculation?
convergence of a network?
6. List the main components of ilie biological
22. State the role of vigilance parameter iE:l ART
neuron.
network.
7. Compare and contrast biological neuron and
artificial neuron. 23. Why is the McCu!loch-Pins neuron widely used
in logic functions?
8. Srate ilie characteristics of an artificial neural
network. 24. Indicate the difference between excitatory and
inhibitory weighted interconnections.
9. Discuss in derail ilie historical development of
artificial neural networks. 25. Define linear separability.
10. What are the basic models of an artificial neural 26. Justify- XOR function is nonlinearly separable
network? by a single decision boundary line.
11. Define net architecmre and give ilS classifica 27. How can the equation ofa straight line be formed
tlons. using linear separability?
12. Define learning. 28. In what ways is bipolar representation better rhan
binary representation?
13. Differentiate beP.veen supervised and unsuper-
vised learning. 29. Stare the uaining algorithm used for the Hebb
nerwork.
14. How is the critic information used in the learning
process? 30. Compare feedfonvard and feedback network.

I 2.11 Exercise Problems

1. For the neP.vork shown in Figure 20, calculate the net input to rhe output neuron.

0.2
-
~ 6
y

0.3 ~

Figure 20 Neural net.

2.12 Projects 47

2. Calculate the output of neuron Y for the net {b) Construct a recurrent network with four
shown in Figure 21. Use binary and bipolar input nodes, three hidden nodes and two output
sigmoidal activation functions. nodes that has feedback links from the hidden
layer to the input layer.
6:. l)singlinear separability oo~cept, obtain the
.0.9
response for NAND funccion.
7. Design a Hebb net to implement logical AND
0.7~ y
function with
(a) binary inputs and targets and
(b) binary inputs and bipolar targets.
8. Implement NOR function using Hebb net with
Figure 21 Neural net. {a) bipolar inputs and targets and
(b) bipolar inputs and binary targets.
3. Design neural networks wiili only one M-P
neuron that implements the three basic logic 9. Classify the input panerns shown in Figure 22
operations: using Hebb training algorithm.

(i) NOT (x.J; + + + + + +

"
(ii) OR (x,, X2h + + +
+ + + + + +
(iii) NAND (x" "2), where x1 and"2 E {0, 1].
+ + +
4. (a) Show that ilie derivative of unipolar sig- + + + + +
moidal function is 'A' 'E'
Target value + 1 -1

j'(x) =AJ(x)[1 - [(x)j

Figure 22 Inpur panern.
(b) Show that the derivative of bipolar sigmoidal
ftmcrion is
10. Using Hebb rule, find dte weighLS required ro
A perform following classifications. The vecrors
/' (x) = 2[1 +f(x)][1 - [(x)]
(1 -1 1 -1) and (111-1) belong to class (target
5. {a) Construct a feed-forward nerwork wirh five ,aJue+1);,eetors(-1-11l)and(11-1-l)
input nodes, three hidden nodes and four output do nor belong to class (target value -1). Also
nodes that has lateral inhibition structure in the using each of training xvecmrs as input, test the
output layer. response of net.

I 2.12 Projects

1. Write a program to classify ilie letters and numer- 2. Wtit$:.~~ira~ programs for implementing logic
als using Hebb learning rule. Take a pair of letters functions usin~cCulloch-Pitts neuron.
or numerals of your own. Also, after training 3. Write a computer program to train a Madaline to
the fl.erwork, test the response of ilie net using perform AND function, using MRI algorithm.
suitable activation function. Perform the clas-
4: Write a program for implementing BPN for
sification using bipolar data as well as binary
training a singlehiddenlayer back-propagation
dara.
48 Artificial Neural Network: An Introduction
'.;;
network with bipolar sigmoidal units (A= 1) ro
achieve the following [)YO-to-one mappings:
testing. The input-output data are obtained by
varying inpuc variables (xt,Xz) within [-1,+1]
.,
. ~!
~

y = 6sin(rrxt) + cos(rrx,) randomly. Also the output dara are normalized it

y = sin(nxt) cos(0.2Jr"2) within [-1, 1]. Apply training ro find proper ]
weights in the network. :f
Ser up rwo sets of data, each consisting of 10 :?
input-output pairs, one for training and oilier for

~
 Ot"fv!. 0v'J..oM

,
!
~

t
]
f
?.
Supervised Learning Network 3
~
Learning Objectives -----'''-----------------,
The basic networks in supervised learning. Adaline, Madaline, back~propagarion and
How the perceptron learning rule is better radial basis funcrion network.
rhan the Hebb rule. The various learning facrors used in BPN.
Original percepuon layer description. An overview of Ttme Delay, Function Link,
Delta rule with single output unit. Wavelet and Tree Neural Networks.

Architecture, flowchart, training algorithm Difference between back-propagation and

and resting algorithm for perceptron, RBF networks.

I 3.1 Introduction
The chapter covers major topics involving supervised learning networks and their associated single-layer
and multilayer feed-forward networks. The following topics have been discussed in derail- rh'e- perceptron
learning r'Ule for simple perceptrons, the delta rule (Widrow-Hoff rule) for Adaline and single-layer feed-
forward flC[\VOrks with continuous activation functions, and the back-propagation algorithm for multilayer
feed-forward necworks with cominuous activation functions. ln short, ali the feed-forward networks have
been explored.

I 3.2 Perceptron Networks

1 3.2.1 Theory
Percepuon networks come under single-layer feed-forward networks and are also called simple perceptrons.
As described in Table 2-2 (Evolution of Neural Networks) in Chapter 2, various cypes of perceptrons were
designed by Rosenblatt (1962) and Minsky-Papert (1969, 1988). However, a simple perceprron network was
discovered by Block in 1962.
The key points to be noted in a perccptron necwork are:

I. The perceptron network consists of three units, namely, sensory unit (input unit), associator unit (hidden
unit), response unit (output unit).
 50 SupeNised Learning Network

2. The sensory units are connected to associamr units with fixed weights having values 1, 0 or -l, which are
assigned at random. -
3. The binary activation function is used in sensory unit and associator unit.
4. The response unit has an'activarion of l, 0 or -1. The binary step wiili fixed threshold 9 is used as
activation for associator. The output signals hat are sem from the associator unit to the response unit are
only binary. - ---
5. TiiCQUt'put of the percepuon network is given by
c

{', r' '-

..., \.~
y = f(y,,)
:.I\
' <
.,cl. 1
where J(y;n) is activation function and is defmed as

" ~. tr
f(\- ,-. \~ ..
~
) ~ )\t
if J;n> 9
f(y;,) ={ if -9~y;11 56
'Z -1 if y;71 <-9

6. The perceptron learning rule is used in the weight updation between the associamr unit and the response
unit. For each training input, the net will calculate the response and it will Oetermine whelfier or not an
error has occurred.
7. The error calculation is based on the comparison of th~~~~~rgets with those of the ca1~t!!_~~ed
outputs.
8. The weights on the connections from the units that send the nonzero signal will get adjusted suitably.
9. The weights will be adjusted on the basis of the learning_rykjf an error has occurred for a particular
training patre_!Jl.,..i.e..,-

Wi{new) = Wj{old) + a tx1

b(new) = b(old) + at
If no error occurs, there is no weight updarion and hence the training process may be stopped. In the above
equations, the target value "t" is+ I or-land a is the learningrate.ln general, these learning rules begin with
an initial guess at rhe weight values and then successive adjusunents are made on the basis of the evaluation
of an ob~~ve function. Evenrually, the lear!Jillg rules reac~.a near~optimal or optimal solution in a finite __
number of steps. -------
APcrceprron nerwork with irs three units is shown in Figure 3~1. A shown in Figure 3~1. a sensory unir
can be a two-dimensional matrix of 400 photodetectors upon which a lighted picture with geometric black
and white pmern impinges. These detectors provide a bif!.~.{~) __:~r-~lgl.signal__if.f\1_~ i~.u.~und
co exceei~. certain value of threshold. Also, these detectors are conne ed randomly with the associator ullit.
The associator unit is found to conSISt of a set ofsubcircuits called atrtre predicates. The feature predicates are
(
hard-wired to detect the specific fearure of a pattern and are e "valent to the feature detectors. For a particular
fearure, each predicate is examined with a few or all of the ponses of the sensory unit. It can be found that
the results from the predicate units are also binary (0 1). The last unit, i.e. response unit, contains the
pattern~recognizers or perceptrons. The weights pr tin the input layers are all fixed, while the weights on
the response unit are trainable.
 ~
3.2 Perceptron Networks
51

Output
oor 1 Output Desired
II
Fixed _weight
t Oar 1 output
valUe ciN., 0, -1
at randorr\ .
\ 0 G) y,

9~
i . \.--- '

~~
' iX1
X X X
\i
;x,
G) G)
X
X I I \ Xn
&.,
Sensory unit
'~
<
1

sensor grid " /

\ .. ._ representing any-'
lJa~------ .
@ @ ry~
e,
Assoc1ator un~ . Response unit

Figure 31 Ori~erceprron network.

w~t fL 9-
b'"~ r>-"-j ::.Kq>
(l.. u,>J;>.-l? '<Y\
' ~ AJA I &J)
I 3.2.2 Perceptron Learning Rule '

In case of the percepuon learrling rule, the learning signal is the difference between esir.ed...and.actuaL...- ---,
~ponse of a neuron. The perceptron learning rule IS exp rune as o ows: j ~ f.:] (\ :._ PK- A-. )
Consider a finite "n" number of input training vectors, with their associated r;g~ ~ired) values x(n) {
and t{n), where "n" r~o N. The target is either+ 1 or -1. The ourput ''y" is obtained on the
basis of the net input calculated and activation function being applied over the net input.

y = f(y,,) = l~
-1
if J1i1 > (}
if-{} 5Jirl 58
if Jin < -{}
\r~~ ~r
I
-~~
~
'

The weight updacion in case of perceprron learning is as shown.

lfy ,P then
.,
X~ -~~. .
/I

w{new) = w{old) + a tx {a - learning rate)

else, we have
w(new) = w(old)
 ~I
l
52 Supervised Learning Network

Figure 3-2 Single classification perceptron network.

training patterns, and this learning takes place within a finite number of steps provided that the solution
exists."-

I 3.2.3 Architecture

In the original perceptron ne[Work, the output obtained from the associator unit is a binary vector, and hence
that output can be taken as input signal to the res onse unit and classificanon can be performed. Here only
the weights be[l.veen the associator unit and the output unit can be adjuste , an, t e we1ghrs between the
sensory _and associator units are faxed. As a result, the discussion of the network is limited. to a single portion.
Thus, the associator urut behaves like the input unit. A simple perceptron network architecrure is shown in
Figure 32. --~------
In Figure 3-2, there are n input neurons, 1 output neuron and a bias. The inpur-layer and output-
layer neurons are connected through a directed communication link, which is associated with weights. The
goal of the perceptron net is to classify theJ!w.w: pa~~tern as a member or not a member to a p~nicular
class. ~1 --.- -.. --- ......
~.J.';) clo...JJ<L [j 1f'fll r').\-~Oo-t" Cl....\ ~ ~Len sy (\fll-
1 3.2.4 Flowchart for Training Process

The flowchart for the perceprron nerwork training is shown in Figure 3-3. The nerwork has to be suitably
trained to obtain the response. The flowchan depicted here presents the flow of the training process.
As depicted in the flowchart, fim the basic initialization required for rhe training process is performed. .~
'
The entire loop of the training process continues unril the training input pair is presented to rhe network.
The training {weight updation) is done on the basis of the comparison between the calculated and desired
output. The loop is terminated if there is no change in weight.

3.2.5 Perceptron Training Algorithm for Single Output Classes

The percepuon algorithm can be used for either binary or bipolar input vectors, having bipolar targets,
threshold being fixed and variable bias. The algorithm discussed in rh1~ section is not particularly sensitive
to the initial values of the wei~fr or the value of the learning race. In the algorithm discussed below, initially
the inputs are assigned. Then e net input is calculated. The output of the network is obtained by app1ying
the. activation function over the calculated net input. On performing comparison over the calculated and
 3.2 Perceptron Networks 53

For
each No
s:t

Apply activation, obtain

Y= f(y,)

y!l~~~

Yes

w1(new) = W1{old)+ atx1 W1(new)= w1{old)

b{new) = b(old) +at b(new) = b(old)

If
Yes
weight
changes

No
Stop

Figure 33 Flowcha.n: for perceptron network with single ourput.

54 Supervised Learning Network

ilie desired output, the weight updation process is carried out. The entire neMork is trained based on the
mentioned stopping criterion. The algorithm of a percepuon network is as follows:

I StepO: Initi-alize ili~weights a~d th~bia~for ~ ~culation they can b-e set to zero). Also initialize the /
learning race a(O < a,;:= 1). For simplicity a is set to 1.
Step 1: Perform Steps 2-6 until the final stopping condition is false.
Step 2: Perform Steps 3-5 for each training pair indicated by s:t.
Step 3: The input layer containing input units is applied with identity activation functions:

x; =si

Step 4: Calculate the output of the nwvork. To do so, first obtain the net input:

"
Yin= b+ Lx;w;
i=I

where "n" is the number of input neurons in the input layer. Then apply activations over the net
input calculated to obmin the output:

~
ify,:n>B
y= f(y;.) = { if -8 S.y;, s.B
-I ify;n < -9
Step 5, Weight and bias adjustment: Compare ilie value of the actual (calculated) output and desired
(target) output.

Ify i' f, then

w;(new) = w;(old) + atx;
b(new) = b(old) + Of
else, we have
7Vi(new) = WJ(old}
b(new) = b(old)
Step 6: Train the nerwork until diere is no weight change. This is the stopping condition for the network.
If this condition is not met, then start again from Step 2.

The algorithm discussed above is not sensitive to the initial values of the weights or the value of the
learning rare.

3.2.6 Perceptron Training Algorithm for Multiple Output Classes

For multiple output classes, the perceptron training algorithm is as follows:

\ Step 0:-- Initialize the weights, biases and learning rare suitably. I
Step 1: Check for stopping c?ndirion; if it is false, perform Steps 2-6.
3.2 Parceptron Networks 55

Step 2: Perform Steps 3--5 for each bipolar or binary training vector pair s:t.
Step 3, Set activation (identity) of each input unit i = 1 ton:

x;;= ~{

Step 4, irst, the net input is calculated as i A

I ,_..... It: -.-' --.~ ',_ -~,
---- :::::::::J~ "'( ;-; J)'
~----

(,. Yinj = bj + Lx;wij

n
.1~'
r<t' V" p\ \ '' /
. u \'.
'
(}.C: \\ , :/ r
\ ~
i=l
. v '<.''
Then activations are applied over the net input to calculate the output response:

~
ify;11j > 9
Jj = f(y;.y) = { if-9 :S.Jinj :S.9 II
,,
-I ify;11j < -9

Step 5: Make adjustment in weights and bias for j = I to m and i = I to n.

If; # Jj then
Wij(new) = Wij(old) + CXfjXi
bj(new) = bj(old) + Ofj
else, we have \1
wij(new) = Wij(old) li'
~{new) = ~{old)

Step 6: Test for the stopping condition, i.e., if there is no change in weights then stop the training process,
1 else stan again from Step 2. 1
I.'

It em be noticed that after training, the net classifies each of the training vectors. The above algorithm is
I
i
suited for the architecture shown in Figure 3~4. ~
j
i
I
3.2. 7 Percept ron Network Testing Algorithm
~
~
It is best to test the network performance once the training process is complete. For efficient performance
of the network, it should be trained with more data. The testing algorithm (application procedure) is as
I
~
follows: ~!I
il\!
I Step 0: The initi~ weights to be used here are taken from the training algorithms (the final weights I
obtained.i:l.uring training).
Step 1: For each input vector X to be classified, perform Steps 2-3.
Step 2: Set activations of the input unit.
II
I:;i
I

.,,.
011
 r
56 Supervised Learriing Network
~-
~~
~,

x, 'x,

./~~ \~\J
/w,,
w,l
Xi
(x;)~ "/ ~ y 1:
~(s)--
YJ
I -----+- YJ

x, ( x,).::::___ _ _~
w -

Figure 34 Network archirecture for percepuon network for several output classes.

Step 3: Obrain the response of output unit.

Yin = L" x;w; / '

i=l

I if y;, > 8
Y = f(yhl) = { _o ~f ~e sy;, ~8 _,/'\
1 tfy111 <-8

Thus, the testing algorithm resLS the performance of nerwork.

The condition for separaring the response &om re~o is

WJXJ + tiJ2X]. + b> (}

_______
The condition for separating the resPonse from_...r~~o t~~ion of nega~ve
..
~--
is

WI X} + 'WJ.X]_ + b < -(}

The conditions- above are stated for a siilgie:f.i~p;;~~~ ~~~~;~k~ith rwo Input neurons and one output
neuron and one bias.
r
- 3.3 Adaptive Unear Neuron (Adaline) 57

,
~~3.3 Adaptive Linear Neuron (Adaline)
'
1
I 3.3.1 Theory ,

The unirs with linear activation function are called li~ear.~ts. A network ~ith a single linear unit is called
an Adaline (adaptive linear neuron). That is, in an Adaline, the input-output relationship is linear. Adaline
uses bipolar activation for its input signals and its target output. The weights be.cween the input and the
omput are adjustable. The bias in Adaline acts like an adjustable weighr, whose connection is from a unit
with activations being always 1. Ad.aline is a net which has only one output unit. The Adaline nerwork may
be trained using delta rule. The delta rule may afso be called as least mean square (LMS) rule or Widrow~Hoff
rule. This learning rule is found to minimize the mean~squared error between the activation and the target
value.

I 3.3.2 Delta Rule for Single Output Unit

The Widrow-Hoff rule is very similar to percepuon learning rule. However, rheir origins are different. The
perceptron learning rule originates from the Hebbian assumption while the delta rule is derived from the
gradienc~descem method (it can be generalized to more than one layer). Also, the perceptron learning rule
stops after a finite number ofleaming steps, but the gradient~descent approach concinues forever, converging
only asymptotically to the solution. The delta rule updates the weights between the connections so as w
minimize the difference between the net input ro the output unit and the target value. The major aim is to
minimize the error over all training parrerns. This is done by reducing the error for each pattern, one at a
rime.
The delta rule for adjusting rhe weight of ith pattern {i = 1 ro n) is

D.w; = a(t- y1,)x1

where D.w; is the weight change; a the learning rate; xthe vector of activation of input unit;y;, the net input
to output unit, i.e., Y Li=l
= x;w;; t rhe target output. The deha rule in case of several output units for
adjusting the weight from ith input unit to the jrh output unit (for each pattern) is

IJ.wij = a(t;- y;,,j)x;

I 3.3.3 Architeclure

As already stated, Adaline is a single~unir neuron, which receives input from several units and also from one
unit called bias. An Adaline inodel is shown in Figure 3~5. The basic Adaline model consists of trainable
weights. Inputs are either of the two values (+ 1 or -1) and the weights have signs (positive or negative).
Initially, random weights are assigned. The net input calculated is applied to a quantizer transfer function
(possibly activation function) that restOres the output to +1 or -1. The Adaline model compares the actual
output with the target output and on the basis of the training algorithm, the weights are adjusted.

I 3.3.4 Flowchart lor Training Process

The flowchan for the training process is shown in Figure 3~6. This gives a picrorial representation of the
network training. The conditions necessary for weight adjustments have co be checked carefully. The weights
and other required parameters are initialized. Then the net input is calculated, output is obtained and compared
with the desired output for calculation of error. On the basis of the error Factor, weights are adjusted.
58 Supervised Learning Network

Ym= I.A/W1 y
X, \
X2r j w2 ''-
_,..,

w"
Y1"
X"
X"

Adaptive
algorithm I e = t- Ym 1 Output error
generator +t
~... .. ................................. Learning supervisor

Figure 35 Adaline model.

I 3.3.5 Training Algorithm

The Adaline nerwork training algorithm is as follows:

.Step 0: Weights and bias are set to some random values bur not zero. Set the learning rate parameter ct.
Step 1: Perform Steps 2-6 when stopping condition is false.
Step 2: Perform Steps 3~5 for each bipolar training pair s:t.
Step 3: Set activations for input units i = I to n.

x;=s;

Seep 4: Calculate the net input to the output unit.

"
y;, = b+ Lx;w;
i=J

Step 5: Update the weights and bias fori= I ron:

w;(new) = w;(old) + a (t- Yin) x;

b(new) = b (old) + a (t- y,,)
Step 6: If the highest weight change rhat occurred during training is smaller than a specified toler-
ance ilien stop ilie uaining process, else continue. This is the rest for stopping condition of a
network.

The range of learning rate Can be be[Ween 0.1 and 1.0.

I

I
I

1._
 3.3 Adaptive Linear Neuron (Adaline) 59

Set initial values-weights

and bias, learrltrig-state
If b, a

For
each
No
s: t

Yes

Activate input layer units

X =s (i=1ton)
1 1

Weight updation
w;(new) = w1(old) + a(t- Y1n)Xi
b(new) = b(old) + a(r- Yinl

No If
E;=Es

Figure 36 Flowcharr for Adaline training process.

_
60 Supervised Learning Network

I 3.3.6 Testing Algorithm

Ic is essential to perform the resting of a network rhat has been trained. When training is completed, the
Adaline can be used ro classify input patterns. A step &merion is used to test the performance of the network.
The resting procedure for thC Adaline nerwc~k is as follows:

J Step 0: Initialize the weights. (The weights are obtained from ilie ttaining algorithm.) J
Step 1: Perform Steps 2-4 for each bipolar input vecror x.
Step 2: Set the activations of the input units to x.
Step 3: Calculate the net input to rhe output unit:

]in= b+ Lx;Wj
Step 4: Apply the activation funcrion over the net input calculated:

1 ify,"~o
y=
{ -1 ifJin<O

I 3.4 Multiple Adaptive Linear Neurons

I 3.4.1 Theory

The multiple adaptive linear neurons (Madaline) model consists of many Adalin~el with a single
output unit whose value is based on cerrain selection rules. 'It may use majOrity v(;re rule. On using this rule,
rhe output would have as answer eirher true or false. On the other hand, if AND rule is used, rhe output is
true if and only ifborh rhe inputs are true, and so on. The weights that are connected from the Adaline layer
to ilie Madaline layer are fixed, positive and possess equal values. The weighrs between rhe input layer and
the Adaline layer are adjusted during the training process. The Adaline and Madaline layer neurons have a
bias of excitation "l" connected to them. The uaining process for a Madaline system is similar ro that of an
Adaline.

I 3.4.2 Architectury>

A simple Madaline architecture is shown in Figure 3-7, which consists of"n" uniu of input layer, "m" units
ofAdaline layer and "1" unit of rhe Madaline layer. Each neuron in theAdaline and Madaline layers has a bias
of excitation 1. The Adaline layer is present between the input layer and the Madaline (output) layer; hence,
the Adaline layer can be considered a hidden layer. The use of the hidden layer gives the net computational
capability which is nor found in single-layer nets, but chis complicates rhe training process to some extent.
The Adaline and Madaline models can be applied effectively in communication systems of adaptive
equalizers and adaptive noise cancellation and other cancellation circuits.

I 3.4.3 Rowchart of Training Process

The flowchart of the traini[lg process of the Madaline network is shown in Figure 3-8. In case of training, the
weighu between the input layer and the hidden layer are adjusted, and the weights between the hidden layer
.~

3.4 Multiple Adaptive Linear Neurons 61

Figure 37 Archireaure of Madaline layer.

and the output layer are ftxed. The time raken for the training process in the Madaline network is very high
compared to that of the Adaline network.

I 3.4.4 Training Algorithm

In this training algorithm, only the weights between the hidden layer and rhe input layer are adjusted, and
the weighu for the output units are ftxed. The weights VI, 112, ... , Vm and the bias bo that enter into output
unit Yare determined so that the response of unit Yis 1. Thus, the weights entering Yunit may be taken as

Vi ;::::V2;::::;::::vm;::::!

and the bias can be taken as

bo;:::: ~
The activation for the Adaline (hidden) and Madaline (output) units is given by

{_
lifx~O
f(x) = 1 if x < 0

Step 0: Initialize the weighu. The weights entering the output unit are set as above. Set initial small
random values for Adaline weights. Also set initial learning rate a.
Step 1: When stopping condition is false, perform Steps 2-3.
Step 2: For each bipolar training pair s:t, perform Steps 3-7.
Step 3: Activate input layer units. Fori;:::: 1 to n,

x;:;: s;

Step 4: Calculate net input to each hidden Adaline unit:

"
Zinj:;:bj+ LxiWij, j:;: l tom
i=l
62 Supervised Learning Network

(
p
Initial & fixed weights
& bias between hidden &
output layers

T
Set small random value
weights for adallne layer.
Initialize a

c}----~

No
>--+---{8

Activate input units

X10': s,, b1 ton

j
Find net input to hidden layer

...
Zn~=b1 +tx1 w~,j=l tom

I
Calculate output
zJ= f(z.,)

I
Calculater net input to output unit
Y..,=b0 ;i:zyJ
,.,
T
Calculate output
Y= l(y,)

cb
Figure 38 Flowcharr for rraining ofMadaline,
I
L
 3.4 Multiple Adaptive Linear Neurons 63

u Yes
t=y

t= 1" No

Yes

Update weights on unit z1whose

net input is closest to zero.
b1(new) = b1(old) + a(1-z~)
w,(new) = wi(old) + a(1-zoy)X1

Update weights on units zk which

has positive net inpul.
bk(new) = bN(old) + a(t-z.,.,)
wilr(new) = w,.(old) + a(l-z.)x1

If no
No ( weight changes
c) (or) specilied
' number of
epochs

' / (8
Yes '

Figure 38 (Continued).

L
 -
Supe!Vised Learning Network
64

Step 5: Calculate output of each hidden unit:

Zj = /(z;n)

Step 6: Find the output of the net:

y;, = bo + Lqvj
"'
j=l

y =f(y;")

Step 7: Calculate the error and update ilie weighcs.

1. If t = y, no weight updation is required.

2. If t f y and t = +1, update weights on Zj, where net input is closest to 0 (zero):
bj(new) = bj(old) + a (1 - z;11j}
wij(new) = W;i(old) + a (1 - z;11j)x;

3. If t f y and t = -1, update weights on units Zk whose net input is positive:

w;k(new) = w;k(old) + a (-1 - z;, k) x;

1

b,(new) = b,(old) +a (-1- z;,.,)

Step 8: Test for the stopping condition. (If there is no weight change or weight reaches a satisFactory level,
or if a specifted maximum number of iterations of weight updarion have been performed then
1 stop, or else continue). I

Madalines can be formed with the weights on the output unit set to perform some logic functions. If there
are only t\VO hidden units presenr, or if there are more than two hidden units, then rhe "majoriry vote rule"
function may be used. /

I 3.5 BackPropagation Network

1 3.5.1 Theory
The back~propagarion learning algorithm is one of the most important developments in neural net\vorks
(Bryson and Ho, 1969; Werbos, 1974; Lecun, 1985; Parker, 1985; Rumelhan, 1986). This network has re-
awakened the scientific and engineering community to the model in and rocessin of nu
phenomena usin ne networks. This learning algori m IS a lied !tilayer feed-forward ne_two_d~
con;rung o processing elemen~S with continuous renua e activation functions. e networks associated
with back-propagation learning algorithm are so e ac -propagation networ. (BPNs). For a given set
of training input-output pair, chis algorithm provides a procedure for changing the weights in a BPN to
classify the given input patterns correctly. The basic concept for this weight update algorithm is simply the
gradient-des em method as used in the case of sim le crce uon networks with differentiable units. This is a
method where the error is propagated ack to the hidden unit. he aim o t e neur networ IS w train the
net to achieve a balance between the net's ability to respond (memorization) and irs ability to give reason~e
responses to rhe inpm mar "simi,.,. bur not identi/to me one mar is used in ttaining (generalization). I
 3.5 BackPropagation Network 65

The back-propagation algorithm is different from mher networks in respect to the process by whic
weights are calculated during the learning period of the ne[INork. The general difficulty with the multilayer
pe'rceprrons is calculating the weights of the hidden layers in an efficient way that would result in a very small
or zero output error. When the hidden layers are incteas'ed the network training becomes more complex. To
update weights, the error must be calculated. The error, Which is the difference between the actual (calculated)
and the desired (target) output, is easily measured at the"Output layer. It should be noted that at the hidden
layers, there is no direct information of the en'or. Therefore, other techniques should be used to calculate an
error at the hidden layer, which will cause minimization of the output error, and this is the ultimate goal.
The training of the BPN is done in three stages - the feed-forward of rhe input training pattern, the
calculation and back-propagation of the error, and updation of weights. The tescin of the BPN involves the
compuration of feed-forward phase onlx.,There can be more than one hi en ayer (more beneficial) bur one
hidden layer is sufhcienr. Even though the training is very slow, once the network is trained it can produce
its outputs very rapidly.

I 3.5.2 Architecture

A back-propagation neural network is a multilayer, feed~forv.rard neural network consisting of an input layer,
a hidden layer and an output layer. The neurons present in che hidden and output layers have biases, which
are rhe connections from the units whose activation is always 1. The bias terms also acts as weights. Figure 3-9
shows the architecture of a BPN, depicting only the direction of information Aow for the feed~forward phase.
During the b~R3=l)3tion phase of learnms., si nals are sent in the reverse direction
The inputs sent to the BPN and the output obtained from the net could be e1ther binary (0, I) or
bipolar (-1, + 1). The activation function could be any function which increases monotonically and is also
differentiable.

...>,
:fu I\"..L.'"''
,-- J f.

~ I
~
-
(""~-~

I'

r(~.
''
~~ure39

l
Architecture of a back-propagation network.
66 Super.<ise_d Learni~g Network

I 3.5.3 Flowchart for Training Process

The flowchart for rhe training process using a BPN is shown in Figure 3-10. The terminologies used in the
flowchart and in the uaining algorithm are as follows:
x = input training vecro.r (XJ, ... , x;, ... , x11 )
t = target output vector (t), ... , t/r, ... , tm) -
a = learning rate parameter
x; :;::. input unit i. (Since rhe input layer uses identity activation function, the input and output signals
here are same.)
VOj = bias on jdi hidd~n unit
wok = bias on kch output unit
~=hidden unirj. The net inpUt to Zj is

"
Zinj = llOj +I: XjVij
i=l

and rhe output is

Zj = f(zi"j)

Jk = output unit k. The net input m Yk is

p

]ink = Wok + L ZjWjk

j=:l

and rhe output is

y; = f(y,";)

Ok =. error correction weight adjusrmen~. for Wtk ~hat is due tO an error at output unit Yk which is
back-propagared m the hidden uni[S thai feed into u~
Of = error correction weight adjustment for Vij that is due m the back-proEagation of error to the
hidden uni<zj- b> '\f"-( L""'-'iJ ~-fe_,l.. ,,'-'.fJ Z-J' ...--
Also, ir should be noted that tOe commonly used acrivarion functions are l:imary sigmoidal and bipolar
sigmoidal activation functions (discussed in Section 2.3.3). These functions are used in the BPN because of
the following characteristics: (i) continui~; (ii) djffereorjahilit:ytlm) nQndeCreasing mon09.11Y
The range of binary sigmoid is fio;Q to 1, and for bipolar sigmoid it is from -1 to+ 1.

I 3.5.4 Training Algorilhm

The error back-propagation learning algorithm can be oudined in ilie following algorithm:

!Step 0: Initialize weights and learning rate (take some small random values).
Step 1: Perform Sreps 2-9 when stopping condition is false.
Step 2: Perform Steps 3-8 for~ traini~~r.

I
L
 3.5 BackPropagalion Network 67

"
FOr each No
training pair >-~----(B
x. t

Yes

Receive Input signal x1 &

transmit to hidden unit

In hidden unit, calculate o/p,

"
Z;nj::: Voj + i~/iVij

z;=f(Z;nj), ]=1top
i= 1\o n

Calculate output signal from

output layer,
p
Yink =- Wok+ :E z,wik
"'
Yk = f(Yink), k =1 tom

Figure 310

L
 Supervised learning Network
68

Compute error correction !actor

t,= (1,-yJ f'!Y~o.l
(between output and hidden)

Find weight & bias correction term

ll.Wjk. = aO,zj> l\W01c = ~J"II

Calculate error term bi

(between hidden and input)
m
~nJ=f}kWjk
~ = 0,,1f'(z1,p

Compute change in weights & bias based

on bj.l!.vii= aqx;. ll.v01 = aq

I Update weight and bias on

output unit
w111 (new) = w111 (old) + O.w_;11
I'
wok (new)= w0k (old)+ ll.w011
\

Update weight and bias on

hidden unil
v 11 (new) =V~(old) +I.Nq
V01 (new)= V01 (old) + t:N01

I
3.5 BackPropagation Network 69
_, - ------------._
lf:edjorward p~as' (Phas:fJ_I

--
Step 3: Each input unit receives input signal x; and sends it to the hidden unit (i
Step 4: Each hidden unit Zj(j = 1 top) sums irs Weighted inp~;~t signals to calculate net input:
..:/

Zfnf' =
-
v;j + LX
"
ill;;
I
-v
Y. '..,
= l to n}.

,I

'rJ
i=l

Calculate output of the hidden uilit by applying its activation functions over Zinj (binary or bipolar

-
sigmoidal activation function}:

Zj = /(z;,j)
and send the output signal from the hidden unit to the input of output layer units.
Step 5: For each output unity,~o (k = I to m),_ca.lcuhue the net input: ,I
,\--. o\\
' I
p
Yink = Wok + L ZjWjk
j~l

-----:::~
......
f~ropagation ofen-or (Phase ll)j
St:ql-6: --Each output unu JJr(k
-
and apply the activation function to compute output signal

Yk = f(y;,,)

I to m) receives a target parrern corr~ponding to rhe input training

pattern and computes theferrorcorrectionJffii'C)

= (t,- ykl/'(y;,,)
The derivative J'(y;11k) can be calculated as in Section 2.3.3. On the basis of the calculated error
correction term, update ilie change in weights and bias:
\,
t1wjk = cxOkzj; t1wok = cxOrr {j
Of
Also, send Ok to the hidden layer baCkwards. rJ
Step 7: Each hidden unit (zj,j = I top) sums its delta inputs from the output units:

"'
8inj= z=okwpr
k=l

The term 8inj gets multiplied wirh ilie derivative of j(Zinj) to calculate the error tetm:

8j=8;11jj'(z;nj)

The derivative /'(z;71j) can be calculated as C!TS:cllssed in Section 2.3.3 depending on whether
binary or bipolar sigmoidal function is used. On the basis of the calculated 8j, update rhe change
in weights and bias:

t1vij = cx8jx;; tlvoj = aOj

,.
\.
 -I

70 Supervised Learning Network

. Wlighr and bias upddtion (PhaJ~ Ill): I

Step 8: Each output unit (yk, k = 1 tom) updates the bias and weights:
I
Wjk(new) = Wjk(old)+6.wjk I
= WQk(oJd)+L'.WQk '
WOk(new)
i
Each hidden unit (z;,j = 1 top) updates its bias and weights: I
Vij(new) = Vij(o!d)+6.vij
'<y(new) = VOj(old)+t.voj

Step 9: Check for the sropping condition. The stopping condition may be cenain number of epochs
1 reached or when ilie actual omput equals the t<Uget output. 1

The above algorithm uses the incremental approach for updarion of weights, i.e., the weights are being
changed immediately after a training pattern is presented. There is another way of training called batch-mode
training, where the weights are changed only after all the training patterns are presented. The effectiveness of
rwo approaches depends on the problem, but batch-mode training requires additional local storage for each
connection to maintain the immediate weight changes. When a BPN is used as a classifier, it is equivalent to
the optimal Bayesian discriminant function for asymptOtically large sets of statistically independent training
pauerns.
The problem in this case is whether the back-propagation learning algorithm can always converge and find
proper weights for network even after enough learning. It will converge since it implements a gradient-descent
on the error surface in the weight space, and this will roll down the error surface to the nearest minimum error
and will stop. This becomes true only when the relation existing between rhe input and the output training
patterns is deterministic and rhe error surface is deterministic. This is nm the case in real world because the
produced square-error surfaces are always at random. This is the stochastic nature of the back-propagation
algorithm, which is purely based on the srochastic gradient-descent method. The BPN is a special case of
stochastic approximation.
If rhe BPN algorithm converges at all, then it may get smck with local minima and may be unable to
find satisfactory solutions. The randomness of the algorithm helps it to get out of local minima. The error
functions may have large number of global minima because of permutations of weights that keep the network
input-output function unchanged. This"6.uses the error surfaces to have numerous troughs.

3.5.5 Learning Factors _of Back-Propagation Network

The training of a BPN is based on the choice of various parameters. Also, the convergence of the BPN is
based on some important learning factors such as rhe initial weights, the learning rare, the updation rule,
the size and nature of the training set, and the architecture (number of layers and number of neurons per
layer).

3.5.5.1 Initial Weights

The ultimate solution may be affected by the initial weights of a multilayer feed-forward nerwork. They are
initialized at small random values. The choice of r wei t determines how fast the network converges. I
The initial weights cannm be very high because t q~g-~oidal acriva ed here may get samrated I

L
 '
3.5 Back-Propagation Network 71 :IIf
from the beginning itself and the system may be smck at a local minima or at a very flat plateau at the starting

point itself. One method of choosing the weigh~ is choosing it in the range

-3' 3 J.
[ .fO;' _;a,'

V,j'(new) =y Vij(old)
llvj(old)ll

where Vj is the average weight calculated for all values of i, and the scale factory= 0.7(P) 11n ("n" is the
number of input neurons and "P" is the nwnber of hidden neurons).

3.5.5.2 Learning Rate a

The learning rate (a) affects the convergence of the BPN. A larger value of a may speed up the convergence
but might result in overshooting, while a smaller value of a has vice-versa effecr. The range of a from 10- 3
to 10 has been used successfulfy for several back-propagation algorithmic experiments. Thus, a large learning I
rate leads to rapid learning bm there is oscillation of wei_g!lts, while the lower learning rare leads to slower
learning. -

3.5.5.3 Momentum Factor

The gradient descent is very slow if the learning rare a is small and oscillates widely if a is roo large. One
very efficient and commonly used method that altows a larger learning rate without oscillations is by adding
a momentum factor ro rhc;_.!,LQ!DlaLgradient-descen_t __m~_r]l_Qq., _
The-iil"Omemum E'cror IS denoted by 1] E [0, i] and the value of 0.9 is often used for the momentum
factor. Also, this approach is more useful when some training data are ve rem from the maoriry
of clara. A momentum factor can be used with either p uern y pattern up atillg or batch-"iiii e up a -
ing.-I'iicase of batch mode, it has the effect of complete averagirig over rhe patterns. Even though the
averaging is only partial in the panern-by-pattern mode, it leaves some useful i-nformation for weight
updation.
The weight updation formulas used here are

Wjk(t+ I)= Wji(t) + ao,Zj+ry [Wjk(t)- Wjk(t- I)]

ll.uj~(r+ 1)

and

Vij(t+ 1) = Vij(t) + a8jXi+1J{Vij(t)- Vij(t- l)]

ll.v;j(r+ l)

The momenlum factor also helps in fas"r convergence.

L
 '.
72 Supervised Learning Network

3.5.5.4 Generalization
The best network for generalization is BPN. A network is said robe generalized when it sensibly imerpolates
with input networks thai: are new to the nerwork. When there are many trainable parameters for the given
amount of training dam, the network learns well bm does not generalize well. This is usually called overfitting
or overtraining. One solurion to this problem is to moniror the error on the rest sec and terminate the training
when che error increases. With small number of trainable parameters, ~e network fails to learn the training
_r!-'' ~.,_,r; data and performs very poorly. on the .test data. For improving rhe abi\icy of the network ro generalize from
.-.!( ~o_ a training data set w a rest clara set, ir is desirable to make small changes in rhe iripur space of a panern,
}{i 1
.,'e,) without changing the output components. This is achieved by introducing variations in the in pur space of
c..!( '!f.!' training panerns as pan of the training set. However, computationally, this method is very expensive. Also,
,-. ,:'\ j a net With large number of nodes is capable of membfizing the training set at the cost of generali:zation ...As a
?\ Ji result, smaller nets are preferred than larger ones.
r
3.5.5.5 Number of Training Data
The training clara should be sufficient and proper. There exisrs a rule of thumb, which states !!!:r rhe training
dat:uhould cover the entire expected input space, and while training, training-vector pairs should be selected
randomly from the set. Assume that theffiput space as being linearly separable into "L" disjoint regions
with their boundaries being part of hyper planes. Let "T" be the lower bound on the ~umber~ of training
pens. Then, choosing T suE!!_ that TIL ) will allow the network w discriminate pauern classes using
fine piecewise hyperplane parririomng. Also in some cases, scaling.ornot;!:flalization has to be done to help
learning. __ ,' : }) \ ..

3.5.5.6 Number of Hidden Layer Nodes .. A/77 _/

If there exists more than one hidden layer in a BPN, rhe~~ICufarions
performed for a single layer are
repeated for all the layers and are summed up at rhe end. In case of"all mufnlayer feed-forward networks,
rhe size of a h1dden layer i'f"VeTy important. The number of hidden units required for an application needs
to be determined separately. The size of a hidden lay~_:___is usually determi_~Q~~p_qim~~- For a network
of a reasonable size,~ SIZe of hidden nod -- araariVel}r~mall fraction of the inpllrl~For
example, if the network does not converge to a solution, it may need mor hidduJ lmdes:-i3~and,

overa.ll system performance.

3.5.6 Testing Algorithm of Back-Propagation Network

---
if rhe net\vork converges, the user may try a very few hidden nodes and then settle finally on a size based on

The resting procedure of the BPN is as follows:

Step 0: Initialize the weights. The weights are taken from the training algorithm.
Step 1: Perform Steps 2-4 for each input vector.
Step 2: Set the activation of input unit for x; (i = I ro n).
Step 3: Calculate the net input to hidden unit x and irs output-. For j = 1 ro p,
"
Zinj = VOj + L XiVij
i:=l

Z; = f(z;n;)
3.6 Radiat Basis Function Network 73

Step 4: Now c?mpure the output of the output layer unit. Fork= I tom,
p
link =:WOk + L ZjWjk
. j=l

Jk = f(yj,,)
Use sigmoidal activation functions for calculating the output.

-0
I 3.6 Radial Basis Function Network

I 3.6.1 Theory

The radial basis function (RBF) is a classification and functional approximation neural network developed
by M.J.D. Powell. The newark uses the most common nonlineariries such as sigmoidal and Gaussian kernel
functions. The Gaussian functions are also used in regularization networks. The response of such a function is
positive for all values ofy; rhe response decreases to 0 as lyl _. 0. The Gaussian function is generally defined as

f(y) = ,-1
The derivative of this function is given by

['(yl = -zy,-r' = -2yf(yl

The graphical represemarion of this Gaussian Function is shown in Figure 3-11 below.
When rhe Gaussian potemial functions are being used, each node is found to produce an idemical outpm
for inputs existing wirhin the fixed radial disrance from rhe center of the kernel, they are found m be radically
symmerric, and hence the name radial basis function network. The emire network forms a linear combination
of the nonlinear basis function.

f(y)

~----~~--r---L-~--~r-----~Y
-2 -1 0 2

Figure 311 Gaussian kernel fimcrion.

74 Supervised Learning Network

x,

X,

x,
Input Hidden Output
layer layer (RBF) layer

Figure 312 Architecture ofRBE

I 3.6.2 Architecture

The archirecmre for the radial basis function network (RBFN) is shown in Figure 3-12. The architecture
consim of two layers whose output nodes form a linear combination of the kernel (or basis) functions
computed by means of the RBF nodes or hidden layer nodes. The basis function (nonlinearicy) in the hidden
layer produces a significant nonzero response w the input stimulus it has received only when the input of it
falls within a smallloca.lized region of the input space. This network can also be called as localized receptive
field network.

I 3.6.3 Flowchart for Training Process

The flowchart for rhe training process of the RBF is shown in Figure 3-13 below. In this case, the cemer of
the RBF functions has to be chosen and hence, based on all parameters, the output of network is calculated.

I 3.6.4 Training Algorithm

The training algorithm describes in derail ali rhe calculations involved in the training process depicted in rhe
flowchart. The training is starred in the hidden layer with an unsupervised learning algorithm. The training is
continued in the output layer with a supervised learning algorithm. Simultaneously, we can apply supervised
learning algorithm to ilie hidden and output layers for fme-runing of the network. The training algorithm is
given as follows.

I Ste~ 0: Set the weights to small random values.

Step 1: Perform Steps 2-8 when the stopping condition is false.
Step 2: Perform Steps 3-7 for each input.
Step 3: Each input unir .(x; for all i ::= 1 ron) receives inpm signals and transmits to rhe next hidden layer
unit.
3.6 Radial Basis Function Network 75

For "'- No
each >--

Select centers of RBF functions;

sufficient number has to be
selected to ensure adequate sampling

If no
'epochs (or)
no
No weight
hange

Yes f+------------'

Figure 3-13 Flowchart for the training process ofRBF.

76 Supervised Learning Network

Step 4: Calculate the radial basis function.

Step 5: Select the cemers for che radial basis function. The cenrers are selected from rhe set of input
vea:ors. It should be noted that a sufficient number of centen; have m be selected to ensure
adequate sampli~g of the input vecmr space.
Step 6: Calculate the output from the hidden layer unit:

exp [- t,rxji- Xji)']

v;(x;) = J-
a2
'
where Xj; is the center of the RBF unit for input variables; a; the width of ith RBF unit; xp rhe
jth variable of input panern.
Step 7: Calculate the output of the neural network:

Y11n = L W;mv;(x;) + wo
i=l

where k is the number of hidden layer nodes (RBF funcrion);y,m the output value of mrh node in
output layer for the nth incoming panern; Wim rhe weight between irh RBF unit and mrh ourpur
node; wo the biasing term at nrh output node.
Step 8: Calculate the error and test for the stopping condition. The stopping condition may be number
of epochs or ro a certain ex:renr weight change.

Thus, a network can be trained using RBFN.

I 3.7 Time Delay Neural Network

The neural network has to respond to a sequence of patterns. Here the network is required to produce a
particular ourpur sequence in response to a particular sequence of inputs. A shift register can be wnsidered
as a tapped delay line. Consider a case of a multilayer perceptron where the tapped outputs of rhe delay line
are applied to its inputs. This rype of network constitutes a time delay Jlfurtzlnerwork (TONN}. The ourpm
consists of a finite temporal dependence on irs inpms, given a~

U(t) = F[x(t),x(t-1), ... ,x(t- n)]

where Fis any nonlinearity function. The multilayer perceptron with delay line is shown in Figure 3-14.
When the function U(t) is a weigh red sum, then the TDNN is equivalent to a finite impulse response
filter (FIR). In TDNN, when the output is being fed back through a unit delay into rhe input layer, then the
net computed here is equivalent to an infinite impulse response (IIR) filter. Figure 3-15 shows TDNN with
output feedback.
Thus, a neuron with a tapped delay line is called a TDNN unit, and a network which consists ofTDNN
units is called a TDNN. A specific application ofTDNNs is speech recognition. The TDNN can be trained
using the back-propagation-learning rule with a momentum factor.
3.8 Functional Link Networks 77

X( I)
Delay line
l
X( I) !<(1-D X( I-n)

-
Multllayar perceptron
r
T
0(1)

Figure 314 Time delay neural network (FIR fiher).

X(!) X( I-n)

Multilayer perceptron z-1

0(1)

Figure 315 TDNN wirh ompur feedback (IIR filter).

I 3.8 Functional Link Networks

-
These networks are specifically designed for handling linearly non-separable problems using appropriate
input representacion. Thus, suitable enhanced representation of the inpm data has to be found out. This
can be achieved by increasing the dimensions of the input space. The input data which is expanded is
used for training instead of the actual input data. In this case, higher order input terms are chosen so that
they are linearly independent of the original pattern components. Thus, the input representation has been
enhanced and linear separability can be achieved in the extended space. One of the functional link model
networks is shown in Figure 316. This model is helpful for learning continuous functions. For this model,
the higher-order input terms are obtained using the onhogonal basis functions such as sinTCX, cos JrX, sin 2TCX,
cos 2;rtr, etc.
The most common example oflinear nonseparabilicy is XOR problem. The functional link networks help
in solving this problem. The inputs now are

x:z t
"'-I-I
"'"'
-I I -I -I
-I -I -I
 78
Supervised ~aming Network

Figure 316 Functional line nerwork model.

x, 'x,
~
/

x, x, 0
y y
1

~G
Figure 317 The XOR problem.

Thus, ir can be easily seen rhar rhe functional link nerwork in Figure 3~ 17 is used for solving this problem.
The li.Jncriona.llink network consists of only one layer, therefore, ir can be uained using delta learning rule
instead of rhe generalized delta learning rule used in BPN. As, a result, rhe learning speed of the fUnc6onal
link network is faster rhan that of the BPN.

I 3.9 Tree Neural Networks

The uee neural networks (TNNs) are used for rhe pattern recognition problem. The main concept of this
network is m use a small multilayer neural nerwork ar each decision-making node of a binary classification
tree for extracting the non-linear features. TNNs compbely extract rhe power of tree classifiers for using
appropriate local fearures at the rlilterent levels and nodes of the tree. A binary classification tree is shown in
Figure 3-18.
The decision nodes are present as circular nodes and the terminal nodes are present as square nodes. The
terminal node has class label denoted 'by Cassociated with it. The rule base is formed in the decision node
(splitting rule in the form off(x) < 0 ). The rule determines whether the panern moves to the right or to the
left. Here,f(x) indicates the associated feature ofparcern and"(}" is the threshold. The pattern will be given
the sJass label of the terminal node on which it has landed. The classification here is based on the fact iliat
the appropriate features can be selected ar different nodes and levels in the tree. The output feature y = j(x)

L_i.._
3.10 Wavelet Neural Networks 79

Yes No

I "=' I I C=21 I C=1 I I C=31

Figure 318 Binary classification tree.

obtained by a multilayer network at a panicular decision node is used in the following way:

x directed to left child node tL, if y < 0

x directed to right child node tR, if y ::: 0

The algorithm for a TNN consists of two phases:

1. Tree growing phase: In this phase, a large rree is grown by recursively fmding the rules for splitting until
all the terminal nodes have pure or nearly pure class membership, else it cannot split further.
2. Tree pnming phase: Here a smaller tree is being selected from the pruned subtree to avoid the overfilling
of data.
The training ofTNN involves [\VO nested optimization problems. In the inner optimization problem, the
BPN algorithm can be used to train the network for a given pair of classes. On the other hand, in omer
optimization problem, a heuristic search method is used to find a good pair of classes. The TNN when rested
on a character recognition problem decreases the error rare and size of rhe uee relative to that of the smndard
classifiCation tree design methods. The TNN can be implemented for waveform recognition problem. It
obtains comparable error rates and the training here is faster than the large BPN for the same application.
Also, TNN provides a structured approach to neural network classifier design problems.

I 3.10 Wavelet Neural Networks

The wavelet neural network (WNN) is based on the wavelet transform theory. This nwvork helps in
approximating arbitrary nonlinear functions. The powerful tool for function approximation is wavelet
decomposition.
Letj(x) be a piecewise cominuous function. This function can be decomposed into a family of functions,
which is obtained by dilating and translating a single wavelet function: !(' --')- R as

j(x) = L' w;det [D) 12] [D;(x- 1;)]

i::d

where D,. is the diag(d,), d,. EJ?t

ate dilation vectors; Di and t; are the translational vectors; det [ ] is the
determinant operator. The w:..velet function selecred should satisfy some properties. For selecting: If' --')o
R, the condition may be
,P(x) =1 (XJ) .. t/J1 (X 11 ) forx:::: (x, X?. . . , X11 )
80 Supervised Learning Network

-r
~)-~-{~Q-{~}-~~-~ 7

:_,, : \] : : ~
0----{~J--[~]-----{~-BJ------0-r
: I
K
Input( X

Output

&-c~J-{~~J-G-cdFigure 319 Wavelet neural network.

where

, (x) = -xexp ( -~J

is called scalar wavelet. The network structure can be formed based on rhe wavelet decomposirion as

y(x) = L" w; [D;(x- <;)] +y

i=l

where J helps to deal with nonzero mean functions on finite domains. For proper dilation, a rotation can be
made for bener network operation:

y(x) = L" w; [D;R;(x- <;)] + y

i=l

where R; are the rotation marrices. The network which performs according to rhe above equation is called
wavelet neural network. This is a combination of translation, rotarian and dilation; and if a wavelet is lying on
the same line, then it is called wavekm in comparison to the neurons in neural networks. The wavelet neural
network is shown in Figure 3-19.

1 3.11 Summary

In chis chapter we have discussed the supervised learning networks. In most of the classification and recognition
problems, the widely used networks are the supervised learning networks. The.architecrure, the learning rule,
flowchart for training process-and training algorithm are discussed in detail for perceptron network, Adaline,
Madaline, back-propagation network and radial basis function network. The percepuon network can be
trained for single output clasSes as well as mulrioutput classes. AJso, many Adaline networks combine together
 ..~~'"
3.12 Solved Problems 81

ro form a Madaline network. These networks are trained using delta learning rule. Back-propagation network
is the most commonly used network in the real time applications. The error is back-propagated here and is
fine runed for achieving better performance. The basic difference between the back-propagation network and
radial basis function network is the activation funct'ion. use;d. The radial basis function network mostly uses
Gaussian activation funcr.ion. Apart from these nerWor~; some special supervised learning networks such as
time delay neural ne[Wotks, functional link networks, tree neural networks and wavelet neural networks have
also been discussed.

I 3.12 Solved Problems

I. I!Jlplement AND function using perceptron net- Calculate the net input
~ //works for bipol~nd targets.
y;, = b+xtWJ +X2W2
Solution: Table 1shows the truth table for AND
function with bipolar inputs and targelS:
=O+Ix0+1x0=0
The output y is computed by applying activations
Table 1
over the net input calculated:

I {
X]
"'I I ify;,> 0 - .
-I -I y = f(;y;,) = 0 if y;, = 0
-I I -I -1 ify;71 <0
-I -I -I . - - . .- -- .--_-==-...
Here we have rake~-1) = O.)Hence, when,y;11 = 0,
The perceptron network, which uses perceptron
y= 0. ---
learning rule, is used to train the AND function. Check whether t = y. Here, t = 1 andy = 0, so
The network architecture is as shown in Figure l. t f::. y, hence weight updation takes place:
The input patterns are presemed to the network one
w;(new) = zv;(old) + ct.t:x;
by one. When all the four input patterns are pre-
sented, then one epoch is said to be completed. The WJ(new) = WJ(oJd}+ CUXJ =0+] X I X l = 1
initial weights and threshold are set to zero, i.e., W2(ncw) = W2(old) + atx:z = 0 + 1 x l x 1 = I
WJ = WJ. = h = 0 and IJ = 0. The learning rate b(ncw) = h(old) + at= 0 + 1 x I = l
a is set equal to 1.
Here, the change in weights are

x,~
Ll.w! = ~Yt:q;
Ll.W2 = atxz;
b..b = at
X, ~ y y

w, The weighlS WJ = I, W2 = l, b = 1 are the final

weighlSafrer first input pattern is presented. The same
~
X, process is repeated for all the input patterns. The pro-
cess can be stopped when all the wgets become equal
Figure 1 Perceptron network for AND function. to the cllculared output or when a separating line is
For the first input pattern, x 1 = l, X2 = I and obrained using the final weights for separating the
t = 1, with weights and bias, w1 = 0, W2 = 0 and positive responses from negative responses. Table 2
b=O, shows the training of perceptron network until its
 82 Supervised Learning Network

Table2
Weights
Calculated Weight changes
Input b
Target Net input output W) W].

X) X]. (t) (y,,) (y) ~WI f:j.W'l M (0 0 0)

EPOCH-I
I I 0 0 I
-I -1 -I -I 0 2 0
-I -I 2 +I -1 -I I -I
-1 -1 -I -3 -I 0 0 0 1 -1 -I
EPOCH-2
I I 0 0 0 -I
I -1 -I -1 -I 0 0 0 -I
-I -I -I -I 0 0 0 -1
-I -1 -3 -I 0 0 0 -I

target and calculated ourput converge for all the ~

patterns.
The final weights and bias after second epoch are
/'~-..._.};.- . .~:-
=l,W'l=l, b=-1 (-1, 1)
,_~--
W[

0 . _,. \.. . .. ,
Since the threshold for the problem is zero, the
equation of the separating line is '
-x,
l~
~
,x,. . J/
";?")..
. ~i
w,
X2 = - - X i - -
b :.. /1--'
(-1,-1) (1,-1) ~=-X,+1 '/

Here
'"' ""
W[X! + lli2X2 + b > $
W]X] + UlzX2 + b> Q -X,

Thus, using the final weights we obtain Figure 2 Decision boundary for AND function
in perceptron training{$= 0).
I (-1)
X2 = -}x' - -~- ~'mplemenr OR function with binary inputs and

lil~J
L _ -xt+l bipolar targw using perceptron training algo-
rithm upto 3 epochs.
h can be easily found that the above straight line
separates the positive response and negative response Solution: The uuth table for OR function with
region, as shown in Figure 2. binary inputs and bipolar targets is shown in Table 3.
The same methodology can be applied for imple- Table 3
menting other logic functions such as OR, AND- t
NOT, NAND, etc. If there exists a threshold value
Xj
"'-
f) ::j:. 0, then two separating lines have to be obtained, I
i.e., one to se-parate positive response from zero 0
and the other for separating zero from the negative 0 I
0 0 -I
response.

"'
3.12 Solved Problems 83

0----z The final weights at the end of third epoch are

w, =2,W]_ = l,b= -1
x, X w,~y y
Fu-rther epochs have to be done for the convergence

~
of'the network.
3. _Bnd-the weights using percepuon network for
/AND NOT function when all the inpms are pre-
sented only one time. Use bipolar inputS and
Figure 3 Perceptron network for OR function.
' targets.
The perceptron network, which uses perceptron Solution: The truth table for ANDNOT function is
learning rule, is used to train the OR function. shown in Table 5.
The network architecture is shown in Figure 3. TableS
The initial values of the weights and bias are taken
Xj "'- t
as zero, i.e.,
I I -I
WJ=W]_:::::b:::::O 1 -I I
-I I -1
Also the learning rate is 1 and threshold is 0.2. So, -I -I -I
the aaivation function becomes
The network architecture of AND NOT function is
1 if y;/1> 0.2 ~ shown as in Figure 4. Let the initial weights be zero
[(yin) ;::: { O if - 0.2 ~Yin ~ 0.2 and ct = l,fJ = 0. For the first input sample, we
compme the net input as
The network is trained as per the perceptron training "
algorithm and the steps are as in problem 1 (given for Yin= b+ Lx;w; = h+x1w 1 +xzlil2
first pattern}. Table 4 gives the network rraining for i=-1

3 epochs. =O+IxO+IxO=O

Table4
Weights
Input Calculated Weight changes
Target Net input output w, W2 b
Xi X2 (t) {y;,,) (y) ~W) ~., ~b (0 0 0)
EPOCH-I
I 0 0 I I I
0 2 0 0 0
0 2 0 0 0 I I 0
0 0 -I 0 0 -I I I 0
EPOCH-2
2 0 0 0 I I 0
I 0 0 0 0 I I 0
0 I I 0 0 0 I I 0
0 0 -I 0 0 0 0 0 I I -I
EPOCH-3
I I I 0 0 0 I I -I
I 0 0 0 I 0 I 2 I 0
0 I I I I 0 0 0 2 I 0
0 0 -I 0 0 0 0 -I 2 I -I
 C:
Supervised Learning Network
84

For the third input sample, XI = -1, X2 = 1,

0----z t = -1, the net input is calculated as,

x, x, w,~y _..,;'
y '
]in= b+ Lx;w;= b+XJWJ +X2WJ.
w, i=l

X, =0+-1 X O+ 1 X -2=0+0-2= -2
X,

Figure 4 Network for AND NOT function. The output is oblained as y = fi.J;n) -1. Since =
t = y, no weight changes. Thus, even after presenting
Applying the activation function over the net input, clJe third input sample, the weights are
we obtain
w=[O -2 0]

=I ~
ify;,. > 0
y=f(y,,) if-O~y;11 ::S:0 For the fourth input sample, x1 = -1, X2 = -1,
l-1 ify;,. < -0
t = -1, the net input is calculated as

'
Hence, the output y = f (y;,.) = 0. Since t ::/= y, U.e
]in= b+ Lx;w; = b+x1w1 +X21112
new weights are computed as
i=l
WJ (new) = W] (o\d) + (UX] = 0 + 1 X -} X 1 = -} =0+-lxO+(-lx-2)
U12(new) = W2.(old) + cttx2_ = 0 + 1 x -1 x l = -1 =0+0+2=2
b(new) = b(old)+ at= 0 + 1 x -1 = -1
The output is obtained as y = f (y;n) = 1. Since
The weights after presenting the first sample are t f. y, the new weights on updating are given as

w=[-1-1-1] WJ (new) = WJ (old)+ UXj = 0+ l X -I X -I = 1

For the seconci inpur sample, we calculate the net IU2(new) = Ul!(old) + ct!X'z = -2 +I x -1 x -1 =-I
inpur as
b(ncw) = b{old) +at= O+ 1 X -1 = -1
'
Yin= b + L:x;w; = b +x1w1 +X2W.Z The weights after presenting foun:h input sample are
i:= I
w= [1 -1 -1]
=-l+lx-1+(-lx-1)
One epoch of training for AND NOT function using
=-1-1+1=-1
perceptron network is tabulated in Table 6.
The output y = f(y;") is obtained by applying
activation function, hence y = -1. Table&
Since t i= y, the new weights are calculated as Weights
Calculated
Input
Wj{new) = WJ(oJd) + CUXJ = -l + 1 X I X J = 0 _ _ _ Target Net input output WJ "'2 b
XI X:Z 1 (t) (y;,) (y) (0 0 0)
Ul2(new) = Ul2(old) + CtD:l = -1 + 1 x l x-I= -2
1 1 -1 0 0 -1 -1 -1
b(new) = b{old) +at= -1 + l xI =0
0 -2 0
1 -1 1 1 -1 -1
The weights after presenting the second sample are -1 1 1 -1 -2 -1 0 -2 0
-1 -1 1 -1 2 1 1 -1 -l
w= [0 -2 0]
:J
3.12 Solved Problema 85

4. Pind the weights required to perform the follow- Table.7

/ ing classification using percepuon network. The Input
(/ vectors (1,), 1, 1) and (-1, 1 -1, -1) are belong-
ing to the class (so have rarger value 1), vectors X2 b Targ.t (t)
(1, 1, 1, -1) and (1, -1, -1, 1) are not belong-
'J
) 1 "'1 "'
ing to the class (so have target value -1). Assume -1 1 -1 -1
learning rate as 1 and initial weights as 0.
-1 1 -1
Solution: The truth table for lhe given vectors is given -1 -1 1 1 -1
in Table_?.- ----.. ><

Le~Wt = ~~.l/l3. = W< "' b ,;;-p and the

lear7cng ratec; = 1. Since the thresWtl = 0.2, so Thus,ln the third epoch, all the calculated outputs
the.' ctivation function is become equal to targets and the necwork has con-
verged. The network convergence can also be checked

y., { ~
if ]in> 0.2 by forming separating line equations for separating
if -0.2 :S Yin :S 0.1 positive response regions from zero and zero from
-1 if Yin< -0.2
negative response region.
The network architecture is shown in Figure 5.
The net input is given by
5. Classify the two-dimensiona1 input pattern shown
]in= b+x1w1 +xzWJ. +X3W3 _/ in Figure 6 using perceptron network. The sym~
+x4w4 bol "*" indicates the da[a representation to be +1
and "" indicates data robe -1. The patterns are
The training is performed and the weights are tabu- I-F. For panern I, the targer is+ 1, and for F, the
lated in Table 8. target is -1.

Tables
Weights
Inputs Target Net input Output Weight changes (w, w, w, w4 b)
(x, X2 X4 b) (t) (Y;,) (y) (.6.w1 /J.llJ2 .6.w3 IJ.w4 !:J.b) (0 0 0 0 0)
EPOCH-! "'
( 1 1 1 1 1) 1 0 0 1 1 1 l 1 1 1 1 1 1
(-1 1 -1 -1 1) 1 -1 -1 -1 1 -1 -1 1 0 2 0 0 2
( 1 1 l -1 1) -1 4 I -1 -1 -I 1 -1 -1 1 -1 1
( 1 -1 -1 1 1) -1 1 1 -1 1 1 -1 -1 -2 2 0 0 0
EPOCH-2
( 1 1 1 1 1) 1 0 0 1 1 1 1 1 -1 3 1
(-1 1 -1 -1 1) 1 3 1 0 0 0 0 0 -1 3 1
( 1 1 1 -1 1) -1 4 1 -1 -1 -1 1 -1 -2 2 0 2 0

I
( 1 -1 -1 1 1) -1 -2 -1 0 0 0 0 0 -2 2 0 2 0
EPOCH-3
( 1 1 1 1 1) 1 2 1 0 0 0 0 0 -2 2 0 2 0
I (-1
( 1 1 1
1 -1 -1
-1
l)
1) -1
1 2
-2 -1
1 0
0
0
0
0
0
0
0
0
0
-2 2
-2 2
0
0
2 0
2 0

l
!__1_ -1 -1 1 1) -1 -2 -1 0 0 0 0 0 -2 2 0 2 0
86 Supervised learning Network
I
= b +x1w1 + XZW2 +X3w3 +X4W4 +xsws 1
+XGW6 + X7WJ + xawa + X9W9
=0+1 x0+1 x0+1 x 0+(-1) xO
+1xO+~Dx0+1x0+1x0+1xO

Yin= 0
Therefore, by applying the activation function the
output is given by y = ff.J;n) = 0. Now since t '# y,
the new weights are computed as
Wi(new) = WJ(oJd)+ atx1 =-0+ 1 X 1 X 1 = 1
w,(new) = w,(old) + 01>2 = 0 + 1 x 1 x 1 = 1
w3(new) = w3(old) + at:q =0+ 1 x 1 x 1= 1
Figure 5 Network archirecrure. W.j(new) = W4(o!d) + CUX4:;:: 0 + l X l X -1 = -1
w;(new) = w;(old) + atx;_ = 0 + 1 x 1 x l = 1
WG(new) = W6(old) + CttxG = 0 + 1 X 1 X -1 = -1
W)(new) = W)(old)+ O"'J = 0 + 1 x 1 x 1 = 1

ws(new) = wg(old) + ""' = 0 + 1 x 1 x 1 = 1

W<J(new) = rlJ9(old) + O:fX9 = 0 + 1 x l x 1 = 1
'I' 'P
b(new) = b(old) + ot = 0 + 1 x 1 = 1
Figure 6 I~F data representation.
The weights afrer presenting first input sample are
Solution: The training patterns for this problem are w = [11 1 - 1 1 - 1 1 1 1 1]
tabulated in Table 9.
Forrhesecondinputsample,xz=[1111111-1
Table 9 -1 1], t= -1, rhe ner inpm is calculated as
Input

Pattern x 1 xz X3 .r4 x5 X6 Xi xa X9 1 Target (t) Yir~ = b+ L:x;w;

r"=l
1-11-111111
F 1 1 1 1 1 ' 1 -1 -11 -1 = b +X] W] + XZWJ. + X3W3 + X4W4 + X5W5
+ X6W6 + X7lll] + XflWB +X<) IV<)
The initial weights are all assumed to be zero, i.e., =1+ l X 1+ 1 X l+ l X 1+ 1 X -1 + 1 X 1
e = 0 and a = 1. The activation function is given by
+1x-1+1x1+(-1)x 1+(-1)x1

~y~ {~ ifJ.rn> o
if-O:Sy;, 1 .::;:0 i Yin= 2

. -1 ifyrn < -0 I Therefore the output is given by y = f (y;u) = l.

I Since t f= y, rhe new weights are
For the first input sample, Xj = [l 1 L ~ I--1 -1 1 1
1 1], t = l, the net input is calculated as
w,(new) == WJ(old) + o:oq == l +1x -1 X\== 0
fV2(new)== fV2(old) + O:tx]. = 1 + 1 X -1 X l = 0
w3(new) = w3(old)+ O:b:J =I+\ X -1 X 1 = 0
y;, = b + Lx;w;
i=l
w~(new) = wq(old) + CtP:4 =-I+ 1 x -1 x t = -2
 3.12 Solved Problems

w;(new) = w;(old)+ O:IXS = 1 + 1 x -1 x 1 = 0 lnitiaJly all the weights and links are assumed to be
W6(new) == WG(oJd) + 0:0:6 = -1 + 1 X -1 X 1 = -2 small raridom values, say 0.1, and the learning rare is
87
II
also set to 0.1. Also here the least mean square error
W?{new) = W?(old) + atx'] =I+ 1 x -1 x 1 = 0
wg(new) = ws(old)+ o:txs = 1 + 1 x -1 x -1 = 2
miy Qe set. The weights are calculated until the least
m~ square error is obtained.
I
fU9(new) == fV9(old) + etfX9 = 1 + 1 x -1 x -1 "== 2 The initial weighlS are taken to be WJ = W2 =
b[new) = b(old) +or= I+ 1 x -1 = 0 b = 0.1 and rhe learning rate ct = 0.1. For the first
input sample, XJ = 1, X2 = 1, t = 1, we calculate the
The weighlS afrer presenting rhe second input sam~ net input as
pie are ~

' 2
w = [0 0 0 - 2 0 -2 0 2 2 0] Yin= b+ Lx;w; = b+ Lx;w;
The network architecture is as shown in Figure 7. The i=l i=l

network can be further trained for its convergence. = b+x1w1 +xzwz

= 0.1 + 1 X 0.1 + 1 X 0.1 = 0.3

Now compute (t- y;n) = (1- 0.3) = 0.7. Updating

the weights we obrain,

w;(new) = w;(old) + a(t- y;n)x;

where a(t- y;11 )x; is called as weight change fl.w;.

The new weights are obtained as
y
w,(new) = WJ(old)+fl.wl = 0.1 +O.l X 0.7 X 1
= 0.1 + 0.07 = 0.17
w,(new) = w,(old)+L>W2 = 0.1
+ 0.1 X 0.7 X 1 = 0.17
b(new) = b(old)+M = 0.1 + 0.1 x 0.7 = 0.17

where

6.w1 = a(t- JirJ~l

I~
Figure 7 Network architecture.
.6.wz = a(t- y;,)X2
lmplemenr OR function with bipolar inputs and
t.b = o(t- y;,)
targelS using Adaline network.
Solution: The truth table for OR function with Now we calculare rhe error:
bipolar inpulS and targers is shown in Table 10.
E = (r- y;,) 2 = (0.7) 2 = 0.49
Table 10
Xj X:z t The final weights after presenting ftrsr inpur sam
- pie are
1
-1 w= [0.17 0.17 0.17]
-1
-1 -1 -1 and errorE= 0.49.
88 Supervised learning Network

These calculations are performed for all the input Table 12

samples and the error is caku1ared. One epoch is Epoch Total mean square error
completed when all the input patterns are presented.
Epoch I 3.02
Summing up all the errors obtained for each input
Epoch 2 1.938
sample during one epoch will give the mtal mean
Epoch 3 1.5506
square error of that epoch. The network training is
Epoch 4 1.417
continued until this error is minimized to a very small
Epoch 5 1.377
value.
Adopting the method above, the network training

~-
is done for OR function using Adaline network and
is tabulated below in Table 11 for a = 0.1.

-".~
The total mean square error aft:er each epoch is
given as in Table 12. ,1 @ w1 == 0.4893 f::\_ ~
Thus from Table 12, it can be noticed that as
training goes on, the error value gets minimized.
~~1'~Y
Hence, further training can be continued for fur~ -
t:her minimization of error. The network archirecrure ~
of Adaline network for OR function is shown in
Figure 8. Figure 8 Network architecture of Adaline.

Table 11
Net Weights
Inputs T: Weight changes
- - a<get input Wt b Enor
X] x:z I t Yin (r- Y;,l) i>wt
"'"" i>b (0.1 ""
0.1 0.1) (t- Y;,?
EPOCH-I
I I I I 0.3 0.7 0,07 0,07 om 0.17 0.17 0.17 0.49
I -1 I I 0.17 0.83 0.083 -0.083 0.083 0.253 0.087 0.253 0.69
-I I I I 0.087 0.913 -0.0913 0,0913 0,0913 0.1617 0.1783 0.3443 0.83
-1 -1 1 -I 0.0043 -1.0043 0.1004 0.1004 -0.1004 0.2621 0.2787 0.2439 1.01
EPOCH.2
1 I 1 1 0.7847 0.2153 0.0215 0.0215 0.0215 0.2837 0.3003 0.2654 0.046
I -1 1 I 0.2488 0.7512 0.7512 -0.0751 0.0751 0.3588 0.2251 0.3405 0.564
-I I 1 I 0.2069 0.7931 -0.7931 0.0793 0.0793 0.2795 0.3044 0.4198 0.629
-1 -1 I -I -0.1641 -0.8359 0.0836 0.0836 -0.0836 0.3631 0.388 0.336 0.699
EPOCH-3
I I I I 1.0873 -0.0873 -0.087 -0.087 -0.087 0.3543 0.3793 0.3275 0.0076
I -1 I I 0.3025 +0.6975 0.0697 -0.0697 0.0697 0.4241 0.3096 0.3973 0.487
-I I I I 0.2827 0.7173 -0.0717 0,0717 0,0717 0.3523 0.3813 0.469 0.515
-1 -1 1 -1 -0.2647 -0.7353 0.0735 0.0735 -0.0735 0.4259 0.4548 0.3954 0.541
EPOCH-4
I I I I 1.2761 -0.2761 -0.0276 -0.0276 -0.0276 0.3983 0.4272 0.3678 0,076
I -1 I I 0.3389 0.6611 0.0661 -0.0661 0.0661 0.4644 0.3611 0.4339 0.437
-I I 1 I 0.3307 0.6693 -0.0669 0.0669 0.0699 0.3974 0.428 0.5009 0.448
-1 -1 I -I -0.3246 -0.6754 0.0675 0.0675 -0.0675 0.465 0.4956 0.4333 0.456
EPOCH-5
I I I I 1.3939 -0.3939 -0.0394 -0.0394 -0.0394 0.4256 0.4562 0.393 0.155
I -1 I I 0.3634 0.6366 0.0637 -0.0637 0.0637 0.4893 0.3925 0.457 0.405
-I I I I 0.3609 0.6391 -0.0639 0.0639 0.0639 0.4253 0.4654 0.5215 0.408
-1 -1 I -I -0.3603 -0.6397 0.064 0.064 -0.064 0.4893 0.5204 0.4575 0.409

I
-~
 11

II
3.12 Solved Problems 89

7. UseAdaline nerwork to train AND NOT funaion w,(new) = w,(old) + a(t- y,,)x:z
'
with bipolar inputs and targets. Perform 2 epochs
of training.
= 0.2+ 0.2 X (-1.6) X I= -0.12
b(new) = b(old) + a(t- y;,) !
Solution: The truth table for ANDNOT function = 0.2+ 0.2 (-1.6) = -0.12
X '!:
with bipolar inputs and targets is shown in Table 13.
Table 13 Now we compute the error,

E= (t- y;,) 2 = (-1.6) 2 = 2.56

The final weights after presenting first input sample

a<e w = [-0.12- 0.12- 0.12] and errorE= 2.56.
The operational steps are carried for 2 epochs
of training and network performance is noted. It is
Initially the weights and bias have assumed a random
tabulated as shown in Table 14.
value say 0.2. The learning rate is also set m 0.2. The
The total mean square error at the end of two
weights are calculated until the least mean square error
epochs is summation of the errors of all input samples
is obtained. The initial weights are WJ = W1. b = = as shown in Table 15.
0.2, and a= 0.2. For the fim input samplex1 = 1,
.::q = l, & = -1, we calculate the net input as Table15
Yin= b + XtWJ + X2lli2
)
= 0.2+ I X 0.2+ I X 0.2= 0.6
Epoch Total mean square error
ll
Epoch I 5.71 :~
Now compute (t- Yin} = (-1- 0.6) = -1.6. Epoch 2 2.43 '
Updacing ilie weights we obtain
Hence from Table 15, it is clearly undersrood rhat the .,
w,-(new) = w,-(old) + o:(t- y,n)x; mean square error decreases as training progresses. '
The new weights are obtained as Also, it can be noted rhat at the end of the sixth \;
epoch, rhe error becomes approximately equal to l.
WI (new) ::::: w, (old) + ct(t- Jj )x,
11 The network architecture for ANDNOT function
= 0.2 + 0.2 X (-1.6) X I= -0.12 using Adaline network is shown in Figure 9.
Table 14
Weights
Inputs Ne< Weight changes
_ _ Target input w, b Error
t>w, M (0.2 ""
0.2 0.2) (t- Y;n)2
X[ X:Z
EPOCH-I
I t Y;" (t-y;rl)
"'""
I -I 0.6 -1.6 -0.32 -0.32 -0.32 -0.12 -0.12 -0.12 2.56
-I I I -0.12 1.12 0.22 -0.22 0.22 0.10 -0.34 0.10 1.25
-I I I -I -0.34 -0.66 0.13 -0.13 -0.13 0.24 -0.48 -0.03 0.43
-1 -1 I -I 0.21 -1.2 0.24 0.24 -0.24 0.48 -0.23 -0.27 1.47
EPOCH-2
-I -0.02 -0.98 -0.195 -0.195 -0.195 0.28 -0.43 -0.46 0.95
I -1 I I 0.25 0.76 0.15 -0.15 0.15 0.43 -0.58 -0.31 0.57
-I I I -I -1.33 0.33 -0.065 0.065 0.065 0.37 -0.51 -0.25 0.106
-1 -1 I -I -0.11 -0.90 0.18 0.18 -0.18 0.55 -0.38 0.43 0.8

~
90 Supervised learning Network

input sample, XJ = 1, X2 = l, target t = -1, and

...
b.,o learning rate a equal to 0.5:

x, x ') w1=o.ss y
Calculate net input to the hidden units:
1
Zinl = b1 + XJ WlJ + X2U/2J
>Nz"'_o.~ = 0.3 + 1 X 0.05 + 1 X 0.2 = 0.55
x, x, Zin2 = /n. +X} WJ2 + xiW22
= 0.15 + 1 X 0.1 + 1 X 0.2 = 0.45
Figure 9 Network architecrure for ANDNOT
function using Adaline nerwork.. Calculate the output z 1,Z2 by applying the activa-
tions over the net input computed. The activation
8 Using Madaline network, implement XOR func- function is given by
tion with bipolar inputs and targets. Assume the
required parameters for training of the network. I ifz;,<:O
! (Zir~) = ( -1 ifz;11 <0
Solution: The uaining pattern for XOR function is
given in Table 16. Hence,
Table 16 z1 = j(z;,,) = /(0.55) = I
z, = /(z;,,) = /(0.45) = 1
After computing the output of the hidden units,
then find the net input entering into the output
unit:

Yin= b3 +zJVJ +z2112

The Madaline Rule I (MRI) algorithm in which the = 0.5 + 1 X 0.5 + I X 0.5 = 1.5
weights between the hidden layer and ourpur layer
remain fixed is used for uaining the nerwork. Initializ- Apply the activation function over the net input
ing the weights to small random values, the net\York Yin to calculate the output y.
architecture is as shown in Figure 10, widt initial
y = f(;y;,) = /(1.5) = 1
weights. From Figure 10, rhe initial weights and bias
are [wu "'21 bd = [0.05 0.2 0.3], [wn "'22 b,] = Since t f:. y, weight updation has to be performed.
[0.1 0.2 0.15] and [v 1 v, b3] = [0.5 0.5 0.5]. For fim Also since t = -1, the weights are updated on z1
and Zl that have positive net input. Since here both
1lbj=0.3 net inputs Zinl and Zinl are positive, updating the
weights and bias on both hidden units, we obtain
Wij(new) = Wij(old) + a(t- Zin)x;
bj(new) = bj(old) + a(t- z;"j)
y
This implies:
WI! (new)= WI! (old)+ a(t- ZinJ)XJ
=0.05+0.5(-1-0.55) X 1 = -0.725
WJ2(new) = WJ2(old) + a(t- Zin2)Xl
'bz =0.15 =0.!+0.5(-1-0.45) X I =-0.625
b1(new)= b1(old)+a(t-z;"Il
Figure 10 Nerwork archicecrure ofMadaline for
XOR funcr.ions .(initial weights given). =0.3+0.5( -I- 0.55) = -0.475
 3.1 '2 Solved Problems 91

w11 (new) =W21 (old)+a(t-ZinJ)XZ

=0.2+0.5(-1-0.55) X 1 =-0.575
"'22 (new)= "'22 (old)+ a(t- z;" 2)"2
=0.2+0.5(-1-0.45)x 1=-0.525'
b2 (new]= b2 (old)+ a(t- z;d y

= 0.15+0.5(-1-0.45)=-0.575
All the weights and bias between the input layer and
hidden layer are adjusted. This completes the train-
~::-1.08
ing for the first epoch. The same process is repeated
until the weight converges. It is found that the weight
Figure 11 Madaline network for XOR function
converges at the end of 3 epochs. Table 17 shows the
(final weights given).
training performance of Madaline network for XOR y
function.
The network architecture for Madaline network
with final weights for XOR function is shown in
Figure 11.
9._}Jsing back-propagation_ network, find the new
/ weights ~or the ~et shown in Figure 12. It is pre- .0.5
, semed wuh the mput pattern [0, 1] and the target 0.3,
output is 1. Use a learning rare a = 0.25 and
binary sigmoidal activation function.
Solution: The new weights are calculated based
on the training -algorithm in Section 3.5.4. The
-oj
Figure 12 Ne[Work.
initial weights are [v11 v11 vod = [0.6 -0.1 0.3],
Table 17
Inputs Target
X~ (t} wn b, b2
Zinl Zinl ZJ Zl Y;11 Y "'21 W12
'""
EPOCH-I
I I 1 -1 0.55 0.45 I 1 1.5 1-0.725 -0.58 -0.475-0.625 -0.525 -0.575
1-1 I I -0.625 -0.675 -1-1 -0.5 -1 0.0875-1.39 0.34 -0.625 -0.525 -0.575
-I 1 1 I -1.1375 -0.475 -I -1 -0.5 -I 0.0875 -1.39 0.34 -1.3625 0.2125 0.1625
-1-1 1 -1 1.6375 1.3125 1 1 1.5 1 1.4065 -0.069 -0.98 -0.207 1.369 -0.994
EPOCH-2
1 I I -1 0.3565 0.168 1 I 1.5 I 0.7285 -0.75 -1.66 -0.791 -0.207 -1.58
1-1 I 1 -0.1845-3.154 -1-1-0.5-1 1.3205-1.34 -1.068-0.791 0.785 -1.58
-1 1 I 1 -3.728 -0.002 -1-1-0.5-1 1.3205 -1.34 -1.068- 1.29 0.785 -1.08
-1-1 I -1 -1.0495-1.071 -1-1-0.5-1 1.3205 -1.34 -1.068-1.29 1.29 -1.08
EPOCH-3
-1 -1.0865-1.083 -1-1-0.5-1 1.32 -1.34 -1.07 - 1.29 1.29 -1.08
1 1 1
1-1 I I 1.5915-3.655 1-1 0.5 I 1.32 -1.34 -1.07 -1.29 1.29 -1.08
-I 1 I I -3.728 1.501 -1 1 0.5 1 1.32 -1.34 -1.07 -1.29 1.29 -1.08
1-1 1 -1 -1.0495-1.701 -1-1-0.5-1 1.32 -1.34 -1.07 -1.29 1.29 -1.08
92 SupeJVised Learning Network
-I
[v12 vn "02l = [-0.3 0.40.5] and [w, w, wo] = [0.4 This implies
0.1 -0.2], and the learning' rate is a = 0.25. Acti-
!, = (I - 0.5227) (0.2495) = 0.1191
vation function used is binary sigmoidal activation
function and is given by Find the change5~Ulweights be~een hidden and
output layer:.
I
f(x) = I+ ,- <'>wi = a!1 ZI = 0.25 X 0.1191 X 0.5498
,-- 0.0164 ::>
Given the output sample [x 1, X2] = [0, 1] and target
t= 1, t.w, = a!1 Z2 = 0.25 X 0.1191 X 0.7109

Calculate the net input: For zt layer ---=o:o2iT7

<'>wo = a! 1 = 0.25 x 0.1191 = 0.02978
Zinl = !lQJ + XJ + X2V21
V11
Compute the error portion 8j between input and
= 0.3+0 X 0.6+ I X -0.1 = 0.2 hidden layer (j = 1 to 2):
For z2 layer ~f'(
Dj= O;,j Zinj)

Zjril = VQ2 + Xj V!2 + X2.V1.2 '

O;,j= I:okwjk
= 0.5 + 0 X -0.3 +I X 0.4 = 0.9 k=!/
8;nj = 81 Wj! I.' only one output neuron]
Applying activation co calculate Ute output, we
obrain ------
=>!;,I= !1 wn = 0.1191.K0ft = 0.04764
-~
I
ZI = f(z;,,) = - - - = - - - = 0.5498
1 + e-z.o.1 1 + t-0.2
I =>O;,z = Ot Wzl = 0.1191
_,- X 0.1 = 0.01191
_-:~

Error, 81 =O;,,f'(Zirll).
I 1 j'(z;,I) = f(z;,,) [1- f(z;,,)]
z2 = f(z 2l = - - - = - - - = 0.7109
m 1 + e-Zilll 1 + e-0.9
= 0.5498[1- 0.5498] = 0.2475
Calculate the net input entering the output layer. 0 1 =8;,1/'(z;,J)
For y layer
= 0.04764 X 0.2475 = 0.0118
Ji11 = WO+ZJWJ +z2wz Error, Oz =0;,a/'(z;,2)
= -0.2 + 0.5498 X 0.4 + 0.7109 X 0.1
j'(z;,) = f(z;d [1 - f(z;,2)]
= 0.09101
= 0.7109[1 - 0.7!09] = 0.2055
Applying activations to calculate the output, we Oz =8;,zf' (z;,2)
obtain
= 0.01191 X 0.2055 = 0.00245
1 1
Y = f{y;n) = ~ = 1 + e-0.09101 = 0.5227 Now find rhe changes in weights between input
and hidden layer:
Compute the error portion 811.:
.6.v 11 =a0 1x1 =0.25 x0.0118 x0=0
!,= (t,- y,)f'(y,,.,) <'>"21 = a!pQ=0.25 X 0.0118 X I =0.00295
<'>vo1 =a!, =0.25 x0.0118=0.00295
Now
.6.v 12 =a82x1 =0.25 x0.00245 xO=O
f'(J;,) = f(y;,)[1 - f(J;,)] = 0.5227[1- 0.5227] ll:"22 =a!2X'2 =0.25 X 0.00245 X I =0.0006125
!' (J;,) = 0.2495 <'>v02 =a!2=0.25 x 0.00245 =0.0006!25
-I 3.12 Solved Problems 93
I Compute rhe final weights of the network:

v11(new) = VIt(old)+b.vJI = 0.6 + 0 = 0.6

vn(new) = vn(old)+t.v12 = -0.3 + 0 = -0.3 .
"21 (new) = "21 (oldl+<'>"21
= -0.1 + 0.00295 = -0.09705
vu(new) = vu(old)+t>vu
= 0.4 + 0.0006125 = 0.4006125
w,(new) = w1(old)+t.w, = 0.4 + 0.0164,
= 0.4164
Figure 13 Network.
w2(now) = w,(old)+<'>W2 = 0.1 + 0.02!17
= 0.!2!17 For z2layer
VOl (new) = VOl (old)+<'>OI = 0.3 + 0.00295
= 0.30295
z;,2 = V02 + XJVJ2 + X2V22
= 0.5 + (-1) X -0.3 +I X 0.4 = l.2
vo2(new) = 1102(old)+.6.vo2
= 0.5 + 0.0006125 = 0.5006!25 Applying activation to calculate the output, we
.,.(new)= .,.(old)+8wo = -0.2 + 0.02976 obtain

= -0.!7022 1_ t-"inl 1 _ t'0.4

ZI =f(z; 1l = - - - = - - = -0.!974
1 + t'-z:;nl 1 + /1.4
Thus, the final weights hav~ been computed for the
n
1- t'-Z:,;JL l - t'-1.2
network shown in Figure 12.
zz =/(z;,2) = - - - = - -1- 2 = 0.537
1+t-Zin2 1 +e-.
19. Find rhe new weights, using back-propagation
network for the network shown in Figure 13. Calculate lhe net input entering the output layer.
The network is presented with the input pat- For y layer
tern l-1, 1] and the target output is +1. Use a
learning rate of a = 0.25 and bipolar sigmoidal Yin= WO + ZJWJ +zzWz
activation function. = -0.2 + (-0.1974) X 0.4 + 0.537 X 0.1
Sn_ly.tion: The initial weights are [vii VZI vod = [0.6 = -0.22526
0.1 0.3], [v12 "22 vo2l = [ -0.3 0.4 0.5] and [w,
Wz wo] = [0.4 0.1 -0.2], and die learning rme is Applying activations to calculate the output, we
a= 0.25. obtain
Activation function used is binary sigmoidal 1 1 0.22526
activacion function and is given by 1 - t'- '" _-_--",=< -0.1!22
y = f(y;,) = l + t'-y,.. = 1 + 11-22526
2 1 -e-x
f (x )----1---
- 1 +e-x - 1 +e-x Compute the error portion 8k:

Given the input sample [x1, X21 = [-1, l] and target !, = (t, - yllf' (y;,,)
t= 1:
Calculate the net input: For ZJ layer

Zin\ =VOl +xJVJJ +X2t121

Now

'
----------------
I f'(J;.) = 0.5[1 + f(J;,)] [I- f(J;,)]
= 0.5[! - 0.1122][1 + 0.1122] = 0.4937 .
-- .
-~~
I = Q.3 + (-1) X 0.6 +I X -0.1 = -0.4
I '-..
----
)
l _...-/

l
94 Supervised learning Network

This implies f>'OI =01 = 0.25 X 0.1056';'0.0264

t,.,,=o 2x, =0.25 x 0.0195 x -1 =-0.0049
,, = (l + 0.1122) (0.4937) = 0.5491
[,."22 = cl02X, =0.25 X 0.0195 X 1 =0.0049
Find the changes in weights between hidden and l>'02 = o2= 0.25 X 0.0195 =0.0049
output layer:
Comp'Lite the final weights of the nerwork:
L\w1 = a81 ZJ = 0.25 X 0.5491 X -0.1974
= -0.0271 ""(new) = "" (old)+t., 11 = 0.6- 0.0264
/).w, = 01 Z2 = 0.25 X 0.549! X 0.537 = 0.0737 = 0.5736
L\wo = a81 = 0.25 x 0.5491 = 0.1373 ,,(n<w) = ,,(old)+t.,, = -0.3-0.0049
Compute the error portion Bj beMeen input and = -0.3049
hidden layer (j = 1 to 2): "21 (new) = "21 (old)+t...., 1 = -0.1 + 0.0264
= -0.0736
81 = 8;/ljj' (z;nj)
m ...,,(new) = "22(old)+t."22 = 0.4 + 0.0049
8inj = L 8k Wjk = 0.4049
~I
WI (new) = WI (old)+t.w 1 = 0.4- 0.0271
._ 8inj = 81 WjJ [ only one output neuron]
= 0.3729
=>8in1 =81 WJJ = 0.5491 X 0.4 = 0.21964
w,(n<w) = w,(old)+t.w, = 0.1 + 0.0737
( =>o;., =o, ""' = o.549I x o.1 = o.05491 = 0.1737
Error, 81 =8;,J/'(z;nJ) = 0.21964 X 0.5
1 ''' (n<w) = "OI (old)+l>'OI = 0.3 + 0.0264

~
X (I +0.1974)(1- 0.1974) = 0.1056
= 0.3264
Error, 82 =8;112/'(z;,2) = 0.05491 X 0.5
"oz(n<w) = '02(old)+t..,, = 0.5 + 0.0049
X (1- 0.537)(1 + 0.537) = 0.0195
= 0.5049
Now find the changes in weights berw-een input wo(new) = wo(old)+t.wo = -0.2 + 0.1373
and hidden layer: = -0.0627
f'l.V]J =Cl:'8]X1 =0.25 X 0.1056 X -1 = -0.0264
Thus, the final weight has been computed for the
/).'21 =OiX, =0.25 X 0.1056 X 1 =0.0264 network shown in Figure 13.

I 3.13 Review Questions

1. What is supervised learning and how is it differ- 7. Smte the activation function used in perceprron
em from unsupervised learning? network.
2. How does learning take place in supervised 8. What is the imporrance of threshold in percep-
learning? tron network?
3. From a mathematical point of view, what is the 9. Mention the applications of perceptron network.
process of learning in supervised learning? 10. What are feature detectors?
4. What is the building block of the perceprron? 11. With a neat flowchart, explain the training
5. Does perceprron require supervised learning? If process of percepuon network.
no, what does it require? 12. What is the significance of error signal in per-
6. List the limitations of perceptron. ceptron network?

,L
 3.14 Exercise Prob!ems 95

13. State the testing algorithm used in perceptron 34. What are the activations used in back-
algorithm. propagation network algorithm?
14. How is _the linear separability concept imple- 35. What is meant by local minima and global
mented using perceprron network training? minima?
15. Define perceprron learning rule. 3i5. Derive the generalized delta learning rule.
16. Define d_dta rule. 37. Derive the derivations of the binary and bipolar
1.1~ SGlte the error function for delta rule. sigmoidal activation function.
18. What is the drawback of using optimization 38. What are the factors that improve the conver-
gence of learning in BPN network?
algorithm?
19. What is Adaline?
39. What is meant by incremenrallearning?
20. Draw the model of an Adaline network. 40. Why is gradient descent method adopted to
minimize error?
21. Explain the training algorithm used in Adaline
41. What are the methods of initialization of
network.
weights?
22. How is a Madaline network fOrmed?
42. What is the necessity of momentum factor in
23. Is it true that Madaline network consists of many
weight updation process?
perceptrons?
43. Define "over fitting" or "over training."
24. Scare the characteristics of weighted interconnec-
tions between Adaline and Madaline. 44. State the techniques for proper choice oflearning
rate.
25. How is training adopted in Madaline network
using majority vme rule? 45. What are the limitations of using momentum
factor?
26. State few applications of Adaline and Madaline;
46. How many hidden layers can there be in a neural
27. What is meant by epoch in training process?
network?
28. Wha,r is meant by gradient descent meiliod?
47. What is the activation function used in radial
29. State ilie importance of back-propagation basis function network?
algorithm.
48. Explain the training algorithm of radial basis
30. What is called as memorization and generaliza- function network.
tion? 49. By what means can an IIR and an FIR filter be
31. List the stages involved in training of back- formed in neural network?
propagation network.
50. What is the importance of functional link net-
32. Draw the architecture of back-propagation algo work?
rithm.
51. Write a short note on binary classification tree
33. State the significance of error portions 8k and Oj neural network.
in BPN algorithm.
52. Explain in detail about wavelet neural network.

I 3.14 Exercise Problems

1. Implement NOR function using perceptron are belonging to the class (so have targ.etvalue 1),
network for bipolar inputs and targets. vector (-1, -1, -1, 1) and (-1, -1, 1 1) are
not belonging to the class_ (so have target value
2. Find the weights required to perform the fol-
-1). Assume learning rate 1 and initial weighlS
lowing classifications using perceptron network
The vectors (1, 1, -1, -1) ,nd (!,-I. 1, -I) "0.

L
 96 Supervised l.eaming Network

3. ClassifY the two-dimensional pattern shown in parrern [1. 0) and target output I. Use learning
figure below using perceptron n~rwork. rate of a == 0.3 and binary sigmoidal activation
function.



"C" "A"
Target value : +1 Target value :- 1

4. Implement AND function using Ad.aline net- y

work.
5. Using the delta rule, find the weights required
to perform following classifications: Vectors (1,
I, -I, -I) and (-1, -I, -I, -I) are belong-
ing to the class having target value 1; vectors 0.4
(1, I, I. I) and (-1, -1, I, -1) :uce not
belonging to- the class having rarget value -1.
9. Find the new weights for the network given in
Use a learning rate of 0.5 and assume ran-
clte above problem using back-propagation net-
dom value of weights. Also, using each of the
work. The network is presented the input pattern
training vectors as input, test the response of
[1, -1] and target output+ 1. Use learning rate
the net.

1 6. Implement AND function using Madaline net-

work.
of a = 0.3 and bipolar sigmoidal activation
function.
10. Find the new weights for the activation func-
.. 7. With suitable example, discuss the perceptron tion with the network shown in problem 8 using
network training with and without bias.
,, 8. Using back-propagation network, find the new
BPN. The network is presemed wilh the input
pattern [-1, I] and target output -l. Use learn-
' weights for the network shown in the following ing rate of a = 0.45 and suirable activation
figure. The network is presented with the input function.

I 3.15 Projects
1. Classify upper case letters and lower case leuers achieve the following rwo-ro-one mappings.
using perceptron ne[Work. Use as many output
units based on training set as possible. Test the y = 6 sin(JC x,) + cos(JCX'2.)
network with noisy pattern as well. y = sin(n x1) + cos(0.2JCX2.)
2. Write a suitable computer program to classify the Set up cwo sets of data, each consisting of l 0
numbers becween 0-9 using Adaline network. input-output pairs, one for training and ocher for
3. Write a computer program to train a Madaline to testing. The input-output data are obtained by
perform AND function using MRJ algorithm. varying input variables (x,, X2.)_ within [-1, +l]
4. Write a program for implementing BPN for train- randomly. Also the output daca is normalized
ing a .single hidden layer back-propagation net- within [-1, l}. Apply training to find proper
work with bipolar sigmoidal units (x = I) to weights in the network.

l
 U>>v.;,w-,

-~

Associative Memory Networks 4

Learning Objectives ----;-,------------~

Gives derails on associative memories. Hopfield network with its electrical model is
Discusses rhe training algorithm used for par- described with training algorithm.
tern association networks - Hebb rule and Analysis of energy function was performed
outer products rule. for BAM, discrete and continuous Hopfield
The architecture, flowchart for training pro- networks.
cess, training algorithm and testing algorithm An overview is given on rhe iterative aumasso-
of autoassociarive, heteroassociarive and bidi- ciative necwork - linear autoassociaror mem-
rectional associative memory are discussed in ory brain-in-the-box network and autoassoci-
detail. aror with threshold unit.
Variants of BAM - continuous BAM and Also temporal associative memory is discussed
discrete BAM are included. in brief.

I 4.1 Introduction

An associative memory network can store a set of patterns as memories. When the associative memory is being
presented with a key panern, it responds by producing one of the scored patterns, which closely resembles
or_ relates ro the key panern. Thus, che recall is through association of the key paJtern, with the help of
inforiTiai:iOO.rnernomed: These types of memories are also called as content-addressable memories (CAM) m
contrast to that of traditional address-addressable memor;es in digital computers where stored pattern (in byres)
is recalled by its address. It is also a matrix memory as in RAM/ROM. The CAM can also be viewed as
associating data to address, i.e.;--fo every data in the memory there is a corresponding unique address. Also,
ir can be viewed ~ fala correlato Here inpur data is correlated with chat of rhe stored data in the CAM.
It should be nored rh.rt-r- stored patterns must be unique, i.e., different patt:erns in each location. If the
same pattern exists in more than one locacion in rhe CAM, then, even though the correlation is correct, the
address is noted to be ambiguous. The basic srrucrure of CAM is ive Figure 4-1.
Associative memory makes aral e searc ithin a stored dar he concept behind this search is
to Output any one or .ill Stored items W i match the gi n search argumem and tO retrieve cite stored data
either complerely or pama:Ily.
Two cypes of associative memories can be differentiated. They are auwassodativt mnnory and httaoasso~
dative mtmo . Both these nets are_ sin .e-la_ er nets in which the wei hts are determined in a manner that -1-.
the nee srores a set of ia:ern associa"tionS. ch of this associatiOn "iS"an "iii.jmr-outpUYveCfoTfi"a.ir,-say,-.r.r:--
I each of the ourput..VeCt:ors-isSame as the input vecrors with which it is associated, then the net is a said to
I
I ,) JJ
~() " \ ~
'~ 1
.
_,\:.,f)

l
. \ l c'
,, 1 \ . ,,-: ~ :-S c(
( '" .
98 Associative Memory Networks

Match/No match

Input CAM
data Output
Matrix
bus data

Figure 41 CAM architecture.

be amoassociative memory neL On the orher hand. if the output vectors are different from rhe input vecwrs
rhen the net is said to be hereroassociarive memory net.
If rhere exist vectors, say, x = (Xt,X:!, ... , x11 )T and :J = (x 11,x;!', ... , x11 ')T, then the hamming distance
(HD) is defined as rhe number of mismatched components Qf x and i vectors, i.e.,

f.lxi- xJI if x;, xj E [0, \]

i==l
HD {x,x') =
I
l L \xi -x:l if x;, x; El-l,\}
"

i=l ---~-----
The architecture of an associative n_er may be eirhe feed~ forward or iterative (recu~ As is already known,
in a feed~forward net the information flows from rhe input umts to t e output umrs: on the other hand,
in a recurrcnr neural net, rhere are connections among the units ro form a dosed~loop strucwre. ln rhe
forthcoming sections, we will discuss the training algorithms used for pattern association and various cypes
of association nets in detail.

4.2 Training Algorithms for Pattern Association

There are two algorithms developed for training of pattern association nets. These are discussed below.

I 4.2.1 Hebb Rule

The_ t"lebb ryl_e:;)~ ytide_ly usedfodinding the weights of a,n_;~ssociative memory neural ner. The training vector
pairs here are denoted as s:r. The f1owcharr for the training algorithm orpaiietiCiSs"OChtrton is as shown in
Figure 4~2. The weights are updated unril there is no weight change. The algorithmic steps followed are given
below:

I Step 0: Set all the initial weights to zero, i.e., I

Wij = 0 (i = 1 w n,j = 1 to m)

Step 1: For each training target input output vector pairs s:t, perform Steps 2-4.
Step 2: Activate the input layer units to current training input,

Xi=Si (fori= lton)

1
 4.2 Training Algorithms for Pattein Association 99

cy
Initialize all wei9hts .
w, = O{i= 1ton,j,= 1t0m)

Fm
No
each
1

Yes

Present input signals

x, = s,

Present output signals

y =,
' '

Weight adjustment
w,(new) = w,(old)+X.Y,

\'
Figure 42 Flowchart ~Or Hebb rule.

Step 3: Activate rhe omput layer units ro current target output,

\-'
Yj = lj (for j = I to m) <.v ,., ,)

Step 4: Start the weight adjustmenr 0' u'.. -:'

~~
"
wy(new) = w,Aold) +X1Jj (for i = I to n,j = l to m)

This algorithm is used for the calculation- of the weights of the associative nets. Also, it can be used with
patterns that are being represented as either binary or bipolar vectors.

1
 100 Associative Memory Networks

I 4.2.2 Outer Products Rule

_Outer products rule is an alternative method for finding weights of an associative net. This is depicted as
follows:

Input=> s = (!J, ... ,s;, ... ,s11 )

Output=> t= (t11tm)

The outer product of the two vecmrs is the product of the mauices S = l and T = t, i.e., between [n X 1]
marrix and [1 x m} matrix. The aanspose is to be taken for the input matrix given.
The matrix multiplication is done as follows:
. --:;:--...._
ST=Jt.J
,,
'-
d'
o/
,. .,-.\
6.
= ,, [~+'"'&
'.
; Sn{n~
\
. SJ!J-- ... S]tj . S]tm
'c
")

1:
\ :
\
\. W = I s;t1 . . . s;tj s;tm
;;./

s11 t1 Sn~ Sntm...J >rXm

This weight matrix is same as the weight mauix obtained by Hebb rule to store~ s;t.
For storing a set of associations, s(p):t(p), p = 1 toP, wherein, ---- , .:_ 1

t ,'\ '~
,(p) = (s 1(p}, ... , s;(p), ... , s.(p))
t(p) = (~ (p), ' lj(p}, '<m(p)} y
.;. o~vf
rhe weight matrix W = {wij} can be given as~---------\ If o,I '-
' r' N _T
'\!:."' : "\ .
t:E(p)t,(p) } ' '
~ ~-
,)- ': \
'\,
':.::!-'
~. s
0 '
' '\ .~
\'
This can also be rewriuen as
~--p--,

\w~2?:J
 4.3 Autoassociative Memory Network 101

for finding the weights of the net using Hebbian learning. Similar to the Hebb rule, even the delta rule
discussed in Chapter 2 can be used for storing weights of panern association nets.

I 4.3 Autoassociative Memory Network

I 4.3.1 Theory

In the case of an auroassociative neural net, rhe trainin input and the target output vectors are the same.
The-determination of weights of the associafion net is called st ors. IS type o memory net needs
suppressiOn o t e output no1se at the memory output. The vectors that have been stored can be retrieved
from distorted (noisy) input if the input is sufficiently similar to iL The net's performance is based on irs
ability to reproduce a stored parrern from a noisy inpuL lr should be noted, that in rhe case of autoassociative
net, the wei hrs on the diagonal can be set to zero. This can be called as auto associative net with no self-
connection. The main reason behin semng the weights to zero is that it improves the net's ability to generalize
or in~e rhe biologicall-'lausibiliry of rhe net. This may be more suited for iterative nets and when defi:a
I'iile iS being usea~ --
@
I 4.3.2 Architecture

The architecture of an autoassociative neural net is shown in Figurt' 4-3. It shows that for an auroassociative
net. rhe uaining input and target output vectors are the same. The input layer consists of'' input units
and tht' output layer also consists of'!.. output units. The input and outp~ layers are connecred through
weighted interconnections. The inR!!.!-an~ut vectors are _perfectly correlated With each other component
___ _
by component.
,
~---------

I 4.3.3 Flowchart for Training Process

The flowchart here is the same <IS discussed in Section 4.2. \,bur it may be noted that rhe number ofinpm
units and outpur units are rhe same. The flowchart is shown in Figure 4-4.

x, lx'l
' ""
w,_/0-----
w Yl Y1
" ' Wnl'

w11
x, trx;)? 0( w~0--YJ
WnJ/

'
'
W~lnl'
x.
ix.;)Y w~@-------
n
1 Yn

l
Figure 43 Architecture of auroassociarive net.
102 Associative Memory Networks

Initialize the weights to zero

w ::0
'

.--------- t-ors:: 1 1u 11 -------

I
I
I
I I
I
I
I
r---- For j:: 1 ton ----. I
I I
I I
I I
I I
I
I I
I I
I I I
I
I I
I I Fo No I I
I I each
I
I
I vector I I
I I
I I I
I
Yes I I
I I
I I
I I
I I
I I
Activate input units I
I I
I I
I I x,== s,
I I I
I
I I
I I
I I
I I
I I
I I
I I
I I
I I
I I
I
I
I
I I
Weight adjustment
w,(new) =w,(old)+x,~
'

l
I
I
I
Continue
"---- J

Continue

Figure 44 Flowchan for training of auroassociarive ner.

4.3 Autoassociative Memory Network 103

I 4.3.4 Training Algorithm

The uaining algorithm discussed here is similar to that P,iscussed in Section 4.2.1 but there are same numbers
of output units as that of the input units.

\ Step 0: Initialize all the weights to~-- l

Wij = 0 (i = I ron, j = I m n)
Step 1: For each of the vector that has w be stgred perform Steps 2-4.
Step 2: Activate each of the input unit,
f' ~ .,/'~ \'
x1 = (i = I to n) \"
rP oV ~s'''
Si
~~
Step 3: Activate each of the output unit, \.9 {_Q ~)& ()) '

y_; = s; {j = I ton) ;\ / ,r'

,, ,,It
Step 4: Adjust the weights, \ \V
w;;(new) = wij(o\d) + x;y1
The weights can also be determined by the formula
/'
w= L:)iplf(pi
r= I ; .!

I 4.3.5 Testing Algorithm

An amoassociative memory neural network can be used to determine whether the given input vector is a
"known'' vecror or an "unknown" vector. The net is s~id to recognize a "known" vector if the net produces a
pattern of acrivation on rhe ourEm units which is s.iffie ~-~~~ -~[ ihe vectors srored iri ir."The n!stingprocedure
-. -----
of an auro.as50Ciatlve neural net is as follows:
.... -.. "-------~ - --
Step 0: Ser rhe weights obtained for Hebb's rule or outer products.
Step 1: For each of the resring inpm vector presented perform Sreps 2---4.
Step 2: Ser the activations of the input units equal to rhat of input vecror.
Step 3: Calculate the net input ro each ourput unit j = I ton:

~ ]itJj = L:" XjiVij

i=l

Step 4: Calculate rhe output by applying the activation over the ner input:

+1 if y,., > 0
Jj=ft:J,.)= -1 r. <o
1 I ]mJ-
1

This rype of network can be used in speech processing, image processing, pattern classification, etc.
104 Associative Memory Networks

I 4.4 Heteroassociative Memory Network

I 4:4.1 Theory
In case of a hereroassociarive' neural ner, rhe training input and clte target ourput vectors are different. The
weights are determined in a wa that rhe net can store a set of attern associations. The association here
is a pair of training input target ourpur vector pairs (s(p), t(p)), with p ch vecror s(p) has n
componems and each vecmr t(p) has m components. The determination of weights is done either by using
Hebb rule or delra rule. The ner finds an appropriate output vecmr, which corresponds to an input vector x,
rhar ~ne of rhe stored pauerns or a new pauern.

I 4.4.2 Architecture

The archirecwre of a hcreroassociarive net is shown in Figure 45. From rhe figure, ir can be noticed that for
a hereroassociarive net, the training input and mrget output vecmrs are differenr. The input layer consists of
n number of input units and the output layer consists of m number of output units. There exist weighted
imerconnections berween the input and output layers. The input and output layer units are nor correlated
~ith each other. The flowchart of the training process and the 7iauung a:ig&frli:m are the same as discussed m
Section 4.2.1.

I 4.4.3 Testing Algorithm

The tesring algorithm used for resting the heteroassociarive net wirh either noisy input or with known input
is as follows:
-------------1
1 Step 0: Initialize the wei~hts tTom rhe training algorithm.
Step 1: Perform Steps 2-4 tOr each input vector presented.
Step 2: Set the auivarion tOr in pur layer units equal ro that oF the current in pur vecror given, x;. I

,, ( x, --;.y
w, (Y,I_______.. Y1
W-.

w,.
I

,,
x, )<:( ) ( .:'.'! ?"( f---~Y,

Xn .(xn .....,.,_r
\ - - - - - Ym

Figure 45 Archirecrure ofheteroassociarive ner.

4.5 Bidirectional Associative Memory {BAM) 105

Step 3: Calculate the net input to rhe output units:

"
Yinj = LxJWij (j = 1 tom)
i=l
. \1'
Step 4: Determine the activations of the output units ov~r the calculated net input:)('
,
Jj =
1 if Y>i >.0
0 if y,.;=O
.
~,("""
''J
{
I . -~o I
Thus, the output vector y obtained gives the pattern associated with the input vecror x.
Note: HeterottJsociative memory is not an iterative memory network. Ifthe rerponses ofthe net are binary, then the
activation function to be used is - - ---
--------
1 if }inJ~O :--\ ~~I'\: , ,. r< ,
Jj=
1
--=
0 if y;,'.i<O
1
\~r-)<:1' ' 'Cf"'
'\'- ' .L-. -\.'

>., . , '
-">

.--~~- .,eL' \-< ,, ~~,.'

4.5 Bidirectional Associative Memory (BAM)
'
I 4.5.1 Theory

The BAM was developed by Kosko in the ear 1988. The BAM network performs forward and backward
associative searches fm{s[o~timulus responses. :fhe BAM is a recurrent heteroassociati_ve pattern-marching
nerwork that encodes~~~~ or 1p0 ar patterns using Hebbian ~g rule. It associates patterns, say from
set A to patterns from set B and v1ce versa is also performed. BAM neural nets can respoild to input from
either layers {input layer and output layer). There exist two types of BAM, called discrete and continuous BAM.
These two types of BAM are discussed in the following sections.

I 4.5.2 Architecture

The architecture of BAM network is shown in Figure 46. It consists of two layers of neurons which are con-
nected by directed weightclparh interconnecrions The network dynamics involve rwo layers of interaction.
The BAM ncrwork iterates by sending the signals back and forrh between the cwo layers until all the neurons
reach equilibrium. The weights associated with the network are b1duecnona:I. Thus, BAM can respond to
the mputs m either layer. Figure 4-6 shows a single layer BAM network consisting of n units in X layer and
m units in Y layer. The layers can be connected in botli dm:ctiOIIS (bidirectional) with the result the weight
matrix sent from the X layer to theY layer is Wand the weight matrix for signals sent from theY layer to the
~~e~ i~ wT. Thus, theWeigln matnx IS calcWareO in borh directions.

4.5.3 Discrete Bidirectional Associative Memory

The str~cture of discrete BAM is same as shown in Figure 4-6. When the me~ory neu~~ns ~Q_gn~acrivared
by punmg an initial vector at the i~J!!._Q_(a.l~, rhr neoyork evolves a\&oPiittern Sra\:lle state,With each
_pauern at the mJtp''r of ope I~Vc(Thus, the network involves two layers of interaction berween each orher
106 Associative Memory Networks

-
x layer y layer
x, w
)--~YJ

x, )---~ Y;

x,
)--~Ym
~w'

Figure 46 Bidirectional associuive memory net.

The rwo bivalent forms of BAM are found to be related with each other, i.e., binary and bipolar. The weights in
both the cases are found as the s oducrs of the bipolar form of rhe g1ven training vecror
In case of BAM, de m1te nonzero rhresho is ass1gne . Thus, the acnvauon uncuon IS a step function,
with the defined nonzero threshold. 'When compare , to the binary vccwrs, bipolar vectors improve ilie
pe~~<!E-to~nt. --------=---
4.5.3.1 Determination of Weights
Let the input vectors be denoted by s(p) and target vectors by t(p). p = 1, ... , P. Then the weight mauix to
store a set of input and target vectors, where

s(p) = (sJip), .. , s;(p), ... , s,(p))

t(p) = (1 1 (p), .. , rj(J>), ... , t,(p))

can be determined by Hebb rule training a1gorithm discussed in Section 4.2.1. In case of input vectors being
binary, the weight matrix W = {wy} is given by

p
Wij = L [2s;(p)- 1][2t;(p)- 1]
p=l

On the other hand, when the input vectors are bipolar, the weight matrix W = {wij} can be defined as

Wij = L s;{p)t;{p)
p=l

-
The weights matrix in both the cases is going to be in bipolar form neither rhe inpm vectors are in
binary or not. The formulas mentioned above can be directly applied ro the determination of weights
of a BAlvl.

l
4.5 Bidirectional Associative Memory (BAM) 107

4.5.3.2 Activation Functions for BAM

The step activation function with a nonzero rhresho ion function for discrete BAlvl
ne r . e acnvacion function is based on whether the input target vector pairs used are binary or 1po ar.
I he activation function for theY layer ,
1. with binary input vectors is

{ 1 if J;.;> 0
Jj= Jj ~f Yii=O
~ 0 1f ]inj< 0
2. with bipolar input vectors is

{ 1 if J;.;>Bj
Jj = Jj ~f ]inj =9j
-1 If ]inj< 9j

The activation function for the X layer

1. with binary input vectors is

{ 1 if x,.,> 0
x,= x, ~fx;11 ;=0
Ej) 1f X;11;<0

2. with bipolar input vectors is

{ I if x;.;> 9;
X1 = Xjif X;11 ; =9;
-1 if x;.;< e;
lt may be noted that if the threshold value is equal to ~ar of rhe net in pur calculated, then the previous output
value calculated is left as the activation of that unii At a particular time instant, signals are senr only from
one layer to rhe other and not in both the direction?

4.5.3.3 Testing Algorithm for Discrete BAM

The testing algorithm is used to test th~ n_g1zy pancrfi51nrering into the network. Based on the training
algorithm, weights are determined, by means of which net input is calculated for the given test pattern
and activations is applied over it, to recognize the test panerns. The testing algorithm for the net is as
follows:

Step 0: Initialize the weights to srore p vectors. Also initialize all the activations to zero.
Step 1: Perform Steps 2-6 for each testing input.
Step 2: Ser the activations ofXlayer to current input pauern, i.e., presenting the input pattern x to X layer
and similarly presenting the input pattern y w Y layer. Even though, it is bidirectional memory,
at one rime step, signals can be sent from only one layer. So, either of the input parterns may be
the zero vector.
 108 Associative Memory Networks

Step 3: Perform Steps 4-6 when the acrivacions are not converged.
Step 4: Update the activations of.units in Y layer. Calculate the net input,

"
]u.j= Lx;wij
i=l

Applying ilie activations (as in Section 4.5.3.2), we obtain

Yj: f(y,,;)
Send rhis signal to the X layer.
Step 5: Updare the activations of unirs in X layer. Calculate the net input,
m
X;,;= LYjWij
j=l

Apply the activations over the net inpur,

x; = [(x,-~;)

Send this signal to theY layer.

Step 6: Test for convergence of the net. T~e convergence occu tion vectors x and
equilibrium. If this occurs then stop, O(herwise, continue.

I 4.5.4 Continuous BAM

A continuous BAM transforms the inpur smoothly and continuously in the range 0-1 using Io_g:ri~
_functions as the activation functions for all unirs. The logis~ott-may-be-eith
sigmmdiJ ~ncrion or b1polaf sJgmdfciaHUftenen. W11en a bipolar sigmoidal function wirh a high gain is
chosen, chen the continuous BAM might converge to a state of vecmrs which will appro~ci~,;-es--ofth~
cube_,__Wheflrhat state oftlle vector approaches it"iCr.Slike-;cd~A.M~------- -- --
If rhe input vectors are binary, (s(p), t(p)), p = I to P, the weJgh{s-~e determined using the formula
''
'..0. ~; p
',. .' '.,' \Y'<,,il,,,: "[2s1(p) ~ 1][2t ,(p) ~I]
"'
'
~ -: . \l.'
'.
'-
~-
tct_-
(..
~'Jr ~
p=l
1

i;e., even though rhe inpUt ~~hors are binary, the weight matrix is bipolar. The activation function used here
is rh_e logistic sigmoidal function. If it is binary logistic function, chen the activation function is

I
f(y;n) = l + e-Yi"j
If the activation fimccion used is a bipolar logistic function, chen rhe function is defined as

2 I- e-Yin~
f(y ) ~~--~l:~j
i11j -l+r-y;,'l l+e
4.5 Bidirectional Associative Memory (BAM) 109

These activation functions are applied over the net input to calculate ilie output. The net input can be
calculated with a bias included, i.e.,

Yini = bj + 'EXjWij
"'' \, \' ~ ~-, ""'')
"'/
j'

and all these formulas apply for the units in X layer als~. )
4.5.5 Analysis of Hamming Distance, Energy Function and Storage Capacity

The hamming distance is defined as the numter of mismatched components of two given bipolar or binary
vectors. It can aJso be defined as the number of different bits in rwo binary or bipolar vectors X and X'. It is
denoted as H[X,X']. The average hamming distance between the vectors is (1/n)H[X,X'], where "n" is the
number of components in each vector. Consi<ler the vectors,

X: [I 0 I 0 I I 0] and X': [I I I I 0 0 I]

The hamming distance betw.et:n these two given v-ectors is equal to 5. The average hamming distance between
the corresponding vectors is 5/7.
The stabilicy analysis ofa BAM is based on the definition ofLyapunov function (energy function). Consider
that there are p vecmr association pairs to be stored in a BAM:

{(x' ./). <<'./),, .. ' (,/,/)}

where :! = <4, 4, ...
,x!,)T and I
= (;{,;4, .... /,;)T are either binary or bipolar vectors. A Lyapunov
function must be always bounded and decreasing. A BAM can be said to be bidirectionally stable if the state
converges to a stable point, i.e., I-)- /+
_1+ 1 -)- j+ 2 and 2 = /. This gives the minimum of the energy
function. The energy function or Lynapunov function of a BAM is defined as

-1 T ;r l T T
Er(x,y): 2" W y ~ ly Wx: ~y Wx

The change in energy due to the single bit changes in both vectors y and x given as D.y; and D.xj can be
found as

D.Er(y;): 'Vyf!D.y;: ~ WxD.y; =c ~ (t, xjwij) x D.y;, i: 1 ro n

D.Ej(Xj} = 'VxED.Xj = - WTyb..xj = - (ty;wij) X l:!..xj, j =l tom

1=1
where !:::..y; and !:::..xj are given as

. 2 if "'
LxjWij>O
2 if LYiWji> 0
i=l j=l
m
D.xj:!O
"
if Ly;wp = 0 and 6.y;= 0 if LxjWij=O
i=l j=l
m
~2 if "
LJiWji< 0 ~2 if Lxjwij<O
i=l j=l
110 Associative Memory Networks

tiere the energy function is bounded below by

Ej(x,y) ~ - L L"' lwijl

j=:[ j=l

so the discrete BAM will converge to a stable state.

The memory capacity or rhe stOrage capacity of BAM may be given as

min(m, n)

where "n" is the number of units in X layer and "m" is rhe number of units in Y layer.AJso a more conservative
capacity is estimated as follows:

Jmin(m,n)

I 4.6 Hopfield Networks

John J. Hopfield developed a model in rhe year 1982 conforming to rhe asynchronous nature of biological
neurons. The networks proposed by Hopfield are known as Hopfreld networks and it is his work that promoted
consrruccion of rhe first a~alog VLSI neural chip. This network has found many useful applications in
associative memory and various optimization problems. In this section, rwo types of network are discussed:
discrete and continuo/IS Hop}ield networks.

I 4.6.1 Discrete Hopfield Network

The Hopfield nerwork is an autoassociative fully interconnected single-layer feedback nenvork. It is also a
symmetrically weigh red nenvork. When chis is operated in discrete line fashion it is called as d;screte Hopfield
network and irs architecture as a single-layer feedback network can be called as recurrent. The network rakes
rwo-valued inputs: binary (0, 1) or bipolar (+l, -1); chc use of bipolar inpurs makes rhe analysis easier. The
network has symmetrical weights with no self-connections, i.e.,

Ulij = wp; !Vii =0

The key points to be nored in Hopfield net are: only one unit updates its activation at a time; also each unit
is found to continuously receive an external signal along wirh dte signals it receives from the other units in
rhe ner. When a single-layer recurrent network is performing a sequencia! updating process, an input pattern
is first applied to the network and the network's output is found co be initialized accordingly. Afterwards,
rhe initializing pattern is removed, and the output that is initialized becomes the new updared input through
rhe feedback connections. The first updated input forces the first updated output, which in turn acts as
rhe second updated input through the feedback interconnections and results in second updated output.
This transition process continues unci! no new, updated responses are produced and the network reaches irs
equilibrium.
The asynchronous updacion of ilie units allows a function, called as energy functions or Lyapunov function,
for che nee. The existence of chis function enables us to prove that the net will converge co a stable set of
activations. The usefulness of content addressable memory is realized by ilie discrete Hopfield net.
 111
4.6 Hopfleld Networks

x, x, x,
x,

y, y,
y, y,
Figure 47 Archirecrurc of discrete Hopfield ncr.

4.6.1.1 Architecture of Discrete Hopfield Net

The architecture of discrete Hopfield net is shown in Figure 4-7. The Hopf1eld's model consim of processing
elements with nvo outputs, one invening and the mher non-inverting. The omputs from each processing
element are fed back ro the input of other processing dements bur nor to itself. The connections are found
to be resistive and the connection srrength over it is represented as Wij. Here, as such there are no negative
resistors, hence excitatory connections use positive outputs and inhibitory connections use inverted outputs.
Connections are excitatory if rhe omput of a prncessing element is found to be same as the input, and they are
inhibitory if the inputs differ from the output of the processing element. A connection benveen the processing
elemencs i and j is found to be associated with a connection suength Wij This weight is positive if units i and
j are borh on. On the ocher hand, if the connection strength is negative, it represents rhe situation of unit i
being on and j being off. Also, the weighrs are symmetric, i.e., the weights Wij are same as wp.

4.6.1.2 Training Algorithm of Discrete Hopfield Net

There exist several versions of che discrete Hopfield net. It should be noted rhac Hopfield's first description
used binary input vectors and only later on bipolar input vectors used.
For storing a set of binary patterns s(p), p = l to P, where s(p) := (st (p), ... , s;(p), ... , s,(p)), the weight
matrix W is given as
p
Wij = L [2s;lp)- Ill2,jlp)- !], fo, i # j
p=d
 112 Associative Memory Networks

For storing a set of bipolar input patterns, s{p) (as defined above), the weight matrix Wis given as
p
Wij = L r;(pls;(p), fori I j
p=l

and the weights here have no sdf-connection, i.e., Wij = 0.

4. 6. 1.3 Testing Algorithm of Discrete Hopfie/d Net
In the case of testing, the update rule is formed and the initial weights are those obtained from the training
algorithm. The testing algorithm for the discrete Hopfield net is as foilows:

Step 0: Initialize the weights to srore patterns, i.e., weights obrained from trainingalgoridun using Hebb
rule.
Step 1: When the activations of the net are not converged, then perform Steps 2-8.
Step 2: Perform Steps 3-7" for each input vector X.
Step 3: Make the inirial activations of the net equal m the external input veaor X:

y;=x;(i= I ron)

Step 4: Perform Steps 5-7 for each unitY;. (Here, the units are updated in random order.)
Step 5: Calculate the net input of the network:

];,., = x; + L:JjWji
j

Step 6: Apply rhe activations over the net input to calculate the output:

1 if y;,,> e;
]i = y; ~f ]ini :::: e,.
1 If
Q ]inl < (};

where 9; is the threshold and is normally taken as zero.

Step 7: Now feed back (transmit) the obtained outputy; to all other unit!i. Thus, the activadon vectors
are updated.
Step 8: Finally, test rhe network for convergence.

The updation here iS carried out at random, but it should be noted iliac each unit may be updated at the
same a\erage rate. The asynchronous fashion of updation is carried out here. This means that for a given time
only a single neural unit is allowed to update its output. The next update can be carried out on a randomly
chosen node which uses the already updated output. It can also be said that under asynchronous operation of
dte network, each output node unit is updated separately by taking into accoUnt the most recent values that
have already been updated. This type of updacion is referred to as an d.J)Inthronous stochastic recursion of the
discrete Hopfield network By performing the analysis of the Lyapunov function, i.e., the energy function for
rhe Hopfield net, it can be shown that the main fearure for the convergence of iliis net is che asynchronous
updation of weights and the weight5 with no self-connection, i.e., the zeros exist on dte diagonals of the
weight matrix.

'--
l
 113
4.6 Hopfield Networks

A Hopfield network wiili binary input vectors is used tO determine whedter an input vector is a "known"
vector or an "unknown" vector. The net has rhe capacicy to recognize a known vector by producing a panern
of activations on the unit5 of the net that is same as ilie vector stored in rhe nee. For example, if ilie input
vector is an unknown vector, the activation vectors 'resul~ed during iteration will converge to an activation
vector which is not one of rhe stored patterns; such apa~er~ is called as spurious stable state.

4. 6. 1.4 Analysis of Energy Function and Storage Capacity on Discrete Hopfield Net
An energy function generally is defmed as a function that is bounded and is a nonincreasing function of the
stare of fie system. The energy function, also called as Lyapunov function, determines the stability property
of a discrete Hopfield network. The state of a System for a neural network is the vecmr of activations of the
units. Hence, if it is possible to find an energy function for an iterative neural net, dte net wiU converge to a
stable set of activations. An energy function Etof a discrete Hopfield network is characterized as

I n " '' n
Et= -:z LLJ;Yi Wij- Lx;y;+ z=e;y;
i=l }=1 i=l i=l
j-f=i

If dte network is stable, chen the above energy function decreases whenever rhe state of any node changes.
Assuming that node i has changed its state from y~k) co y~k+l), i.e., che output has changed from +1 to -1 or
from -I to + 1, the energy change b.EJis then given by

(k+l)) ( (!))
!lEt= Et (Yi - Et y;

- (tyj'l w' + .<;- e;) (y!'+'i-

J=l
y~'l)
j#
=- (net;) b. y;
where 6. y; :::: y)k+l, - jk). The change in energy is dependent on the facr that only one unir can update irs
. . . Th h .
acnvanon at a nme. . I . h ' h ik+ll
e c ange m energy equanon u'Efexp OJts t e ract t at J'i = Jj(II 10r
'
J. -"
r 1. an d
Wij = Wji and w;; = 0 {symmetric weight property).
There exist nvo cases in which a change{). y; will occur in the activation of neuron Y;. Ify; is positive, then
it will change co zero if

[
x; + ty;w;;] < e;
J=l

This results in a negative change for y; and D.EJ< 0. On the other hand, if y; is zero, rhen it will change ro
positive if

[
x;+ ty;w;;] > e;
J=l

This results in a positive change for y; and 6.Ej< 0. Hence 6. y; is positive only if net input is pomive and
/:::,. ]i i5 negative only if net input is negative. Therefore, the energy cannot increase in any manner. As a result,

l
 :1
Associative Memory Networks
114

because the energy is bounded, dte net must reach a smble state equilibrium, such that the energy does not
change with further iteration. From this it can be concluded that the energy-change depends mainly on the
change in activation of one unit and on the symmetry of weight matrix with zeros existing on the diagonal.
A Hopfield network always converges to a stable state in a finite number of node-updating steps, where
every stable state is found to be at the local minima of the energy function Ef Also, the proving process uses
the well-known Lyapunov stability theorem, which is generally used tO prove me stability of dynamic system
defined with arbitrarily many interlocked differential equations. A positive-definite (energy) function Ej (y)
can be found such that:
1. Et (y) is continuous with respect to all the components y; for i = 1 to n;
2. d Ef[y(t)]ldt< 0, which indicates iliar the energy function is decreasing with time 3J).d hence the origin of
the state space is asymptotically stable.
Hence, a positive-defmite (energy) function Ef (y) satisfying the above requirementS can be Lyapunov function
for any given system; this function is not unique. If, at least one such function can be found for a system, then
the system is asymptotically stable. According to the I yapunov theorem, the energy function that is associated
with a Hop field nerwork is a 4'apunov function and rhus the discrete Hop field nerwork is asymptotically
stable.
The storage capaciry is another important factor. It can be found that ilie number of binary patterns rhar
can be srored and recalled in a nerwork wiili a reasonable accuracy is given approximately as

Storage capacity C:::: 0.15n

where n is the number of neurons in the neL h can also be given as

II

c=:2log2 71

'.
': I 4.6.2 Continuous Hopfield Network

A discrete Hopfield ncr can be modified to a continuous model, in which time is assumed to be a continuous
variable, and can be used for associative memory problems or optimization problems like traveling salesman
problem. The nodes of this nerwork have a continuous, graded output rather than a rwo-sratc binary ourpur.
Thus, rhe energy of the network decreases continuously with time. The continuous Hopfield networks can
be realized as an electronic circuit, which uses non-linear amplifiers and resistors. This helps building the
Hopf1eld nerwork using analog VLSI technology.

4.6.2.1 Hardware Model of Continuous Hopfie/d Network

The continuous necwork build up of electrical componems is shown ~n Figure 4-8.
The model consists of n amplifiers, mapping itS input voltage u; into an output voltage y; over an activation
function a(uJ The activation function used can be a sigmoid function, say,

I
a(Au;) =" 1 + e-i.u;

where), is called rhe g3.in paramerer.

The continuous modd becomes a discrete one when A-+ Ct. Each of ilie amplifiers consistS of an input
capacitance c; and an input conductance gn. The external signals emering into rhe circuit are x;. The external

ll
1
I
4.6 Hopfield Networks 115

x, X, X. X.

w. w., w.

w, w. w

w,

w, w, w,

~ to lD Y, j Y,
[:?;. ~ 91
Y,
gr,

Y, Y, Y.
I
Y,

Figure 48 Model ofHopfleld network using elecnical componenrs.

signals supply constant current to each amplifier for an actual circuit. The output of the jrh node is connected
to the input of the ith node through conductance IVij. Since all real resistor values are positive, the inverted
node outputs J; are used to simulate the inhibitory signals. The connection is made with the signal from the
noninverted output if the output of a particular node excites some other node. If rhe connection is inhibitory,
then the connection is made with the signal from the inverted omput. Here also, rhe important symmetric
weight requirement for Hopfield nerwork is imposed, i.e., Wij = Wji and w;; = 0.
The rule of each node in a continuous Hopfield network can be derived as shown in Figure 4-9. Consider
the input of a single node as in Figure 4-9. Applying Kirchoff's current law (KCL), which states that the total
current entering a junction is equal to that leaving the same function, we get

C; idut = L"
j=l
Wij (yj- u;)-
n
gr,u;+x; =I:
j=l
WijJj- G,u; +x;
j#i j=foi

j"'
~ ~
 --:
116 Associa!ive Memory Networks

Y, Y, Y,

w.,l w,:;l w,:;l

l
(y,-ul)w,l I ~~:-uJw2, !(yn-ul)w,. u

x, I I ~~
lgr,
1'1
I
-grpl
cp{u)
df

"
Figure 49 Input of a single node of continuous Hopf1eld nerwork.

where

"
G;= Lw,i+gr;
J=l
Jf:i

The equation obcained using KCL describes the rime evolution of the system completely. If each single
node is given an initial value say, u;{O), then the value tt;(t) and thus the amplifier outpur, y,'(t) = a(u,.(t)) at
timer, can be known by solving rhe differential equation obmined using KCL

4.6.2.2 Analysis of Energy Function of Continuous Hopfield Network

For evaluating rhe stability property of conri~uous Hopfield nep.vork, a continuous energy function is defined
such thar the evolution of the system is in the negative gradient of rhe energy function and finally converges
to one of the table minima in the srare space. The corresponding Lyapunov energy function for the model
shown in Figure 4~S is

I
II II II II )I

Er= -2 LL'"ijYiYj- LXiYi+ ~ I:c,

1 1
n-'(y)dy
i=l J=l i=l i=l 0
f'Fi

where a- 1(y) =Au is the inverse of the function y = a(Au). The inverse of the function a- 1(y) is shown in
Figure 4--lO(A) and the integral ofir in Figure 4~10(B).
To prove rhar Erobuined is rhe Lyapunov function for the nerwork, irs rime derivative is taken with
weighrs W1i symmetric:

dE!=
dt .
t
1=1
dE dy;
dy, dt
= L" (-"LYiw,; + Gilti- Xi )dy,dt =
i=l J=l
j=/=i
-""' dy,dw
~c;-_!_
i=l dt dr

L
4.6 Hopfield Networks 117

y
j.- (y)dy
U= ,g-l(y)

0.5 +1 y

~~~--~~~--+X
0 -1 ~
(A)

Figure 410 (A) Inverse and (B) integral of nonlinear acrivation function a- 1(y).

A;

"i =G) a-'(y,)

we get

dtti 1 ,Ja-l (y,) dy; 1 -I'

- = - - - - = - a (y,)-
dy.
dt ). dy; dt ). dt
where the derivative of IC 1(y) is a-l' (y). So, the derivative of energy function equation becomes

dt ~ IC;fl-
dE! =- !- l 1 (dy')'
(y;) dt
t=l

From Figure 4~ 1O(A), we know that [ 1(y;) is a monotOnically increasing &merion ofJi and hence its derivative
is posirive, all over. This shows that dErldt is negative, and dms rhe energy function Et must decrease as rhe
system evolves. Therefore, if Etis bounded, the system will evemually reach a stable scare, where

dEJ dy,
-=- =0
dt . dt
When the values of threshold are zero, the continuous energy function becomes equal to the discrete energy
function, except for the rerm,

~L " G, !'' n- (y)dy

1

A i=l o

From Figure 4-lO(B), rhe integral of a- 1(y) is zero when y; is zero and positive for all other values of Ji
+
The integral becomes very large as y approaches 1 or -I. Hence, the energy funcrion Et is bounded
from below and is a 4'apunov function. The continuous Hop field nets are best suiced for the constrained
optimization problems.
 118 Associative Memory Networks

I 4.7 Iterative Autoassociative Memory Networks

There exists a situation where the nee does not respond to che input signal immediately with a stored rarger
pattern but the response may be more like the stored panern, which suggests using the fim response as inpur
to the net again. The iterative auroassociacive net should be able co recover an original stored vecmr when
presented with a test vector dose to it. These cypes of networks can also be called as recumnt autoassociarive
networks and Hopfield networks discussed in Section 4.6 come under this category.

I 4. 7.1 Linear Autoassociative Memory (LAM)

In 1977, James Anderson focused on the developmem of the LAM. This was based on Hebbian rule, which
scares that connections between neuron like elements are strengthened every time when they are activated.
Linear algebra is used to analyze the performance of the net.
Consider an m X m non singular symmetric matrix having "m" mutually orcltogonal eigenvectors. The
eigenvectors satisfy the properry of onhogonaliry. A recurrent linear autoassociator network is uained using
a set of P orthogonal unit vector u,, ... , up, where the number of times each vector going to be presented is
nor the same.
The weight matrix can be determined using Hebb learning rule, bur this allows the repetition of some of
the stored vectors. Each of these srored vectors is an eigen vector of the weight matrix. Here, eigen values
represent rhe number of times the vector was presented.
When the input vector X is presented, rhe output response of rhe net is XW. where Wis the weight matrix.
{:. From the concepts oflinear algebra, we know that we obtain rhe largest value of IIXWll when Xis the eigen
vector for the largest eigenvalue; the next largest value of IIXWII occurs when Xis the eigenvector for the next
~: largest eigenvalue, and so on. Thus, a recurrent linear autoassociamr produces irs response as the stored vector
' for which the input vecmr is most similar. This may perhaps rake several iterations. The linear combination
I ,., of vecrors may be used to represent an input pattern. When an input vector is presented, the response of rhe
,,
\'" net is the linear combination of irs corresponding eigen values. The eigen vector with largest value in this
linear expa~sion is the one which is most similar ro char of the input vectors. Although, rhe net increases
irs response corresponding ro components of the input pattern over which iris trained most extensively, the
overall output response of the system may grow without bound.
The main conditions oflineariry between the associative memories is that the set of input vector pairs and
outpm vector pairs (since, auroassociative, both are same) should be mutually orthogonal with each other,
i.e., if''A/ is the input pattern pair, for p = I toP, then
T
A;Aj = 0, foralli-:f:.j

Also if all the vectors Ap are normalized to unit length, i.e.,

"
L(a,)~ = 1, forallp= I toP
i=l

then the output Yj = Ap, i.e., the desired output has been recalled.

I 4. 7.2 Brain-intheBox Network

An extension to ilie linear associator is the brain-iiJ-the-box model. This model was described by Anderson,
1972, as follows: an acriviry pattern inside the box receives positive feedback on cenain components, which

i.
 4.7 Uerative Autoassociative Memory Networks
119

has the effect of forcing it outward. When irs element stan: to limit (when it hits the wall of the box), ir
moves to corner of the box where it remains as such. The box resides in the state-space (each neuron occupies
one axis) of the network and represents the saruraiion 'lj~its for each state. Each component here is being
restricted between -1 and +1. The updation of acciva~ions of the units in brain-in-the-box model is done
simultaneously. ,
The brain-in-the-box model consists of n units, each being connected to every oilier unit. Also, there
is a trained weight on rhe self-connection, i.e., the diagonal elements are set to zero. There also exists a
self-connection with weight 1. The algorithm for brain-in-the-box model is given in Section 4.7 .2.1.

4. 7.2.1 Training Algorithm for Brainin-theBox Model

Step 0: Initialize the weights to very small random values. Initialize the learning rates ct and {:J.
Step 1: Perform Steps 2-6 for each training input vector.
Step 2: The initial activations of the net are made equal to the external input vector X:

y;=x;

Step 3: Perform Steps 4 and 5 when the activations continue to change.

Step 4: Calculate rhe net input:

"
y;,i = y;+a LYjWji
j~l

Step 5: Calculate the output of each unit by' applying irs activations:

I if y,.,, > 1
J'j = y;,i if -1 Sy;.,iS I
{
-1 ify;.,j<-1

The venex of the box will be a stable srare for the activation vector.
Step 6: Update the weights:

Wij(new) = w;j(old)+,B JiYj

4. 7.3 Autoassociator with Threshold Unit

If a threshold unit is set, then a threshold fl.mction can be used as the activation function for an iterative
au.toassociator net. The testing algorithm of aumassociator with specified threshold for bipolar vectors and
activations with symmetric weights and no self-connections, i.e., Wij = Wji and Wii = 0 is given in the
following section.

j,;_
a.
 120 Associative Memory Networks

4. 7.3. 1 Testing Algorithm

Step 0: The weights are initialized from the training algorithm to store patterns (use Hebbian learning).
Step 1: Perform Steps 2~5 for each testing input vector.
Step 2: Set the activations of X..
Step 3: Perform Steps 4 and 5 when the stopping condition is false.
Step 4: Update rhe activations of all units:

if L" X;-Wij' > 9;

j=1

Xj;:::: X; if L" XjWij =8;

j=l

-1 if L" XjWij>8;
j=l

The threshold 81 may be taken as zero.

I Step 5: Test for the stopping condition. I
The nernork performs iteration until the correct vector X matches a scored vecwr or the testing input marches
a previous vector or clJe maximum number of iterations allowed is reached.

I 4.8 Temporal Associative Memory Network

The associative memories discussed so far evolve a stable state and stay there. All are acting as content
addressable memories for a set of static patterns. Bur there is also a possibilicy of storing the sequences of
patterns in the form of dynamic transitions. These rypes of patterns are called as tempomi patterns and an
associative memory with this capabilicy is called as a temporal associative memory. In this section, we shall learn
how rhe BAM act as temporal associalive memortes. Assume all temporal patterns as bipolar or binary vectors
given by an ordered set S with p vecmrs:

S= {sl,sz, ... ,Sj, ... ,spJ (p= l roPJ

where column vectors are nclimensional. The neural network can memorize the sequence Sin irs dynamic
state transitions such that the recalled sequence is s1 --)- sz ~ ... ~ s; ~ ... --)- sp --)- s1 --)- sz --)- ... -)o
s; --Jo or in reverse order.
A BAM can be used to generate the sequenceS::::: {s 1 , sz,, .. , s;, ... ,sp}. The pair of consecutive vecmrs Sk
and SJ:+l are taken as hereroassociative. From this point of view, SJ is associated with sz, sz is associated with
s3 , ... and Sp is again associated with s1 The weight matrix is then given as
p
W= L::<s<+Il(s,)T
k:=l

A BAM for temporal panerns can be modified so clJat both layers X and Yare described by identical weight
matrices W Hence, the recalling is based on

x =/(Wy); y =f(W,)

ll
4.10 Solved Problems 121

where f 0 is the activation function of clte network. Also a reverse order recall can be implemented using
the transposed weight matrices in both layers X and Y. In case of temporal BAM, layers X and Y update
nonsimuhaneously and in an alternate circular fashion.
The energy function for a temporal BAM can _be defin.ed as
p
Ej=- Lsk+ Wsk
1
k=l

The energy function /decreases during the temporal sequence retrieval s1 -+ S2 -+ ... -+ sp- The energy is
found to increase stepwise at rhe transition sp --)- s1 and rhen ir continues to decrease in rhe following cycle of
(p- 1) retrievals. The storage capacity of the BANI is estimated usingp ::: min(m, n). Hence, the maximum
length sequence is bounded by p < n, where n is number of components in input vecror and m is number of
components in output vector.

I 4.9 Summary
Pattern association is carried out efficiently by associative memory networks. The cwo main algorithms
used for training a pauern association network are the Hebb rule and the outer products rule. The basic
architecture, flowchart for training process and the training algorithm are discussed in detail for autoasso
ciative net, heteroassociative memory net, BAM, Hopfield net and iterative nets. Also, in all cases suitable
resting algorithm is included. The variations in BAM, discrete BAM and continuous BAM, are discussed
in this chapter. The analysis of hamming distance, energy function and storage capacity is done for few
nernrorks such as BAM, discrete Hopfield network and continuous Hopfield nernrork. In case of itera
rive autoassociative memory network, the linear auroassociarive memory, braininthe-box model and an
autoassociator with a threshold unit are discussed. Also temporal associative memory network is discussed
briefly.

I 4.1 0 Solved Problems

1. Trai9' a hereroassociarive memory network using

lje&b rule ro swre input row vector s :::::
/"'(sl, sz, s3, s4) to the output row vector t = (tl, tz). y,
The vector pairs are given in Table l.
Table 1
Input targets St sz S3 s4 l t1 tz y,

1" ---~-1 _a. 1 olL o

::
2"' 1 0 0

~ ~ ~-~ ~ftf~~f . I 1 0

"''
Solution: The network for the given pJoblem is J..'i
shown in Figure 1. The training algorithm based on
Hebh rule is used to determine the weights. Figure 1 Neural net.
122 Associative Memory Networks

For 1st input vector. For 3rd input vector:

The inpur-ourput vecror pair is (1, l, 0, 0):(0, I)
\ Step 0: Initialize rhe weights, rhe initial weights I Xj = 1, X2 = 1, X3 = 0, X4 = 0, Yl = 0, Y2 = 1
are taken as zero.
Step I' For first pair (I, 0, I. 0),(1, 0) Training, using Hebb rule, evolves the final weighrs
Step 2: Set the activations of input unirs: as follows:
Since Yl = 0, the weightS ofy1 are going m the same.
XJ. = 1, X2 = 0, X'3 = 1, X4 = Q ~ Computing the weighrs ofY2 unit, we obtain

w12(new) = WJ2(old) + XJ)'2 = 0 + 1 x 1 = 1

Step 3: Set the activations of ourput unit:

JJ=I, J2=0
'~ w,z(new) = w,z(old) +xm = 0 + 1x 1 = 1
' w;z(new) = w,z(old) + "3J2 = 0 + 0 x 1 = 0
~
Step 4: Update rhe weights, > w42(new) = W42(old) + X4]2 = 0 + 0 x 1 = 0

wij(new) = Wij(old) ~ "

The final weights after presenting third inpm vecror
are
w,,(new) = WJJ(old) +x1Y1 = 0--+, 1 x 1 = 1
WJJ =2, Wz] =0, w3 1 =I, WljJ = 1
W2J(new) = W2J(old)+X2YJ =o-Ho x 1 =0
I WJ2 = 1, WZ2 = 1, U/32 = 0, W42 = 0
W31(new) = W3I(old)+X3YI =Of 1 x 1 = 1
For 4th input vector:
W4J(new) = W4J(oJd) + X4)'1 = Q -t! Q X 1 = Q The input-output vector pair is (0, 0, I, I):(O, 1)
wn(new) = w12(old) +x1y2 = 0 + 1 x 0 = 0
XJ = 0, X2 = 0, X3 = l, X4 = 1, Yl = 0,)'2 = 1
W:22(new) = fll2z(oJd) + X2J2 = 0 + 0 X 0 = 0
The weights are given by
w~z(new) = w3z(old) + X3)'2 = 0 +I X 0 = 0

1 W4z(new) = W4z(old)+x4)'2 =.0+0 x 0 = 01 lll32(new) = w3z(old) + X3]2 = 0 +I x l = 1

w.u{new) = w42(old) +x4Yl = 0 + 1 x 1 = 1
For 2nd input vector: Since, Xi = "2 =]I = 0, the other weights remains
The inpm-ompur vecror pair is (I, 0, 0, 1):(1, 0) the same. The final weighlS after presenting the fourth
mput vector are
X] = }, X2.= Q, X3 = Q, X4 ;::; l,
WU = 2, W2J = 0, 1lJ31 = }, WljJ ::= }
JJ = 1, Yz = 0
w12 = I,wzz = 1,w32 = l,w42 =I
The final weights obtained for rhe input vecror pair
Thus, the weight matrix in matrix form is
is used as initial weight here:

WJt(new) = wn(old) +x1]1 = 1 + 1 x 1 = 2

w41(new) = W4J(old) +x4y1 = 0 +I x 1 =I \Y./ =
WI! WJZ]
WZI W22 =
[2 0 1
11
[ W31 W32 1 1
Since X2 = X3 = Y2 =
0, the other weightS remains WljJ W42 1 l
the same.
The final weights after second input vecror is pre~
semed are
r,~n the heteroassociative memory network using
. outer products rule to store input row vectors s =
WJJ = 2, W21 = 0, 1031 = 1, WljJ = 1 (sJ,Sz, s3,s4) ro lhe output row veaors t = (t 1, tz).
WJ2 ::= 0, W22 = 0, W32 ::= 0, Wlj2 ::= 0 Use he vector pairs as given in Table 2.

-
 4.10 Solved Problems 123

Table2
Input and targets
l"
,,
1 " "
0 I
,,
0
tj

I
"0
_ OJ
-
o
[~ 4xl
[o
1]\xz= [H] 0 1 4x2
2"' 1 0 0 1 1 0
3"' 1 1 0 0 0 The final weigllt matrix is the summation of all the
4'h 0 0 1 1 0 individual weight matrices obtained for each pair.

Solution: Use ~ to determine the 4

weight matrix: ~ W= L:l(p)t(p)
p
p=l

w= L:?(p) t(p) = ,T (1)1(1) + ,T (2)1(2) + l (3)1(3) + ,T (4)1(4)

[i ~] [~ ~] [~ i] [~ !]
p=l

For 1st pair: The input and output vectors are s =

(1 0 1 0), r = (1 0). For p = 1, = + + +

sT(p) t(p) =sT(I) 1(1)

= [~] [1 0] 1, , = [H] w= [~ \]

O 4xl 0 0 4x2 3. Train a heteroassociative memory network to store

the input vectors s = (sl, sz, 13, s4) to the output
For 2nd pair: The input and output vectors ares = vectors t = (t!, tz). The vector pairs are given in
(l 0 0 1), r =(I 0). For p = 2, Table 3. Also test the performance of the nelWork
,T (p) t(p) = ,T (2) 1(2) using its training input as testing input.

Table 3

= [i]
1 4xl
[1 l1x2 =, [H] 10
4x2
Input and targets
1"
2"'
SJ

l
1
"0
1
,, ,, ,,
0
0
0
0
0
0
"
l
3'' 0 0 0 l I 0
For 3rd pair: The input and output vectors ares = 4'h 0 1 1 l 0
0
(1lOO),r=(01).Forp=3, :::{\.\ ..
r,'J . ,.
,T (p) t(p) = ,T (3) 1(3) Solution: The ne[Work architecture for rhe given

[l~
1 [0
1Jix2 =
l
'[0~ ~
input-target vector pair is shown in Figure 2. Train~
ing the network means the determination of weights
of the network. Here outer products rule is used to
determine the weight.
The weight matrix W using ourer products rule is
4xl 0 0 4x2
given by
For 4th pair: The input and output vectors arc l =
(00 11), r= (0 1). Forp= 4, p

w= L:?(p) t(p)
i: /(p)t(p) =<'(4)1(4) p=l
-k
r'
-~
 124 Associative Memory Networks

Forp=lro4, For 1st testing input

w= ' t(p)
I>T(p) I Step 0: Initialize the weights: I
p=l

= ,T (1)~1) + ,T (2)~2) + ,T (3)~3) + ,T (4)~4) W=

wu
W2I
W3J
WJ2]
W22_01
.,,-10
[0 2]
[
W4J W42 2 0
=UJ[o I]+UJ[o 1] Step 1: Performs Steps 2--4 for each testing
input-output vecmr.
Step 2: Set the activations, x = [1 0 0 0].
Step 3: Compute the net input, n = 4, m = 2.
+[n[1 O]+UJ[I o] Foci= 1 m4andj= 1 to2:

'

[~ ~] [~ i] [~ ~] [! ~]
}inj= LxiWij
i=l

= + + +
'
}inl = Lx;WiJ
i=l

[! ~]
= Xlll!\1 + XZWZI + X'JW3J + X4fV4J
=lxO+OxO+Oxl+Ox2=0
W=
"
Yin2 = Lx;w;2
This is che final weight of rhe matrix. i=l
= XJ Jl/]2 + X2W22 + X3W32 + X.j1V41
=lx2+0xl+OxO+Ox0~2
,, Step 4: Applying activation over the net input to
calculate rhe output.

Jt = f(y;,J) = j(O) = 0
" I Y2 = f(y;,) = f(2) = I I
The ourput is [0, 1] which is correct re.<iponse for first
input panern.

For 2nd testing input

Figure 2 Network archirccrure. Ser the activation x = [1 I 0 0]. Computing the net
inpm, we obtain
Testing the Network
Method! Jinl =X]W]J +X2.tll21 +XJU/31 +X4W4J
The ces,ililg algorithm for a hereroassociarive mem~ =0+0+0+0=0
. I
ory network IS used ro cesr the performance of rhe
]inZ = X]WJ2 + XZWzZ + X3W32 +x4w42
net. The weight obtained from training algoriilim is
the initial weight in testing algorithm. =2+1+0+0=3

i
~
 125
4.10 Solved Problems

Compure the output by applying activations over net test performance of network. The initial weighS for

input, me ne;twork are

YI = f(y;,J) = f(O) = 0
Y2 = f(y;,) =f(3) = 1

The output is {0, 1] which is correct response for

W= [! i]
second input pattern. The binary activations are used, i.e.,

For 3rd testing input

(~
if x>O
Set the activation x = {0 0 0 1]. Compurl:ng net f(x) = if X:::
Q
input, we obtain
For I st usting input
}inl =XJW!J +X2tlf21 +X3W31 +X4W.j] Set the activation x = {1 0 0 0]. The nee input is
=0+0+0+2=2 given by ]in= xW(in vector form):

}in2 = X]W\2 +
=0+0+0+0=0

Calculate output of the network,

XZWz2 + X3W32 + X4W42

[y;,J y;,,] =[I o o Ohx4 [n]

2 0 4x2

YI = f(y;,J) = j(2) = I = [0 + 0 + 0 + 0 2 +0 +0 +OJ

Y2 = f(y;,,) = j(O) = 0 = [0 2]

The output is (1 0] which is correct response for third Applying activations over the net input, we get
testing inpm pattern. (n )'2] = [0 1]
For 4th testing input
Set the activation x = [0 0 l I]. Calculating the net The correct response is obtained for first testing input

input, we obtain pattern.

For 2nd teJting inpm

}ill\~ XJWII + XZWzl +x3w31 +X4W4l Set the activation x = [I 1 0 0]. The net input is
=0+0+1+2=3 obtained by
+ Xzrll22 + XVV3! + X41V42
[! 1]
}i11Z = X\ IVJ2

=0+0+0+0=0
\y;,,y,,]=[IIOO]
Calculate the output of the network,

]I = f(y;,J) = /(3) = I = [0 + 0 +(I+ 0 2 + I + 0 + 0]

]2 = f(y;,,) = /(0) = 0 = [0 3]

The output is [l 0] which is correct response for Apply activations over the net input co obtain output,
fourth testing input pattern. we get

Method II lvinl ]inzl == [0 l]

Since net input is the dot product of the input row
vecror with rhe column of weight matrix, hence a The correct response is obtained for second testing
method using matrix multiplication can be used to input.
],,_
\..
126 Associalive Memory Networks

For 3rd testing input The weight matrix is

Set the accivarion x = [0 0 0 1]. The net input is

~]
obmined by .

~'"' y;.z] = [0 0 0 I] [! i] W=[!

The net input is calculated for the similar vector,

= [0+ 0 + 0 +2 0 + 0 + O+ 0]
= [2 0]

Applying activations to calculate the output, we get

~,,, y;,] = [0 1 0 0] [! ~]
[y, }'2] = [1 0] = [0+0+0+0 O+ 1 +0+0]
= [0 1]
Thus, correct response is obtained for third testing
The output is obtained by applying activations over
input.
the net inpm
For 4th testing input
[y, }'2] = [0 1]
Ser the acrivationx = [0 0 I 1}. The net input is
calculated as The correct response same as che target is found,
hence the vector similar to the input vector is

~'"' [! i] recognized by the network.

y;,, 2 ] = [0 0 1 1] With mi.Simiiar-input vector. The second input
veccorisx=[l I 0 O]withrargety=[O l].To
test the nenvork with unsimilar vectors by making a
= [0 + 0 + I + 2 0 + 0 + 0 +OJ change in\two~COrii.ponen~ of che input vector, we gee
= [3 0] ---. -----
X= [Q 1 1 Q]
The output is obmined by applying activations over
The weight matrix is
rhe net in pur:

~J
[y, }'2] = [1 0]
The correct response is obtained for fourth test~
ing inpu( Thus, training and tesring of a hercro
W= [r
associ:itive necwork is done here.
The net input is ca.lculated for unsimilar vector,
4:''For Problem 3, test a hereroassociative network
with a similar test vector and unsimilar test
vector.
Solution: The heteroassociative network has to be
[y;,, y;,z] = [0 1; 0] [! ~}
tested with similar and unsimilar rest vecror. ,, = [0 + 0 + 1 + 0 0 + 1 + 0 + 0]
With j11#g_test vector: From Problem 3, the sec~ =[1 1]
ondinputvector isx = [I I 0 0] with targecy = [0 1].
'0 test the network with a similar vector, making a The output is obtained by applying activations over
chaitge m one compo~c of the input vector, we get the nee input

x=[0100] [y, }'2] = [1 I]

_
 127
4.10 Solved Problems

The correct response is not obtained when rhe recror W4z=-l xl+-1 x 1+1 x-1+1 x -1
unsimilar to the inpm network is presented to the =-1-1-1-1=-4
uecwork.
The weight matrix W is given by
(( 5. rrain a heteroassociarive network tO store the,
input vectors s = (s, sz 53 s4) to the output vec-
tor t = (tJ. t:z). The training input-target output W=
WII
W31
WJ2]
U121 WZ2
U/32
_
-
[-4 4]
-2
2
2
-2
vector pilrs are in binary form. Obtain the weight [ W4I W42 4 -4
vector in bipolar form. The binary vector pairs are
as given in Table 4. 6. T_?itl a heteroassociacive network to store the
;iven bipolar input vectors s = (sl s2 53 s4) to
Table4
,, ,, ,, the output vector t = {t, tz). The bipolar vector

1" 1 " "

0 0 0 0
" pairs are as given in Table 5.

TableS
2"' 1 1 0 0 0
0 ,, ,, ,, 12
3"'
4'h
0
0
0
0
0
1
1
1
1
1 0 1" 1
"
-1 -1 " -1 -1
2"' 1 1 -1 -1 -1 1
Solution: In this case, the hybrid represenmion of I -1
3'' -1 -1 -1 1
the network is adopted to find the weight matrix in 1 1 I -1
4'h -1 -1
bipolar form. The weight macrix:ca.n be formed using
,...- ~1 -' -~~ Solution: To store a bipolar vecmr pair, the weight
Wll = (2 X 1- 1}(2 X Q- 1)
'matrix is
+ (2 X 1 - 1)(2 X 0- I)
p
+ (2 X 0 -
+ (2 X Q-
I )(2
1)(2
X

X
I - 1)
1 - 1)
r. wu = L s;(p)tj(p)
p=l
= -1- 1- 1- I= -4 If the outer products rule is used, rhen
IVJl ;= (2 1 - 1)(2 I - 1)
+ (2
X

X
X

1 - 1)(2 X 1 - 1) W= L?(p)t(p)
p
+ (2 X 0- 1)(2 X 0- I) f
+(2x0-1)(2x0-1) For 1st pair
.:!
=1+1+1+1=4 >=[1 -1 -1 -1], ,=[-1 1]
fV21 =-1 x-1+1 X -1+-1 X I+-1 X 1
= 1 - 1 - 1 - I = -2
?(1)t(l) = [ ::\] [-1 1] = [-\ ::\]
W22=-l X 1+1 X l+-1 X-1+-l X-I
-1 1 -1
=-1+1+1+1=2
fV31 =-1 x-1+-1 X -1+-1 X l+l X I For 2nd pair
=1+1-1+1=2 s=[1 1 -1 -1], ,=[-1 1]
W32=-l X 1+-1 X 1+-1 X -1+1 X-I

, =; ::;]
I_ =-1-1+1-1=-2
I W4! =-lx-1+-1 X -1+1 X 1+1 X l 7(2)K2l = [ ::\] [-1 1] = [
I' =1+1+1+1=4

_1_ .
128 Associative Memory Networks

For 3rdpair Applying activations w compute the output, we gee

s= [-1 -1 -1. 1], t= [1 -1] [y, Y2l = H 11

Thus, the net has recognized the missing data.

/(3)3)=
[
=:
-1]
[1 -1]= =: _;
[-1 1]
With mistaken data: Let the rest vector be
[ -1 1 1 -1] wirh changes made in two
x
com-
ponents of second input vector [I 1 -1 -1].
For 4th pair Computing the net inpm for the rest vector, using
s=[-1 -1 1 1], t=[1 -1] the final weights obtained in Problem 6, as initial
weight ro rest the test vector, we get

/(4)4) = [ -:
-1]
[1 -1] = -:
[-1
=11] y;,.z] = [-1 1 1 _ 11
-4 4]
The final weight matrix is
[]in!
[ 4 -4
-2 2
2 -2

= [0 0]

4 T [-11 -11] -I
[-1 1
1]
W= ~' (p)t{p) = 1 -1 + 1 -1
Applying the activations over the net input to calcu-
late the output, we obtain
p-l 1 -1 I -1
(y, yz] = [0 0]
-1
-l I
1] -1
[-1 1
1]
+ [ -1 1 + I -1 Thus, the net does nor recognize the mistaken data
b~cause the output obtained [0, 0] has a m1~h
I -1 1 -1
wnh the target vector [ 1 I].
-4
-2 2 4] 8. Traift the aum~ssociarive network for in pur vecmr
- 2 -2 [ -l 1 1 1] and also rest the network for the
[ same input vecror. Test rhe auroassociarive net-
4 --i
yor/ Problem 6, resr the performance of rhe nee-
work with one missing, one mistake, two missing
and two mistake entries in rest vector.
"' work with missing and mistaken data in rhe test
vecmr. Solmion: The input vector is x = [-1 1 I 1].
The weight vccmr is
Solution: \'tlirh missing data
Let ilie rest vecror be x = [0 I 0 -I] wirh
changes made in [\VO com~onents of second inpm
vector [1
we get
1 -I -1]. Computing the net inpur, W= I>T(p)s(p) = [-i] [
1
-1 1 I 1],,,

-4 4]
-2 2
1 -1
-1 I
-]
I
-!]
I
lixl

[Jinl Yiuz] = [0 1 0 -1]

[ 2 -2
4 -4
=
[
-1
-1
1
1
I
1
I
1 lixlj

=[0-2+0-4 0 + 2 + 0 + 4]
Testing the network with same input vector: The rest
=H 6] input is [ -1 1 1 l]. The weight obtained above
 "';'I
4.10 Solved Problems 129 H

is used as the initial weight here. Computing the 1 -1 -1 -1] \i

input, we get
-1 1 1 1
=[-1 -1 1 1]
-1 1 1 1 !~
[
1 -1 -1 -1] -1 1 1 1 '(
y,.;=xW=[-1 1 1 1] -1 1 I 1 =[-2222]
-1 1 1 1
[ -1 1 1 1
=[-4444] Applying the activations, we get Yi = [ -1 1
, 1 1] which is the correct response.
Applying activations over the n~t input to calculate
the output, we have Test input x = [I 1 1 1]. Computing net
\(;~- ,1' input, we get
y=f(y,.)
" ~
=I 1 if y,.;>O \
-I
:'\','?
1f 1,"i<lo 'OI:J' ~
_:j ~'- 'f:l I -1 -1 -1]
= [-1 1 1 1] -1 1 1 1
y,,1 =xW=[I 1 1 1]
-1 1 1 1
[
Hence the correct response is obtained. -1 1 1 1
Testing the network with one missing entry =[-2222]

Test input x = [0 1 11]-:-compuring the Applying the activations, we get Yi = [ -1 1 .~

' ,4
input, we get 1 1] which is the correct response. ,
~

-1
1 -1 -1 -1]
Yi; =X. w = [0 1 1 1] -. 1 1 1 1
1 1 1
Testing the network with two miSsing entry

Test input x = [0 0 1 1]. Computing net

'I
[ input, we gcr
-1 1 1 1
=[-3333] .\
1 -1 -1 -1] I
Applying the activations, we get Yi = [-I I -1 1 1 1
1 1] which is the correct response.
y,,,=xW=[O 0 1 1]
-1 1 I 1 . ''
[
Test input x = [-1 1 0 I]. Computing net -1 1 1 1
input, we obtain =[-2222]

J;roj=xW Applying the activations, we get Yi = l-I 1

1 -1 -1 -1] 1 1] which is the correct response.
-1 1 I 1
=[-1101] Test input x = [-I 0 0 1]. Computing net
-1 I 1 1
[ input, we obtain
-1 1 1 I
= [-3 3 3 3] ]i"j=xW
Applying the activations, we get Yi [-1 1 1 -1 -1 -1]
1 1} which is the correct response. -1 1 1 1
=[-1001]
[ -1 1 1 1
Testing the network with one mistake entry -1 1 1 1
!est inputx = [ -1 -I 1 1]. Computing net =[-2222]
mput, we get
Applying the activations, we get Yi [-1 1
]mj=x W I 1] which is the correct response.
 130 Associative Memory Networks

Testing the network with two mistaken entry Testinpurx::::{l 1 0]

Test input x:::::: [-1 -1 . :. . 1 1}. Computing ncr
input, we obtain 0 1-1]
Yinj=xW
Yr.;=x\Y/=[1 1 OJ
[ 1 0 -1
-1 -1 0
1 -1 -1 -1] = [1 1 -2]
-1 1 1 1
=[-1 -1-1 1] _ 1 Applyingtheacrivarions,wegeryj=[1 1 -1],
(}; [ 1 1 1
-1 1 1 1 hencc;,a correct response is obtained.
= [0 0 0 0] [ .. ' ') 10:i:Js: outer products rule to store the vectors
. t he actLvauons
APP Iymg over the ner tnput, we get ../ [1 1 1 1]and[-l 1 1 -1]inanauro-
Yi = [0 0 0 0] which is rhe incorrect response. associative network. (a) Find the weight matrix
Thus, the network \'[ith two mistakes is not recog- (do nor set diagonal term w zero). (b) Test the
nized. vector using [1 1 1 1] as input. (c) Test the
-. . . . vector[-1 1 1 -l]asinpur.(d)Tesrthenet
9._1=heck the auroassoc1anve ne~ork for m~ut using [1 1 1 O] as input. (e) Repeat (a)-(d)
/ vector [ 1 1 -1 ]. Form the Weight vector_'lDth with the diagonal terms in the weight matrix to
.__;. no self-connection. Test whether the net is able to be zero.
n:wgmLe with one missing enrry.
Solution:
Solution: Inpm vector x = [1 1 - I]. The weight
vecror is Weight matrix for [1 1 l 1] is

W, = I>T(p)s(p)
\Y/ = I/(p)s(p) = [ _:] [1 1 -IJ

=
[ I
1
1-1] 1 -1
=
[
I
1]
1 [1 1 1 1J =
1
1 1
[1 11111]
1 1 1 1
1 I 1 1
-1 -1 1

J
Weight marrix for [ -1 1 I -1] is
The weight vector with no sdf-com1ecrion (make
the diagonal elemems in rhe weight vector zero) is
given by

\YJ;::::
rC{)_ 'I -IJ
r ." o"-.J.
\Y/2 = I>' (p)s(p) = [ [-! 1 1 -1 J

.
[ -1 -"-1 0'
. >-- \ -1
1-1 -1 1]
1 1 -1
Testmg the network wtth one mtssmg ellrty
Tesrinputx=[l 0 -1]
-
[ -1 1
1 -1 -1
1 -1
1

\'I~Fie we1ghr m:rnlxto store rwo vectors-;1s---';.

y1,,=x\Y/=[1 0 -1J
[ 0 1-1]
1
-1 -1
0 -1
0
W-WJ +tllz
~

= [1 2 -1J

Applyingrheacrivarions,wegetyj
hence a correct response is obtained.
= [l I -1], -- [~ ~ ] [- ~
1
1
1 1
1 1
1
I
1 1
1 1
+ -1
-1 -1
I
1
1 -1 -1
I
1 -1
1
-~]
i
r .I
 4.10 Solved Problems 131

2 0 0 2]
0 2 2 0
(i) The weight matrix is

=
[ 0 2 2 0
2 0 0 2 W=
2002]
0 2 2 0
0 2 2 0
[
Test the vector using [1 1 1 1} as input 2 0 0 2
Test vector x = [1 1 1 1). Computing net
input, we obtain {ii) Test the vector using x = [1 1 1 1] as
input. Computing net input, we obtain

2 0 0 2] 0002]
Jr.,=x\Y/=[1 1 1 1J 0 2 2 0
[ 0 2 2 0
2 0 0 2
y,,,=x\Y/=[1 1 1 1J
[
0 0 2 0
0 2 0 0
2 0 0 0
= [4 4 4 4J
=[2222J
Applying the activations to calculate output, we Applying the activations, we get Yi = [ 1
get Jj = {1 1 1 1], hence correct response is 1 1}, hence correct response is obtained .
obtained.
{iii) Test the vecmr Usmg x = t=r-r-r --11
Test the vector using [-1 1 - 1] as input as input.
Test vecmr x = [- 1 I -1]. Computing net
input, we obtain
y,,,=xW=[-1 1 1 -IJ
000']
0 0 2 0
2 0 0 2] [2 0 0 0
0 2 0 0
Jr,,=x\Y/=[-1 1 1 -IJ 0 2 2 0
[ 02 2 2 0
0 0 2
= [-2 2 2 -2J

= [-4 4 4 -4J Applying the activations to calculate output,

we getyj = [-1 1 1 - 1], hence an
Applying the activations to calculate output, we unknown r~onse is obtained.
get Yj = [-1 1 1 -1], hence correct response ....------- -
(iv) Test the vector using x = [I 1 1 0} as
is obtained.
mpur.
Test the ue& wing [1 1 1 0} ns input
Test vecmr x = [1 I 1 OJ. Computing net
input, we obtain 000']
y,,=x\Y/=[1 1 1 OJ 0 0 2 0
0 2 0 0
[
2 0 0 0
2002] = [0 2 2 2J
Yri = x \'{/ = [ 1 1 1 OJ 0 2 2 0
0 2 2 0
[ Applying the activations to calculate outpm, we
2 0 0 2
= [2 4 4 2J get Yi = [-1 1 1 1}, hence an unknown
response is obtained.
i Applying the activations, we get Yi = [ 1 1 11. _9nd the weight matrix required to srore the
I 1 1), hence the known response is obtained. /vectors[} I -1 -1},[-1 I 1 -l}and
I Repeat parts a-d with diagonal elemenr i11 weight _/ [-1 1 -1 1]inroWl,W2,W3respecrively.
matrix set to zero Calculate the total weight matrix to store aH the

i'
j
132 Associative Memory Ne!Works

vector and check whether it is capable of recog-

I -1 I -1]
nizingthesame~d. ~ight
matrix be wi'tliilo sdf-conneccion.
--------~
Solution: For the first vector [1 1 -1 -1]
-
[ -1

-1
I -1
I -1
I -1
I
I
I
I

With no self...conneccion,

W1 = L,T(p)s(p) = [ j] [1 I -I -I]
W30=
0 -1
-1 0 -1
1 -1
I
0 -1
I -1]
I I -1 -1]
[
-1 I -1 0

-
[
-1
-1
I
-1
-1
I -1 -1
I
I
I
I
The total weigh~ matrix required to store all iliis is

W=Ww+ Wzo + W3o

With no self-connection, I 0 I-1 -1] + [ 0-1 I-1]
0 -1 -1 -1 0 -1 I

0 I -1 -1]
=
[-1 -1 0 I I -1 0 -1

l
-1 -1 I 0 -1 I -1 0
I 0 -1 -1
Wio = -1 -1 0 1 0 -1 -1 -1]
-1 -1 I 0 -1 0 -1 -1
- -1 -1 0 -1
[
Forthesecondvecror[-1 1 1 -1] -1 -1 -1 0

Testing the network

W, = Ll(p)s(p) = DJ [-1 I I -I]

Withfirsrvectorx =[I
is given by
1 -1 -I]. Nerinput

I -1 -1 I] J;.;=x W [ 0-I -I -I]

[
-1 I I -1
I 0 -1 -1
- -1 I I -1 =[I I -I -I] -I -I 0 -I
I -1 -1 I
-1 -1 -1 0
=[I 1 -1 -1]
Wich no self-connection,
Applying activations, we get Yj = ( 1 -1 -1]
-I -I

W2o = [ -I0 0 I
-~ I 0 -I
-I -I 0
-:] which is the correct response.
With second vector x = [ -1
input is given by
-I]. Ne<

-I I]
y;"j =x W
For the third vector [ -1 I
[ -1
0 -1 -1 -1]

~j] H
0 -1 -1
=[-1 I 1 -I] -1 -1 0 -1
w, = Ll(p)s(p) = [ I -I I] -1 -1 -1 0
=[-1 I I -I]
 133
4.10 Solved Problems

Applying activations, we getYi = [ -1 1 I -I] 0-I -1 -1]

-1 0 I I
which is the correct response.
Wtth third vector x = [-1 1
puting net input, we ger
- 1 1}. Com-
=[-1000]
[ -1
-1
I
I
0
I
I
0

. = [0 I I 1]

y;.;=xW [ 0 _1-1 -1]

1 0 -1 -1
cApplyingactivations,wegetyj= [-1
i.e., known respons_e is obtained.
l l 1},

= [-1 1 -1 1] -I -I 0 -I For test input vector x = (0 0 0 1]. Compur-

-1 -1 -1 0 ing net input, we obtain

y;,j=x-Wr~ ")~
= [-1 I -I I]

Applying activations, we getyj = [ -1 1 -1 I]

"' .i-'<1' / '
= [0 If 0 I]
[ 0-1 -1 -1]
-I 0 I I
which is the correct response. -1 I 0 1
Thus, the neWork is capable of recognizing the -1 I I 0
"c"
,,_,.J r<' j
~--

vecror.s.- =[-1 I I 0] o>

~ ~suuct an autoassociative network to store Applyingactivations,wegetyj = [-1 l 1 -1],
'- vectors [-1 1 1 1]. Us~auroasso i.e.,.unknown response is obtained. hecate the net-
ciative network to test ilie vecror wiili three work again using the net input calculated as input
missing elemems. vector:
Solution: The input vector is x = [-1 1 I I}.
0 -1 -1 -1]
The weight matrix is obtained as -1 0 1 I
J,;=[-1 I I 0] -1 1 0 I
[
-1 I I 0

w = Flp)s(p) = [ -;] [-1 I I 1) =[-2223]

Applying activations, we get Yj :: [-1 1 1 11,

I -1 -1 -1]
-1 I 1 I
(
i.e., known response is obtained after iteration.
Thus, iterarivc auroassociarive network recogni1.es

-
[ -1
-1
I
l
I
I
1
I
- \.
the rest pmern. Similarly, the network can be
rested for the rest input vectors [0 I 0 0] and
[0 0 l 0].

l
--------
wir~--;;o self:~~~~ is
----
\~-
'
C~trucr an autoassociative discrete Hopfield
The weight matrix
_ network with input vector [1 l I - 1].
Test the discrete Hopfield nerwork wirh miss-
0 -1 -1 -1
W _ -1 0 1 I ing emrics in first and second components of
0- -I I 0 I the stored vector.
[
-1 1 1 0 Solution: The input vector is x:: [I I l -I].
The weight matrix is given by
Test veaor with chree missing elemenu
Foe rest input vector x = [ -1
input is calculated as

Yid=x W
0 0 0], the net
w = l:l(p)r{p) = [ J] [I 1 1 -I]
 134
Associalive Memory Networks

I I I -1 ] Step 4: Choosing unit Y4 for updating its activa~

I I I -1 cions:
= I I I -1
[
-1 -1 -1 I
Yin4 = X4
'
+ L)j'Wj4
The weight marrix with no selfconnection is j=l

w=
[
l
l
0
0
I

I
I -1 ]
l -]
0 -1
-] -] -] 0
=0+[ I 0 I 0] [=l]
=0-1-1=-2
The binary representai:ion for the given input vec-
tor is [I 1 1 0]. We carry our asynchronous Applying activations we get Jin4 < 0 ::::}
updarion of weights here. Let it be Y1, Y4, Y3, Y2. Y< = 0. The<efore, y = [I 0 I 0] ->
No convergence.
For the test input vector with t}Jlo missing entries in Step 5: Choosing unitY3 for updating its activa-
first and seco11d compommts of the stored vector. tions:

Juration I
Jin3 = X3
'
+ L.JjWj3
I Step 0: Weights are initialized to store paucrns: I j=I

0 I I -I]
[
W= I 0 I -1 =I+[IOIO][_n
I I 0 -1
-1 -1 -1 0
=1+1=2
Step 1: The input vector is x = [0 0 1 0]. Applying accivations we get y;,3> 0 :::?
Step 2: For this vecwr y = [0 0 l 0]. ]3= l. Therefore,y = [1 0 I 0] --Jo
Srep 3: Choose unit Y1 for updating irs activa- No convergence.
tions: Step 6: Choosing unit Yz for updating irs activa-
tions:

y;,1 = Xt + Lypujl
j==l Yin2 = xz + LYjW;2
j=l

= 0 + [0 0 I 0] [ _!] = O+ I= I
=0+[1 01 O ] [ j ]

Applying activations we ger Jinl > 0 :::} =0+2=2

}I= l. Broadcastingy 1 to ali mhcr units,
weger Applying activations we get Jin2> 0 ::::}
yz = 1. Therefore, y = [1 1 I 0] --Jo I
y = (1 0 1 0]-+ No convergence 1 Converges wirh vector x. /

1
 4.10 Solved Problems 135

Thus, the output y has converged with vector xin this Applying activations we get Jin3 > 0 ::::}
itemion itself. But, one more iteration can be done y3 = l. The<efore,y = [l 1 I 0].
to check whether further activations are there or not. Step 6: Choose unit Yz for updarion.
Iteration 2 4
Yin2 = XZ + L JjWj2
Step 0: Weights are initialized to smre patterns. j=l

W=
[
0
1
l
I
0
I
1 -1 ]
I -1
0 -1
= 0+ [l I I 0] [ j]
= 3

-1 -1 -1 0
Applying activ:uions we get y;,z > 3 ::::}
Step 1: The input vector is x = {1 l 1 0]. I n=l.The<efore,y=[l 1 1 0]. I
Step 2: For this vector y = [1 l 1 0].
Thus, further iterations do nor change the activation
Step 3: Choosing unit Y1 for updating its activa- of any unit.
tions:

~
orfsrrucr an auroassociativ network to store
1 he veaors x 1 = [1 1 1], X2 = [1 -I
]bll = XJ
'
+ LJJU~ll -11-1],x3 = [-1 -1-1-1]. Find weight
j=l \ matrix with no s -conneccion. Calculate the
energy of the ~.t red patterns. Using discrete
Hopf1eld ne~rk test patterns if the rest par-
=1+[ 1 I 1 0] [ J ] = 3
tern are fti.;;en as x 1 ::::: [I 1 l-1 I], xz =
[I- y-'1 -I -I] 'ndx3 =[I 1 -I -I - 1].
Coffipare the test patters energy with the stored
Apply activations we get Ji11 l > 0 ::::}
YI = l. Now y = [1 I I 0].
/~terns energy.
Step 4: Choose unit Y4 for up dation. S~lurion: The weights matrix for rhe three given
' vectors .IS

)'in4 = X4 + L' Jj1Vj4 W'= '[3ipl '/J>)

[l' {l'"' "{} '"'

j=l

=0+[1 1 1 0]

Applying activations we get y;,1q < 0 ::::}

{}" '"
J4=0.The<erore,y=[1 1 1 O].
Step 5: Choose unitY3 for updation.

Yitd = x3 + L' ypvp,

J=l I I I I I' [ I -I

=0+[1 I I 0] [ J ] = 3
=
I I I I I
1 I
[ I
1 1 I
1 1 1 1
+
-1
-1
1 -1
-:]
-I
l I 1 I 1 -1 I

1
136
Associative Memory Networks

I -1 I I I] Energy for ucond patum

+ I -1 I I I
-1 I -1 -1 -1
, = -0.5[x,WTxj"]

[ I -1 I I I = -0.5 [I -I -I I -I]
I

3-1 I 3I]
-1 I I I
-1 0-1 I 3I] [ I]
0 I -1 I -1

W=
[
-1
I I 3
3 -1 I
3 I -1
I 3
3 I
I
[ I

I
I 0
3 -1 I
I 3
I 3
0 I
I 0
-1

-1
I

I I 3 I 3

The weight matrix with no self~connection is

0-1 I 3I]
-_.,. _,. -H l
-1 0 I -1 I
= -0.5 [2 + 4 + 2 + 2 + 2] = -0.5 [12] = -6
Wo = I I 0 I 3
[ 3 -1 I
I I 3
0 I
I 0
Energy for third pattmz
3 = -0.5[x,WTxj]
The energy function is defined as =-0.5[-1 I -1 -1 -1]

= -O.S[xWT,.T] -1 0-1 I 3I] [-I]

0 I -1 I I

Therefore the energy for rhe irh panern is given by [ I

3 -1 I
I I 3
I 0 I 3
0 I
I 0
-1
-1
-1
E; ~ -O.S[x; W1 xj]

Energy fOr first pattern

1 = -0.5[x1WTxT
1
J
0 ~" ' ' ' _, _,, [ 3]
=-0.5[1 I I I I 1] 1, 5
= -0.5 [6+0+4+ 6+ 4] = -0.5 [20] = -10

-10-1 I 3I] [ I]
0 I -1 I I Applying test patterns

[ I
I
3 -1
I 0

I 3
I
I 3
0 I
I 0 >><S
I
I
I Sxl
For first test pattern x'1 = [1 1 I -1 1]
andy=[l 1 l -1 l].Choosingunir4for
updation, we ger

[-~j
4
y;,lj = X4, + LYj Wjl
=-0.5[1 I I I 1] 1 xs

= -O.S [4 +O + 6 + 4 + 6l 1 x,
6
Sxl
0 _,::, , _, ,
1[ ciJ
= -o.s [20] = -1o

_...
 4.10 Solved Problems 137

=-1+3-1+1-0+1=3>0 for updarion, we get

4
Applying acrivadons, we get Y4 = 1. Therefore, ]in\ =X] + l:YJWjl
~ = [1 1 1 1 1] --+ convergence. The j=l

-j]
energy function is given by

Ei = -0.5[,; WTx'!] 0 '+ ,, ' -' _, _,, [

On substituting the corresponding values, we get

E; = -10 =I+ I -I -3 - I = -5 < 0

For second rest pauern x'2 = [1 -1 -1 -1 -1] Applying activations, we get y = -1. There~
fore, modifiedx~ = {-1 1 -1 -1 -1]--+
andy= [1 -1 -1 -1 -1]. Choosing unit 4 for
updation, we ger convergence. The energy function is given by

'"{ll
4 E3 = -0.5[x'3wTx:("J
Yin4 = X4 + l:Yj Wj4
j=l

-l~l
e-O>H ' _, ' -1
0

'+'' ' ' ' =-0.5[20]=-10

Thus, the energy of the stored pattern is same as that

=-1+3+1-1-0-1=1>0
of the test pauern.
Applying activations, we get ]4 = l. Therefore(' 15. Construct and test a BAM net\York ro associate
:/2 = [ 1 - 1 1 - l] --+ convergence. The letters E and F with simple bipolar input-output
energy function is given by vectors. The target output for E is (-1, 1) and
for F is (1, 1). The display matrix size is 5 X 3.
~ = -0.5[x;wTx;Tl The input patters are

~
""

-0.5 [12]
'

= -6
" -I<'"{! l * * *
*
* * *

* * *

* * *
* * *
*
*
*
"E" "F"
Target output (-1, 1) (!, I)
Forrhirdtestpanernx,3 =[1 1 -1 -1 -1]
andy=[I 1 -1 -1 -I].Choosingunitl SolUtion: The inputs are

Input pattern Inputs Targets Weights

E
F
[I I
[I III I
I 1-1-1 I I I 1-1-1 I I I]
I 1-1-1 1-1-1 1-1-1]
[-I, I]
[I I]
w,
w,

.l
138 Associative Memory Networks I'

(i) X vectors as input: The weighr mauix is obtained by

I
!

w = 'LJ (p) t(p)

-1 1 I
-1 1
-1 1 I
-1 1 I
-1
-1
I I
1 -I
I -I
-1 1
W,=l I I 1-1 1l = 1 -I 1
-1 I
-1 I
-1
-I
I I
1 -1
1 -I
-1 I
-1 I
-1 I I

w, =I -1 I[1 I]= ~-1 -11

-I -1 -1 I
-1
-1
I 1-1 -1
-1 -I
I I
I

-1
-1
I 1-1 -1
-1 -1
I I
 4,10 Solved Problems 139

The mral weight matrix is

1 1 1 0 2'
,-1
-1 1 1 1 0 2
-1 1 1 1 0 2
-1 1 1 1 0 2
I -1 1 1 2 0
1 -I 1 1 2 0
-1 I 1 1 0 2
-I c 1 -1 -1 -2
W=W 1 +Wz= + = 0
-1 1 -1 -1 -2 0
-1 I 1 1 0 2
1 -1 -1 -1 0 -2
1 -1 -1 -1 0 -2
-1 1 1 I 0 2
-1 I -1 -1 -2 0
-1 1 -I -1 -2 oJ f

Testing the network with test vectors "E" and "F." 'l
For test panern , compuring net inpm we get I

~I
0
0
\
')(
I 0 I

0 2
2 0
-2 0
0 2
y;, = {11 I 1 -1 -1 I 1 11-1 -111 l] I xiS -2 0
-2 0
0 2
0 -2
0 -2

~-~
2
0
-2 0
.J 1Sx2

= [-12 18]1 x2
Applying activations, we get y = [ -1 1], hence correct response is obtained.
 140 Associative Memory Networks

For test pattern E Computing net input, we get

0 2
0 2
0 2
0 2
2 0
2 0
0 2
y;. ~ [IIIIIII -I -II -1 -I I -I -1] -2 0
-2 0
0 2
0 -2
0 -2
0 2
-2 0
~ [12 18] -2 0

Applying acdvarions over the net input, to calculate output, we gety = [l I], hence correct response is
obtained.
(ii) Y vectors as input: The weight matrix when Y vectors are used as input is obtained as the transpose of
rhe weight matrix when X vectors were presenred as input, i.e.,

wT ~ [ ~ 0 0 0 2 2 0 -2 -2 0
2 2 2 0 0 2 0
0 0 0 -2 -2 ]
0 2 -2 -2 2 0 0
Testing rhe network

(a) For test pauern E, now the inpm is [-I I]. Computing net input, we have

,- ~
) y;,~xWTi= [-11][0 0 0 0 2 2 0 -2 -2 0 0 0 0 -2 -2]
________ , 2222002 0 02 -2 -2 2 0 0
= [2 2 2 2 -2 -2 2 2 2 2 -2 -2 2 2 2]
Applying rhe activation functions, we get

y~[I I I I -1 -1 I I I I -1 -1 I I I]
which is the correct response.

(b) For rest panern F. now the input is [I, l]. Comp_uring net input, we have

~
-~]
[ 0 0 0 0 2 2 0 -2 -2 0 0 0 0 -2
:'_'~~ ~ [I I]
2 2 2 2 0 0 2 0 0 2 -2 -2 2 0
~ [2 2 2 2 2 2 2 -2 -2 2 -2 -2 2 -2 -2]
.

b .
i&
~<;::
 141
4.10 Solved Problems

Applying the activation functions, we get

y~ [I I I I I I I -I -I I -I -I I -I -I]

which is the correct response. Thus, a BAM network

-1I -1I ] [ -1I -1I ]
has been constructed and rese~IDe-dire,etions
fromXtoYandYtoX. ' '
+
[
+I -1 + -1 I
I -1 I -1
16. (a) Find the W.ight mauix in bipolar form
for the bidirectional assoctanve memory using
outer products rule for the followi~ary
-4 4-4]4
input-output veCtor pa.tiS:
w~
[ -2 2
2 -2
s(I) ~(I 0 0 0), ~I) :! (I 0)
s(2) ~(I 0 0 1), ~2) ~(I 0) (b) The unit step funetion for binary with dueshold
s(3) ~ (0 0 0), ~3) ~ (0 I) 0 is used.
s(4) ~ (0 0), Ml ~ 10 Il
(b) Using th~step function (wi~old For Y layer => Yi ~ I Yi

/---{I
0) as the output units activation hlricnorr,test 0
~e response of the network on each of the input
pauerns. For X layer , , x1 = x,
(c) Test ilie response of the network on various
' - 0

si~~~
combinations of input pauerns with "mistakes"
or "missing" data. Presenting
(i) [I 0 -I -I]; (ii) [-1 0 0 -I];
(ili) [-II 0 -I]; (iv) [II-I-I]; (v) [II] s(l) = 11 0 0 0]. Computing net input,
we have
Solution:
(a) The weight matrix for storing the four input
,,.,~[1000]
4-4]
vectors in bipolar form is
[ -4
-2
2 -2
4
2

' (p) 'f(p)

w ~ "23 ~ [4 -4]
p=l

~ [=l]
Applying activations we get t_; = [1 0] which

[I -I]+ [ =\] [I -I]

is the correct response.
s{2) = [1 0 0 1]. Computing net input,

+ [ ;j] [-II]+ [ ~;] [-1 I} ''i ~ [I 0 0 I]

[
4-4]
-4
-2 2
4
2 -2
~ [6 - 6]
-1I -1I ] [ -1I -1I ]
~ -1 I + -1 +I Applyingactivacions we get t_; = [l 0} which
[
.i -1 I I -1 is the correCt response.
i
,,"-

&
:

u m
142
Associative Memory Networks

s(3) = [0 1 0 0]. Computing rhe ner (c) Test response of network

input, we have
(i) Herex;;: [1 0 -1 -1]. Calculating

'i= [0 1 00] -4 4-4] 4

ner input, we get

[ -2
2 -2
2 J;"l=xW

= [-4 4]
= [1 0 -1 -1]
4 -4
-4 4
J
Applyingactivarions we get tj = [0 1] which [ -2 2
is rhe correct response. 2 -2
= [4 -4]
s(4) = [0 1 I 0]. Computing the net
inpur, we get
Applying activations we getyj = [1 0}

~-i = [0 1 1 0]
4-4] which is the correct response.
(ii) Herex= [-1 0 0 -1]. Calculating
[ -4
-2
4
2
2 -2
the ner input, we get

= [-6 6]
y,,= [-1 0 0 -1] -4
4-4]
4
Applyingactivarionswegetij
is the correct response.
= [0 l]which
[ -2 2
2 -2
Prmnting t-input pattern = [-6 6]

t(l) =[I 0}. Computing the ncr input, we

Applying activations we get Jj' = [0 I}
obmin
which is the correct response.
4 -4
' = [ 1 0] [ -4 4 -2 2]
2 -2
(iii) Herex=[-1 I 0 -l].Calculating
the net input, we get
=[4 -4 -22]

Applyingaccivarionswegeu; =[I 0 0 ll
J;.,=[-1 l 0 -1]
4-4]
-4 4
which is rhe correct response.
t(3) = [0 I]. Compuring the net inpm, we
[ -2 2
2 -2
obrain = [-1 0 I 0]

''"' = [0 1] [ _:

=[-4 4 2
-4 -2
4 2
- 2]
-n Applying activations we get Yi = [0 l]
which is the correct response.
(iv) Here x = ( 1 l -I -I]. Calculating
Applyingactivarionswegetsj = [0 I l 0] the net input, we get
which is the correct response.
On preseming ilie panern [I 0] we obrain
only [1 0 0 I] and nor [1 0 0 0]. Sim-
''i = [1 1 -1 -1]
4-4]
- 4 4
ilarly, on presenring the pattern [0 lJ we
obtain only [0 1 1 0) and nor [0 l 0 0].
[ -2
2 -2
2

This depends upon rhe missing data enrries. = [0 0]

1
 4.11 Review Questions 143

Applying the previous acrivarion and 17. Find the hamming distance and average
raking closely related panern activation hamming distance for the rwo given input
we ger Yi = [0 l]. vecwrs below.
(v) Y = [1
-=
1]. Computing the net input,
X1 = [l I -1 -1 -1 l -1 -I -11 -I -I]
we get ! ,__
x, = ( -1 1 1 -1 1 -1 1 -1 1 -1 -1 1]
4 -4 -2 2]
x,., = [ 1 11 [ -4 4 2 -2 Solution: The hamming distance is number of
different bits in two binary or bipolar vecrors.
= [0 0 0 0] Here
Thus, in this case since ali the X;"; values H[X1,X2] = B
are zero, to apply the activation func-
tion it may take the previous x; values for
x;m = 0. Hence the closely[elated pat-
temcan be taken to obtain dle correct
response.

I 4.11 Review Questions

1. What is content addressable memory? 13. Is it true that input patterns may be applied at
2. Specify the functional difference berv.een a RAM the outputs of a BAM?
and a CAM. 14. List the activation functions used in BAM net.
3. Indicate the two main (}'Pes of associative mem- 15. What are the 1:\VO rypes of BAM?
ocy. 16. How are the weights determined in a discrete
4. State the advantages of associative memory. BAW
5. Discuss the limitations of associative memory 17. State the resting algorithm of a discrete BAM.
nerwork. 18. What is rhe activarion funcrion used in contin-
6. Explain the Hebb rule training algorithm used uous BAM?
in pattern association. 19. Define hamming distance and storage capacity.
7. State the outer products rule used for training 20. What is an energy function of a discrete BAlvl?
pattern association nerworks.
21. What is a Hop field net?
8. Draw the architecture of an autoassociative net-
22. Compare and contrast BAM and Hopfield net-
work.
works.
9. E.xplain the testing algorithm adopted ro test an
23. Mention the applications of Hopfield network.
autoassociative network.
24. What is the necessiry of weights with no self-
10. What is a heteroassociative memory network?
connection?
11. "With a neat architecture, explain the uaining
25. Why are symmetrical weights and weights
algorithm of a hereroassociative network.
with no self-connection important in discrete
12. What is a bidirectional associative memory net- Hopfield neE?
work?

1
144 Associative Memory Networks

26. What is a recurrent neural network? 32. Discuss in derail on continuous Hopfield net~
27. What are the two cypes of Hopfield ner? work.
28. Draw cite architectUre of discrete Hopfi.eld 33. Make an analysis of energy function of a conrin-
net. uous Hopfield net'Nork.
29. State the testing algorithm used in discrete 34. What are iterative autoassociative memory nets?
Hopfield network. 35. Explain in detail on linear amoassodative mem-
30. What is the energy function of a discrete ory. Stare the conditions of linearity.
Hopfi.eld network? 36. Write shan note on brain-in-the box model.
31. Mencion the formula used for derermining.the 37. What is the functional equi...-alent Of a temporal
storage capaciry of a discrete Hopfield ner. associative memory network?

I 4.12 Exercise Problems

1. Train a hereroa.ssociative memory network using bipolar form. The binary vector pairs are:
Hebb rule to store input row vector s =
(sl s2 .!'3 .!'4.) to the output row vector t = (!] &2). s(1) = (1 0), 1(1) = (0 1)
The vector pairs are given as below: s(2) = (I I), 1(2) = (I 0).

Also test the performance of the network with

s{1) = (1 0 0 1), 1(1) = (1 0)
missing and mistaken data.
s(2) = (1 I I 1), 1(2) = (1 0)
s(3) = (1 I 0 0), 1(3) = (0 1) 5. Construct a hereroassociative network for the
s(4) = (0 0 1 I), 1(4) = (0 I) panern given below:

2. Construct and test a heteroassociacive memory

network using outer products rule to store the
given input-rarger vector pairs: '"!"
"C''
s(1) = (I 0 1), 1(1) =(I 0)
The target of "I" and "C" arc (1, -1) and
s(2) = (0 1 1), 1(2) = (0 1)
(-1, 1) respectively. Store the pattern and as well
3. Construct and test a heteroassociative memory recognize the pattern.
net to store the given vector pairs: 6. Train an auroassociativc network for input vec-
tor [-I 1 1 - l] and also rest the net\vork with
s(1) = (0 0 0 1), 1(1) = (0 1) same input vecror. Test the auroassociative net~
s(2) = (0 0 1 1), 1(2) = (0 1) work with one missing, one mistake, two missing
s(3) = (0 I 0 0), 1(3) = (I 0) and two mistake entries in test vector.
s(4) =(I I 0 0), 1(4) = (I 0) 7. Check the auroassociative network for input vee~
ror [-1 - 11]. Form the weight vector with no
Also rest the network with "noisy" input patterns self-connection. Test whether the net is able to
included. recognize with one missing and two missing data.
4. Consrruct and train a heteroassociative nerwork Comment on network performance.
to srore the following input-output vector pair. 8. Use outer products rule to store vectors
The training input-target output vector pairs [-l-1 -1 1] and [1 1 1 -1] in an auro-
are in binary form. Obtain the weight vector in associarive network.

!
I
 145
4.12 Exercise Problems

Find the weight without setting diagonal 15. Consider a discrete Hopfield network with a
terms to zero. synchronous update.

Test vector using [-1-1-1-1] as inpuc. Show that if all given pattern vectors are
Test nerwork using [1 1 1 11 as input. orthogonal, then every original pattern is an
global minimum.
Test the net using [0 1 1 0] as input.
Show that in general other global minima
Repeat (a)-(d) with diagonal elements set to
exist.
zero.
16. Construct and test a BAM network to asso~
9. Find the weight mauix required m store the vee~ ciate letrers T and 0 with simple bipolar
rors[ll-11-1],[1111-1],[-1-11 I -I]
input-output vectors. The target output forT
and[11-l-11]inwi,w2,W3,W4,respec- is (1, -1) and for 0 is (1, 1). The display mauix
cively. Calculate the total weight mauix to store
size is 4 x 3. The input patterns are
ali rhe vecmrs and check whether it is capable
of recognizing the same vectors presented. Per-

form the association for weight matrix with no
self~connection.
*
* *
10. Construct an auroassociative network to store
vector [1 1 -1 +1]. Use iterative autoassocia~

"T"
*
* "0"
rive net\vork to test the vector with three missing
elements. 17. Find the weight matrix. in bipolar form for the
11. Construct and rest an associative discrete Hop- BAM using outer products rule for the following
field network wirh input veccor [1 -1 1 1]. Test binary input-output vecmr pairs.
the net\vork with missing entries in first and
s(l) =(I 0 0 0), l(l) = (0 1)
fourth components of the stored vector.
s(2) = (0 I I 0), 1(2) = (1 0)
12. Construct an auroassociative net\vork to store
the vectors XJ = [1 1 1 I 1 -1], X1 = Using the unit step function as the omput unit's
[-l-1-llll],x3 = [III-1-I-1].Find activation function, test the response of the net~
weight matrix with no self-connection. Calculate work on each of the inpm patterns. Also test the
the energy of the stored patterns. response of the nerwork on various combinations
13. Consider a t\VO node continuous Hop field net of input pattern with "mistakes" or "missing"
work. Assume the conductance is ffi = grz = 3 data.
mho. The gain parameter is A= 1.2 and the 18. Find the hamming distance and average ham-
external inputs are zero. Calculate the accurate ming distance for the two given input vectors
energyvalueofthestatey= [0.1 0.1]T below:
14. Design a linear hereroassociare nerwork that
associates rhe following pairs of vectors. X1 = [1 1 I - I - I I I - I - 1 - 1 - I
I I -I]
X\= [1,3,-5, l]T, Yl = [0 o 01 T
X2 = [2, 2, 0, -4]T, ]2 = [I 0 l]T
X,= [1 I - I I I - I 1 - 1 - 1 I I

X3 = [1,0, -3,4]T, ]3 = [0 I]T - 1 1 - 1]

Verify that vectors x 1 , X'2 and X3 are linearly 19. Prove the stability of the continuous BAM using
independent. Compute weight matrix of linear (a) Kohonen Grossberg theorem and
associates. (b) the Lyapunov rheorem.
 146 Associative Memory Networks

20. Design a BAM-based temporal associative mem- Compute the weight mauix W and check ilie
orywitha thresholdactivacion function to recall recall of panerns in forward and backward direc-
the following sequence: tions.
'= {[1 1 1 1 - 1 1 1], [1 1 1 1 - 1 - 1- 1],
[-11111-1-1]]

I 4.13 Projects

1. Write a compmer program to implement a het- x pattern):

eroassociarive memory nerwork using Hebb rule
to set the weights. Develop the input patterns and
target ourpm of your own.
*.
.
"A"
* .*
.
"B"
. .*
"C"

2. Write a program to construct and test an aumas- * * * * *

sociative nerwork to store numerical values from * .* * * .*
0-9. Also create the patterns for 0-9 using a 5 x 3 * .* .* .* * *
array matrix. Add "noise" to the input signals and *
(1, 1, 1)
* * *
(-1, -1, 1) (1, -1, L)
test the network.
3. Write a "C" program ro implement a discrete
Hopfield net to store rhe letters A-E. Form the
"D"
* *.
. "E"

..
** * * * *
"F"

.
input panerns for the lwers in a 4 x 3 array
* .* * *
matrix.
.* .
* * *. * *.
4. Write a compmer program to implement a bipo~ * .* * *
lar BAM. Allow 15 units in X layer and 3 * *
(-1, 1, 1)
* * *
(1, 1, -1)
*
units in Y layer use the program to store the (-1, -1, -1)
following patterns (the X layer vectors are the
Is it possible to store all six patterns at once? If
leners given in rhe 5 x 3 arrays and the asso-
not, how many can be stored at the same rime?
ciated Y layer vectors are given below in each
Perform some experiments with noisy data.

i
I(' .
~
 Unsupervised Learning Networks 5
Learning Objectives - - - - - - - - - - - - - - - - -
Definition of unsupervised networks. network, adaptive resonance theory and
Gives derails on fLXed weight competitive nets LVQ
like Maxnet, Mexican hat and Hamming net. Enhance the features and star topology of
Discusses the neighborhood topology of CPN network.
Kohonen self-organizing feature maps. De~ails the varianrs of LVQ (LVQ2, LVQ3)

Provides architecture, training algorithm, and ART (ART 1 and ART 2).
flowchut depicting training process and Variecy of solved problems using unsupervised
testing algorithm of different unsupervised learning network.
networks like KSOFM, Coumerpropagation

I 5.1 Introduction
In this chapter, the study is made on the second major learning paradigm-unsupervised learning. In rhis
learning, there exists no feedback from the sysrem (environment) w indicate the desired outputs of a network.
The network by itself should discover any relationships of interest, such as features, patterns, contours,
correlations or categories, c\assif~earions in the inpm data, and thereby uanslate the discovered relationships
imo outputs. Such nerworks are also called self-organizing networks. An unsupervised learning can judge
how similar a new input panern is to rypical patterns already seen, and the network gradually learns what
similaricy is; the network may construct a set of axes along which to measure similariry to previous panerns,
i.e., it performs principal component analysis, clustering, adaptive vector quantization and feature mapping.
For example, when net has been trained to classify the input patterns inro any one of the output classes, say,
P, Q, R, SorT, the net may respond to both rhe classes, P an,d Q orR and S. In the case mentioned, only one
of several neurons should fire, i.e., respond. Hence the network has an added strucrure by means of which the
ner is forced to make a decision, so that only one unit will respond. The process for achieving rhis is called
competition. Practically, considering a set of students, if we want to dassify them on the basis of evaluation
performance, their score may be calculated, and the one whose score is higher than the orhers should be the
winner. The same principle adopted here is followed in the neural networks for pattern classification. In this
case, rhere may exist a tie; a suitable solution is presented even when a tie occurs. Hence these nets may also
be called competitive nets, The extreme form of these competitive nets is called winner-rake~all. The name
itself implies rhat only one neuron in the competing group will possess a nonzero output signal at the end of
competition.

i
I
.I
~
148 Unsupervised Learning Networks

There exist several neural networks that come under this category. To list out a few: Maxnet, Mexican hat,
Hamming net, Kohonen self-organizing feature map, counterpropagation net, learning vector quantization
(LVQ) and adaptive resonance theory (ART). These networks are dealt in detail in forthcoming sections. In
me Cl..'ie of unsupervised learning, the net seeks {0 find patterns or regularity in the in pur data by forming
clusters. ART networks are called clustering nets. In these cypes of clustering nets, there are as many input
units as an input vector possessing components. Since each output unit represents a cluster, the number of
output units will limit the number of clusters that can be formed.
The learning algorithm used m most of these nets is known as Kohonen learning. In this learning, rhe
units update their weights by forming a new weight vector, whi.::h is a linear combination of the old weight
vecror and the new input vecror. Also, the learning continues for the unit whose weight vector is closest
to rhe input vecwr. The weight upd.ation formula used in Kohonen learning for output cluster unit j is
given a5

Wj(new) = wej(old)+a [x- wej(old)]

where x is the input vector; wej the weight vector for unit j; a the learning rare whose value decreases
monotonically as training continues. There exist two methods to determine the winner of the network during
competition. One of the methods for determining the winner uses the square of the Euclidean distance
between the input vector and weight vector, and the unit whose weight vector is at the smallest Euclidean
distance from the input vector is chosen as the winner. The next method uses the dot product of the input
vector and weight vector. The dot product between the input vector and weight vector is nothing but the net
inputs calculated for the corresponding duster units. The unit with the largest dot product is chosen as the
winner and the weight updation is performed over it because the one with largest dot producr corresponds to
the smallest angle between the input and weight vectors, if both are of unit length. Borh the methods can be
applied for vectors of unit length. But generally, to avoid normalization of the input and weight vectors, rhe
square of the Euclidean distance may be used.

I 5.2 Fixed Weight Competitive Nets

These competitive nets arc those where the weights remain fixed, even during uaining process. The idea of
competition is used among neurons for enhancement of contrast in their activation funcrions. In this section,
rhree nets- Maxntr, Mexican har and Hamming net- are discussed in detail.

I 5.2.1 Maxnet

In 1987, Lippmann developed the Maxner which is an example for a neural net based on competition. The
Maxner serves as a sub net for picking the node whose input is larger. All the nodes present in this subnet
are fully interconnected and there exist symmetrical weights in all these weighted interconnections. As such,
there is no specific al~orirhm to train Maxnet; rhe weights are fixed in this case.

5.2. 1.1 Architecture of Maxnet

The_architecrure ofMaxnet is shown in Figure 51, where fixed symmetrical weights are present over the
weighted interconnections. The weights between the neurons are inhibitory and fiXed. The Maxnet with this
structure can be used as a su~net to select a particular node whose net inpm is the largest.
5.2 Fixed Weight Competitive Nets 149

_,

_,

Figure 51 Maxnet structure.

5.2.1.2 Testing/Application Algorithm of Maxnet

The Maxnet uses the following activation function:

jX if x> 0
f(x) = )o if x~O

The resting algorithm is as follows:

I Step 0: Initial weights and initial activations are ser. The weight is set as [0 < < lim], where "m" is I
the total number of nodes. Let

"9(0) = input to the node xj

and
j I if i=j
Wij = l-f; if ;ofj

Step 1: Perform Steps 2-4, when stopping condition is false.

Step 2: Update the activations of each node. Forj = 1 tom,

-'!(new)= r[j(old)-e ~x,(old)]

'""'
Step 3: Save rhe activations obtained for use in the next irerarion. For j = 1 to m,

Xj(old) = x1(new)

Step 4: Finally, test the stopping condition for convergence of the network. The following is the stopping
condition: If more than one node ha5 a nonzero activation, continue; else stop.

In this algorichm, the input given to the function/(-) is simply the total input to node Xj from all others,
including its own inpur.
 Unsupervised Learning Networks
150

I 5.2.2 Mexican Hat Net

In 1989, Kohonen developed the Mexican hat network which is a more generalized contrast enhancement
network compared to the earlier Maxner. There exist several "cooperative neighbors" (neurons in close prox-
imity) to which every other neuron is connected by excitatory links. Also each neuron is connected over
inhibitory weights to a number of"competitive neighbors" {neurons present farther away). There are several
oilier fanher neurons ro which the connections between the neurons are nor established. Here, in addition to
the connections within a particular layer Of neural net, the neurons also receive some orher external signals.
This interconnection pattern is repeated for several other neurons in the layer.

5.2.2.1 Architecture
The architecture of Mexican hat is shown in Figure 52, with the interconnection pattern for node X;. The
neurons here are arranged in linear order; having positive connections between X; and near neighboring units,
and negative connections between X; and farther away neighboring units. The positive connection region is
called region of cooperation and rhe negative connection region is caJled region of competition. The size of
these regions depends on the relative magnitudes existing between the positive and negative weights and also
on the topology of regions such as linear, rectangular, hexagonal grids, ere. In Mexican Hat, there exist two
symmetric regions around each individual neuron.
The individual neuron in Figure 5-2 is denoted by X;. This neuron is surrounded by other neurons Xi+ I,
X;_ 1, X;+2, X;-z, .... The nearest neighbors ro the individual neuron X; are X;+I, X;- I. Xi+2 and Xi-2
Hence, the weights associated with these are considered to be positive and are denoted by WI and w2. The
farthest neighbors m the individual neuron X; are taken as Xi+3 and X;-3, the weights associated with these
are negative and are denoted by w3. Ir can be seen chat X;H and X;-4 are not connected to the individual
neuron X;, and therefore no weighted interconnections exist between these connections. To make it easier,
the units presenr within a radius of2 [query for unit] to the unit X; are connected with positive weights, the
units within radius 3 are connected with negative weights and the units present further away from radius 3
are not connecred in any manner co the neuron X;.

5.2.2.2 Flowchart
The flowchan for MexiCJ.n hat is shown in Figure 5-3. This dearly depicts the flow of the process performed
in Mexican har m=rwork.

w, w,

G) (X,, xi-2 x,.,) @

.
Figure 52 Srructure of Mexican hac.
 5.2 Fixed Weight Competitive Nets 151

Initialize radius of region of

interconnection (R2), radius of+ Ve
reinforcement (R1), total no. of iterations t.n.u

Se!.initial weights
Wk=C 1; k=OtoR,(C1>0)
wk= 0:!; k= R,+1 to~ (0:!<0)

1<1,..) No I

Yes

Compute net input, lor i"' 1 to n

A, -R,-1 R,.
X;=ciLxoj+e;:, LXo.,+c2 Lxo,.kl
k~-R, k"'-R, li=R,>I

Apply activation functions

xj = min{x,.,,, max(O, x,)] i = 1 to n

Figure 53 Flowchart of Mexican hat.

}'
 Unsupervised Learning Networks
152

5.2.2.3 Algorithm
The various parameters used in rhe training algorithm are as shown below.

Rz = radius of regions of interconnections

X;+k and X;-k are connected m the individual units X; fork= 1 w R2 .
R1 =radius of region with positive reinforcement (RI < Rz)
W k = weight between X; and Ute units X;+.l: and Xi-k
O(k~R 1 , Wk = positive
Rt::::;; k:s;;; Rz, Wk = negative
s =external input signal
x = vector of activation
xo = vecwr of activations at previous time step
t10ax = total number of iterations of contrast enhancement.

Here the iteration is started only with the incoming of the external signal presented ro the network.

Step 0: The parameters R1, R2, tmax ate initialized accordingly. Initialize weights as

WJ: = C] fork= 0, ... ,R1 (where q > 0)

WJr = '2 fork= R1 +I. ... , R1 (where t"2 < 0)

Initialize xo = 0.
Step 1: Input the external signals:

x=s

The activations occurring are saved in array xo. Fori= I to 11,

xo; = x;
Once activations are stored, set iteration counter t = l.
Step 2: \'<'hen r is less rban lma.~ perform Steps 3-7.
Step 3: Calculate net input. Fori= 1 m n,

R1 -R1-I Rz

x; = q Z::: xoHl- + C'2 L xo;H + C'2 L xo;+f

k="-R 1 k=-R2 k="R 1+1

Step 4: Apply the activation function. Fori= l to 11,

x; = min[Xmax max(O,x;)]

Step 5: Save the current activations in xo, i.e., fori= 1 w n,

XQj =Xj
 153
5.2 Fixed Weight Competitive Nets

Step 6: Incremem the iteration counter:

t=t+l

Step 7: Test for stopping condition. The followi'ng is the stopping condition:
If t < tmax then continue
- I
The positive reinforcement here has the capacity to increase the activation of units with larger initial
activations and the negative reinforcement has the capacity to reduce the activation of uniL'i with smaller
initial activations. The activation funcrion used here for unit X; at a particular rime instant "t" is given by

x;(t) = 4;(t) + Z:: W!Xi+k + k(r -1)1

'
The terms present within the summation symbol are the weighted signals that arrived from other units at the
previous tirrie step.

I 5.2.3 Hamming Network

The Hamming network selects stored classes, which are at a maximum Hamming distance (H) from ilie
noisy vector presented at the input (Lippmann, 1987). The vectors involved in this case are all binary and
bipolar. Hamming network is a maximum likelihood classifier that determines which of several exemplar
vectors (the weight vector for an output unit in a clustering net is exemplar vector or code book vector for the
pattern of inputs, which the net has placed on that duster unit) is most similar to an input vector (represented
as an n~tuple). The weights of the net are determined by the exemplar vectors. The difference between the
tom! number of components and the Hamming distance between the vecrors gives the measure of similarity
between the input vector and stored exemplar vcctors.lt is already discussed in Chapter 4 that the Hamming
distance between the two vectors is the number of components in which the vectors differ.
Consider two bipolar vectors x andy; we use a relation

x;=a-d
where a is the number of components in which the vecrors agree, d the number of components in which the
vectors disagree. The value "n- d" is the Hamming distance existing between two vectors. Since, the total
number of components is 11, we have,
n=a+d
i.e., d=n-a

On simplification, we get
xy=a-d
xy= a- (n -a)
xy=2a-n
2a=xy+n
1 1
a= -(xy) + -(n)
2 2
154 Unsupervised Leaming Networks

From the above equation, it is clearly understood that the weights can be set to one~half the exemplar vecror
and bias_can be set initially to n/2. By calculating the unirwirh the largest net input, the net is able to locate a
panicular unit that is closest ro the exemplar. The unit with the largest net input is obtained by the Hamming
net using Maxner as its subner.

5.2.3.1 Architecture
The architecture of Hamming network is shown in Figure 5~4. The Hamming network consists of two layers.
The first layer compmes the difference between the rmal number of componentS and Hamming distance
between the inpuc vector x and the stored pattern of veaors in the feedforward path. The efficient response
in this layer of a neuron is the indication of the minimum Hamming distance value between the input and the
category, which this neuron represents. The second layer of the Hamming nei:\Vork is composed of Maxnet
(used as asubnet) or a Winner-take-all network which is a recurrent network The Maxnet is found to suppress
rhe values at Maxnet output nodes except the initially maximum output node of rhe first layer.
The function ofMaxnet is to enhance the initial dominant response of the node and suppress others. Since
Maxnet possesses recurrent processing, the jth node is found to respond positively while the response of all
the remaining nodes decays to zero. This result needs a positive self-feedback connection with itself and a
negative lateral inhibition connection.

5.2.3.2 Testing Algorithm

The given bipolar input vector is x and for a given set of "m" bipolar exemplar vectors say e(l),.
e(j), ... , e(m), the Hamming network is used to determine the exemplar vector that is closest m the input

nf2 (1

y,(O)
y,ll

y2(0)
..,. Y/'''l

Y?' rrl

Ym101
Ym(bl)

'---
~
__) \,
~
Hamming distance matching Maxnel
Figure 54 Structure of Hamming network.
 155
5.3 Kohonen Self-Organizing Feature Maps

vector x. The net input entering unit Yj gives the measure of the similarity bmveen the input vector and
exemplar vector. The parameters used here are the following:
n ::::: number of input units (number of comp.onems of input-output vector)
m ::::: number of output units (number of components of exemplar vector)
e(j) ::::: jth exemplar vector, i.e.,
<(j) = [e 1 (j), ... , e;(j), ... , e,(j)]

The testing algorithm for the Hamming Net is as follows:

'
Step 0: Initialize the weights. For i ::::: 1 to n and j ::::: 1 to m,
e;(j)
Wij=-2-

Initialize the bias for storing the "m" exemplar vectors. For j::::: 1 to m,
n
bj= 2
Step 1: Perform Steps 2-4 for each input vector x.
Step 2: Calculate the net input to each unit Yj, i.e.,
..
y;,y::::bj+ Lx;wij, j:::: ltom
i~l

Step 3: Initialize the activations for Max net, i.e.,

Jj(O) ::::: Yinj j::::: 1 ro m

J Step 4: Max net is found IO iterate for finding rhe exemplar that best marches the in pur panerns.

The Hamming nei:\Vork is found IO retrieve only the closest class index and not rhe entire vector. Hence,
the Hamming network is a classifier, rather than being an asso.:iarive memory. The Hamming network ca.n
be modified to be an associative memory by just adding an extra lay~r over rhe Maxner, such that the winner
unit, y;(k + 1), present in the Maxnet may trigger a corresponding stored weight vector. Such an associative
memory network can be called a Hamming memory network.

I 5.3 Kohonen Self-Organizing Feature Maps

I 5.3.1 Theory

if~
'\

l
 I
I
156 Unsupervised learning Networks

y,

0 0
0
0
0

w"
x, X, x, x.
Figure 55 One-dimensional feature mapping network.

ropology preserving map. For obtaining such feature maps, it is requjred ro 6nd fcl=rg:'Ri::;::-ra!~~
which consim of neurons arranged in. a ane-djmensional array or :l two-dimensional array. To depict chis, a
typical rle'"rwork srruaure where each component of the inpur vecroiXIs connected ro eaCh 'of ilie nodes is
shown in Figure 5-5.
On the orhe~ hand, if the input vector is two-dimensional, the inputs, say x(a, b), can arrange themselves
in a two-dimensional array defining rhe input space (a, b) :lS in Figure 5-6. Here, rhe n.vo layers are fully
connected,
The [Qpological preserving properry is observed in the brain, bur nor found in any other arrificial neural
nenvork. Here, there are m ourpur cluster units arrangeci in a one- or nvo-dimensional array anCthe input
signals are n-mples. The cluster (output) units' weight vector serves as an exemplar of che inputfaRe.!l!,
rhar is assorred with that duster. At rhe rime of self-organization, the we1ghr vector of the duster unit
which marc es the input pattern very do~ely is chosen as the winner unit. The closeness of weight vector
of cluster unit ro the input pattern may be based on the square of rhe n;tinimum Euclidean distance. The
weights are updated for the winning unit and irs neighboring units. It- should be noted that the weight
vectors of the neighboring units are nor dose to the mput pattCrn and rhe connective weights do not multiply
the signal sem from the in pur units to rhe cluster unirs until dot product measure of similarity is being
used.

I 5.3.2 Architecture

Consider a linear array of cluster Wlits as in Figure 5-7. The neighborhoods of the units designated by "o" of
radii N;(k1), N;(k2) and N;(k,), k1 > k, > k,, where k1 = 2, k2 = I, k3 = 0.
For a rectangular grid, a neighborhood (N;) of radii kt, ~ and ~ is shown in Figure 5-8 and for a
hexagonal grid rhe neighborhood is shown in Figure 5-9. In all the three cases (Figures 5-7-5-9), the unit wirh
 157
5.3 Kohonen Self-Organizing Feature Maps

0 0 0
0 0 .. 0
0 eo

0 0
0 0 0
0
0 0
0
0

Figure S-6 Two-dimensional feature mapping network.

o} o} 0 0 0
0 0 (o (0 [#]

t
N,(k,}

N,(k,}

NJ(k1)
r-~------
Figure 57llinear.. ar.r.l)l-Of.d~ster
-------unit'S)
-

:~" :U'W,~PUs"~-~ ..~~-!:1"-~i~.S.J!U..J;. and the orher units-ate~~ by "o." In both rectangular and hexagonal
gnds, k1 > ~ > k3, where kt = 2, k]_ =
1, k3 0. =
For rectangular grid, each unit has eight nearest neighbors but there are onl six nei bars for each unit in
the case of a hexagon grid. Missing neighborhoods may just be ignored. A typical architecture ofKohonen
self-organizing feature map (KSOFM) is shown in Figure 5-10.
158 UnsupeNised Learning Networks

0 0 0 0 0 0 0
0 0
0 0 0 0 0
0 0 0 0 0 0 0
N,(k,)
0 o o[!Jo 0 0
N,(k,)
0 0 0 0 0 0 0

0 0 0 0 0 h,- - Ni(k1)
0 0 0 0 0 0 0

Figure 5~8 Rectangular grid.

~1
:Si ~::~:
N,(k,)

Figure 59 Hexagonal grid.

x,
y,

X,
Y,

x,
Ym

Figure 51 0 K.'ohonen self-organizing feamre map architecrure.

,, \

I 5.3.3 Flowchart c;\'

The flowcharr for KSOFM is shown in Figure 5-11, which indicates the flow of training process. The process
is continued for particular number of epochs or rill the learning me reduces to a very small rate.
The architecture consists of two layers: inpur layer and output layer (duster). There are "n" units in the
input layer and "m" units in the output layer. Basically, here ilie winner unit is identified by using either dot
producr or Euclidean distance method and the weight updation using Kohonen learning rules is performed
over the winning duster unir.
 5.3 Kohonen Self-Organizing Feature Maps
159

For
~ach inpu!)>-_:,;N:.o_ _ _ _ _~
.vector X 1

-,.-------------,
''
''
''
'
''
Calculate square of Euclidean distance ''
D(j) (x1- w~)~
=I.
,.,

'
''
'
'
--------- \ -- ... __ ---------'

'--------------- \ ~~ .. -~I
---------------'

1-
l _.__ - I

Test
{ );
1 ~+ 1) = reduced
No to specified c
level

Figure 511 Flowchart for training process ofKSOFM.

I
l
160 Unsupervised Learning Networks

I 5.3.4 Training Algorithm

The seeps involved in the craiiting algorithm are as shown below.

Step 0: uuu<U,...,. "'"' ....... , """' w, . ._...... ,. . . u, .. Y<Uu...., '"""Y u ... <l.>OIUIJI

n to reflect that prior knowledge.

Set ropological neighborhood parameters: As dustering progresses, che radius of che neighbor-
hood~
Initialize the learning rate a: It should be a slowly decreasing fUnction of time.
Step 1: Perform Steps 2-8 when stopping condition is false.
Step 2: Perform Steps 3-5 for each input vector x.
Step 3: Compute the square of the Euclidean distance, i.e., for each j = I to m,

" m
D(j) = L L (x;- Wij) 2
i=IFI

Step 4: Find the winning unit index J, so that DU) is minimum. (In Steps 3 and 4, dot produce method
can also be used to find the winner, which is basically the calculation of net input, and the winner
will be rhe one wirh the largest dot product.)
Step 5: For all unirs j within a specific neighborhood ofJ and for all i, calculate the new weights:
I" ------~
L_vi;(new) = w;;~_!Vij{t?_l~U
or 1/Jij(new) = (1- ct )wij(old)+ ax,-

Step 6: Update the learning rare a using the formula ct (t + 1) = 0.5a (t).
Step 7: Reduce radius of topological neighborhood at specified time intervals.
Step 8: Test for stopping condition of the network.

Thus using this training algorithm, an efficient training can be performed for an unsupervised learning
nerwork

I 5.3.5 Kohonen Self-Organizing Motor Map

The extension ofKohonen feature map for a multilayer network involve.~ rhe addition of an association layeT
to the output of the self-or-ganizing feature map layer. The output.!!_Ode is found to assnciare rbe desired omp_u~
values with cerrain input vegors. This type of architecture is called as Kohonen self-organizing motor map
(KSONfM; ffitrer, 1992) and layer that is added is called a motor map in w.h.U::h the movement command,
are being mapped into two-dimensional locations of excitation. The architecture of KSOMM is shown in
Figure 5-12. Here, rhe fearure map is d this acts as a competitive network which classifies the_
ingur vectors. The fearure map is mlined as discussed in Section . . . em.Q[Qrli'iap formation is based
on the learning of a control task. The motor map learning may be either supervised or uns4.perviscd lcaming \
and can be performed by ddra learning rule or outsrar learning rule (to be discussed later). The motor rna~/ "\r
learning is an extension ofKohonen's original learning algorithm. -~~b , . ~c- '~ If
J \ , ."; ':.('"' ' J- .
\t~_, h\,;j_ .\:_;-)

\,.
.. ;_\ \ ,. 1 (r-
I
,l
5.4 Learning Vector Quantization 161

Motor map

Actions
performed

Feature map
0
0
0
Partially connected
(unsupervised or
supervised learning)

x, , / Fully connected - unsupervised learning

x,. 'X,

Figure 512 Architecture ofKohonen self-organizing motor map.

~
c
\(-
r
I -
d' (-

(
''- ;'

..Q) (1- ~-t-

1 5.4 Learning Vector Quantization L' 0 -"' I:.,Qi

i 5.4.1 Thea!)'

~__E_~g__v~~on (LV~rocess ofclassi n the anerns, wherein each output unit represents
a particular class. Here, for each class several units should be used. The output unit we1g t vector JS e t e
reference vector or code book vecwr for the class which the unit represents. This is a special case of competmve
net,Wlitcb uses supeCVised learnmg methodology. Durmg tra.mmg,Uleoutput units are found to be positioned
ro approximate the decision surfaces of the existing Bayesian classifier. Here, the set of training patterns with
known classifications is given to the network, aiOrigwith an initial distribution of the reference vectors. When
the uaining process is complete, an LVQ net is found to classify an input vector by assigqjng it to the same
class as that of the ourpur unit, which has its weight vector _ygr. ciQS..~.tbe input yeqQ~ Ibm IY_Q.i~.~
classifier paradigm that adjusts the boundaries between categories to minimize existing misclassification. LVQ
is used for optical character recognition, converting speech mro phonemes and ot1ier apphcau.Oris as well.
LVQ net may resemble KSOFM net. Unlike LVQ, KSOFM output nodes do nor correspond to the known
classes but rather correspond to unknown clusters that the KSOFM finds in the data autonomously.
. v'-"''1
)- ,.._,
"\
'
\--:/_(''
,.,"'"'
~ r--1'-':'' .... ;-~\.:~ ~)\ uJ_,,..
L
-
5.4.2 Architecture
n,v
--:?
"

Figure 5-13 shows the architecture ofLVg, which is almost rhe.same as rhar ofKSOFM, with the difference
being that in the case ofLVQ he to 010 ical strucrure at the out ur unir is nor bein consi ere . Here, each
output unit has knowledge about what a own r
From Figure 5-13 it can be noticed that there extsts mput layer with "n" unic;; and outp ayer with "m"
units. The layers are found to be fully interconnected with weighted linkage acting over the links.
162 Unsupervised Learning Networks

x\ (x\. w11 1----.y,

x, (X, )---y,

x. rx~
n)
w::::~
Ml~ }---~ym

Figure 513 Archireccure of LVQ.

I. 5.4,3 Flowchart

The parameters used for rhe training process of a LVQ include rhe following:

x = uaining vector (x!, .. , x;,. . , x11 )

T =category or class for the training vecmr x
Wj =weight vector for jrh output unir (Wlj ... , Wij, . .. , w,y)
c1 =cluster or class or category associated with jrh output unit.

The Euclidean distance of jth output unit is D(j) = L (x; - w;j) 2 . The flowchart indicating the flow of
uaining process is shown in Figure 5-14.

I 5.4.4 Training Algorithm

In case of training, a set of training in pur vectors with a known classification is provided with some initial
distribution of reference vecmr. Here, each ourpur unit will have a known class. The objective of the algorithm
is ro find rhe output unit that is closest ro the input vector.

f Step 0: Initialize the reference vectors. This can be done using the following steps. - I
From the given set of training vecmrs, rake the first "m'jnumber of dusters) training vectors and
use them as weight yeqors, the remaining vecrors can be used for training. ~
Assign the i!l_itial weights and..sias.sifi.carions randomly.
K~meansct"ustering meiliod.

Ser inirialleaming rate Ct.

Step l: Perform Steps 2-6 if rhe stopping condition is false.
Step 2: Perform Steps 3-4 for each training input vector x.

5.4 Learning Vector Quantization 163

Initialize weight vectors

and learning rate a

<

Fo' No
each input
vector x
');

"''
. j .\
\
,. ! ~~

._I I
Input T '
I target I '

II No
T=- C
'

Yes

Update weights using Update weights using

W {new) == w1{old) + a[x-wJ(old)] w1(new) == w;(old) - a[x- w;(old)]
1

Reduce learning rate

a{/+1) = 0.5 a(/)

II
No a reduces
to a negligible
ue

Figure 514 Flowchart for LVQ.

~
'
.

"'
164 Unsupervised Learning Networks

Step 3: Calculate the Euclidean distance; fori= 1 m n,j::::: 1 tom,

" m
D(j) = L L (x,- Wij)
2

i=l j=l

Find rhe winning unit index J, when DO) is minimum.

Step 4: Update the weights on the winning unit, Wj using the following conditions.

If T= q,then WJ(new) = Wj(old)+a [x- WJ(old)]

lfT;o' q,then WJ(new) = WJ(old)-a[x- Wj(old)]
Step 5: Reduce the learning rate a.
Step 6: Test for the smpping condition of the training process. (The sropping conditions may be fixed
number of epochs or if learning rare has reduced ro a negligible value.)

I 5.4.5 Variants

There exists several variants ofLVQ net proposed by Kohonen. These include LVQ2, LVQ2.1 and LVQ3. In
the LVQ aJgorithm, only the reference vector that is closest ro the in ut vector is updated. The movement
it moves is based on whether or nor the winning vecm[ b,c:lo..oM~B.JIIC class as e input vecror. n the
developed versions ofLVQ, rwo vectors called winner vector and runner-up vector learil'Sif'SeVCral'Conditions
are satisfied. Here two distances have to be calculated. Learn in rakes lace only if ilie input is ~E.!Y~mately
~~~-~arne distancrfuim wmner_~nd r~ One distance is from winner to mput ayer and the other is from
runner to input layer.

5.4.5. 7 LVO 2
The conditions over which bQ[h vectors are modified in ri1e case ofLVQ 2 arc the following.
I. The winner and the runner-up unit belong to different classes.
2. The rUnner-up vector is of the same class as the input vccmr.
3. The distances between the input vector and winner and becween the input vector and runner-up are almost
equal to each other.

If xis the current input vector,y! the reference vector closer ro x(winner),yl the reference vector next closer to,)\ .0
x (runner-up), d1 the distance from x ro Yl, d2 the distance from x to Jl, then the conditions for the updaci~ ~ rS'!.
.

of the reference vector can be defined as follows: . . ~ ~ ~o'

~ '~ ,r .

and
- > (1-e)
d2
~
J;" <(He)
"'l"
rJ

~
~
I
'"
"'>
)>/
o\J
"";;, ,"
'r'~
where the value of E is ba!ied on the number of uaining samples. The weight updacion formulas in this casey,~
are given by
Yl (t+ I) = Yl (t)- a (t)[x{t) - y, (t)] (belongs to different class)
y, (t + I) = )12 (t)+ a (t)[x(t) - y, (t)] (belongs to same class)

i
~.
5.5 Counterpropagation Networks
165

5.4.5.2 LVO 2. 7
In LVQ 2.1, the two closest reference vectors y 1, and J2c are taken. Here updating is done on the basis of the
requirements that (a) Ylr belongs to the correct dass for tfte given input vector; x and (b) Jlr does not belong
to the same class as x. LVQ 2.1 does not distinguish whether ~he closest vector is that re resentiri the correct
class or incorrect class for e given input. ' r t IS case are g1ven y
\{ .....
min [dlr' dl(]- > (I-E)
- ~_/
!.,_II r:S \
d2r d1r V' v )')
I . \.if

and max [d'', &::.]

d2r d1,
< (J+e) ""'~ '\"'"
L

Here, it is not sure whether xis closer to y 1( or to Jk When the above conditions are met, the following
weight updarion formulas are used. If the. reference vecwr ~elongs to the same class as input vector, then
y,,(t + I)= y,,(t)+. (t)[x(t) - y,,(t)]

else y,,(t+ 1) = y,,(t)-a (t)[x{t)- y,,(t)]

5.4.5.3 LVO 3

~
rh~ ~sest vecto are allowed to learn as long as the input vector satisfies the condition (take
) r,---------------
1
I\ min [d,, d,,]
d2/ d1r
:> (1-e)(l+e) \
,
~-~-- _ __/
The weight updarions are done in a similar manner as in LVQ 2"TJoneof the rwo closest vectors, yin belongs
to the same class as the input vecror x and the other vecror, Yln belongs to a different class. LVQ 3 extends
rhis training algorithm to provide training @r andy2( ~elo~g to the same class. The weight updates, here,
are given by the equation --
;,(t+ I)= y,(t)+P(t)[x(t)- y,(t)]

Replace y, withy!, or .Yln as rhe case may be. The learning rate {J(t) is a multiple of the learning rate Cl:'(l) that
is used if}lr and }lr belong ro different classes, i.e.,
p(t) ~ qa(t)

where q is b~nve!:!fl 3
I 5.5 Counterpropagation Networks
i
- 5.5.1 Theory

Counterpropagarion networks were proposed by Hecht Nielsen in 1987. They are multilayer networks ba5ed
on the combinations of the input, output and clustering layers. The applications of coumerpropagarion nets
are data compression, function approximacion and pattern association. The counterpropagation network is
basically constructed from an insrar--outstar model. This model is a three-layer neural nerwork that performs
input-output data mapping, producing an output vector yin response tO an input vector x, on the basis of
competitive learning. The three layers in an instar-outstar model are the input layer, the hidden {competitive)
166 Unsupervised Learning Networks

layer and the output layer. The connections berween the input layer and the competitive layer are the instar
structure, and the connections existing between the competitive layer and the output layer are the owsta:
structure. The competitive la}rer is going to be a winneHake~all network or a Maxnet with lateral feedback
connections. There exists no lateral connection within the input layer and the ourpur layer. The connections
between the layers are fuUy connected.
A coumerpropagation ner is an approximation of its training input vector pairs by adaptively con
strucring a lookuptable. By this method, se~eral data poims can be compressed to a more manageable
number oflookup-rable entries. The accuracy of the function approximation and data compression is ba.<ied
on the number of entries in the look-up-table, which equals the number of units in the cluster layer of
the net.
There are Q.VO stages involved in the training process of a counterpropagation net. The input vectors are
clustered in rhe first stage. Originally, ir is a.<isumed that there is no topology included in the coumerpropa-
gation network. However, on the inclusion of a linear topology, the performance of the net can be improved.
The dusters ~re formed using Euclidean distance method or dot product method. In the second stage
of training, the weights from the cluster layer units to the output units are tuned to obtain the desired
response. There are twotypesofcoumerpropagation nets: (i) Fullcounterpropagation net and (ii) forward-only
coumerpropagation net.

I 5.5.2 Full Counterpropagation Net

Full counterpropagarion net (full CPN) efficiendy represems a large number of vector pairs x:y by adaptively
conmucting a look-up-table. The approximation here is x:l, which is ba.<ied on the vector pairs X:y, possibly
with some distoned or missing elements in either vector or both vecwrs. The nerwork is defmed to approximate
a continuous function[, defined on a compact set A. The full CPN works best if the inverse function[-\
exisrs and is continuous. The vectors x and y propagate through rhe neWork in a counterflow manner to
yield output vecmrs x and l, which are the approximations of x andy, respectively. During competition,
the winner can be determined either by Euclidean distance or by dot product method. In case of dot product
method, the one with the largest net input is the winner. Whenever vectors are to be compared using the
dot product metric, they should be normalized. Even though the normalization can be performed without
loss of information by adding an extra component, yet to avoid the complexicy Euclidean distance method
can be used. On the basis of this, direct comparison can be made between the full CPN and forward-only
CPN.
For continuous function, the CPN is 3..'i efficient as the back-propagation net; it is a universal continuous
function approximator. In case of CPN, the number of hidden nodes required to achieve a particular level
of accuracy is greater than the number required by the back-propagation network. The greatest appeal of
CPN is its speed of learning. Compared to various mapping networks, ir requires only fewer steps of training
to achieve best performance. This is coinmon for any hybrid learning method that combines unsupervised
learning (e.g., instar learning) and supervised learning (e.g., outsrar learning).
As already discussed, the training ofCPN occurs in two phases. In the input phase, the units in the duster
layer and input layer are found to be active. In CPN, no topology is assumed for the cluster layer units; only
the winning units are allowed to learn. The weight updarion learning rule on the winning duster units is

Vij(new) = Vij(old)+ a [x;- Vij(old)], i==lton

w;j(new) = w,j(old)+ p (y; - w,j(old)], k=-1 tom
The above is standard Kohonen learning which consists of competition among the units and selection of
winner unit. The weight updacion is performed for the winning unit.
5.5 Counterpropagation Networks
167

In the second phase of training, only the winner unit J remains active in the cluster layer. The weights
between the winning cluster unit Jand the output units are adjusted so that the vector of activations of the units
in the Y-ourput layer is~ which is an approximation to the input vector y and X* which is an approximation
to the input vector x. The weight updations for the un~ts in theY-output and X-output layers are

Ujllnew) = "JI(old) + a(y,- uj;(old)], k = l tom

tji(new) == tj;(old) + bf_xi- tj1(old)1, j == 1 ton

This is Grossberg learning, a more general case of outstar learning. Outsrar learning is found to occur for
all units in a particular layer; there exiSts no competition among those units. The form of weighr updation
is similar for Kohonen learning and Grossberg learning. The learning rule for the output layers can also be
viewed 3..'i delta learning rule. The weight change in all these cases is the product of the learning rate arid
the error. When tie occurs in the selection of winning unit, the unit with smallest index is chosen as the
winner.

5.5.2. 1 Architecture
The general structure of full CPN is shown in Figure 5-15. The complete architecture of full CPN is shown
in Figure 5-16.
The four major components of rhe instar-outstar model are the input layer, the instar, the competitive layer
and the oumar. For each node i in the input layer, there is an input value x;. An instar responds maximally to
the input vectors from a particular duster. All the insms are grouped into a layer called the competitive layer.
Each of the instar responds maximally to a group of input vectors in a different region of space. This layer of
instars classifies any input vector because, for a given input, the winning instar with the strongest response
identifies the region of space in which the input vector lies. Hence, it is necessary that the competitive layer
single outs the winning instar by setting its output ro a nonzero value and also suppressing the other outputs
ro zero. That is, it is a winner-take-all or a Maxnet-cype network. An outstar model is found to have all the
nodes in the output layer and a single node in the competitive layer. The outstar looks like the fan-out of
a node. Figures 5-17 and 5-18 indicate rhe units that are active during each of the \VO phases of training a
full CPN.
In rhe instar-outstar nerwork model, the competitive layer participates in both the insmr and outsrar
structures of the network. The function of these competitive insrars is to recognize an input pattern through
a winner-rake-all competition. The winner auivates a corresponding outsrar which associates some desired
output pattern with input pattern.

x(lnput) y"(Outpul)
lnstar-outstar network

x"(Output) y(lnpul)
1nstar-outslar network.

Figure 515 General suucrure offull CPN.

168 Unsupervised Learning Networks I
X-lnput Y-in put
layer layer

x,
~
~~
_lnstar
~
lnstar
~Y1 \

x, ( )-<----- y,

x, X,!---__ \ Nl'.V! XY\ I I _..\ 'm Ym I

"

y;----\ 1----x;

y;~----( )--~x;

)---x;

Outstar Oulstar X-output

layer layer

Figure 516 Architecture of full CPN.

 169 I

l
5.5 Counterpropagation Networks

, _____

x, "
v,. !:'\
w,. ~y, I'

}----~Yx
x, (

y.,
x, ,(yr .... ~-~~
'-...7
Y-input
Cluster
X-input layer
layer
layer
Figure 517 First phase of training of full CPN-

Outstar Outstar

y, yo\
~ ~(x. x,
I

y~ ykr------1 1----~',.

Xn" Xn"
Ym" Ym"
Cluster x-outpul
Y-output
layer layer
layer
Figure 518 Second phase of uaining of full CPN-

5.5.2.2 Flowchart
The flowchart for rhe training process of full CPN is shown in Figure 5-19. The parameters used in rhe CPN
are as follows:
x =input training vecror x = (x 1 , . , x;, ... , x 11 )
y =target output corresponding to input x,y == (y1,. . ,_y!.... . yml
Zj ==the omput of cluster layer unit Zj
Vij ==weight from X-input layer unit X; to cluster layer unit z;
iVIIj ==weight from Y-input layer unit Yk to cluster layer unit Zj
lljk ==weight from cluster layer unit z; toY-output layer unit Yk
tp ==weight from cluster layer unit Zj to X-output layer unit X/'
170 Unsupervised Learning Networks

Start

Initialize weights, learning rates

Start phastl' 1 uanm1y

For
'each trainin9)-----'>N=o-----------,
input pair 1

Find winner cluster unit J

I

~----- Fori=1tnn ------,

'------.r---_/ '

-----< Conlinue )-------~

I
;-----( r;;;:k=1lo0----:
1 :
Update weights
wkJ(new) = wk1(old) + P[y)-w~{old)]

----- --( Continue >---- --'

!
Reduce learning rates a& P
a(l+1) = 0.5 a(t)
P(t+r) ~ o.5 p(t)

L----1 Input stopping learning rates

a/f).P,(t)
II
I No (a(t+1) < a1(1j'
p(t+1) </I,(~

Yes

Stop phase 1 training

Figure 5~19 Flo"Ycharr for training of full CPN.

1
 5.5 Counterpropagation Networks 171

A'

Start phase 2 training

For
each inpui"-. ___.:N:;o::,__ _ _ _ _ _ _~
.vector pai~/
x:y

Yes
Set x-inputlayer activations to vector x
Set y-inpul layer activations to vector y
I
l AluA+- ,,~:o I I
Find winning'"''""'""'' '" "'"

, Fori= 1 ton/
I
I Update weights into z I
1
Vi(new) == V,(old) + a{x1-v (old)]
1

---------'
Fork= 1 tom ------1
'
i Update weights into z1to output layers (
1 1
w11 (new) = vt/old) + Plx,-v!'(old)]

,------- Fork=1tom
'
( Update weights from z1 to output layers (
: u11(new) = uk1(old) + a{yk- wll(o!d)] 1
I -- I

Continue

~------ Fori=1ton ----1

~(new)= t,.(old) + p[x,-t1 (old)J

'-------- ---------'
Reduce learning rates a & p
a(t+1) = 0.5 a(l)
p(t+1) ~ 0.5 p(t)

Input stopping learning rates

a,(t),p,(t)
No Yes

I
l1 Figure 5-19 (continued).
172 Unsupervised Learning Networks

X' =calculated approximation ro vector x

Y* = cakulared appr~ximarion to vector y
a, b = lear"ning rates for weights our from cluster layer
a, fJ = learning ra[es for weigh[S into cluster layer

The training phase is performed here in two stages. The sropping conditions here may be number of
epochs robe reached. So the training process is performM until the number of epochs specitled is completed.
The reduction in learning rate can also be a stopping condition. The formula for reduction of learning me is
a(t+ I) = 0.5 a(t), where a(r) is learning rare ar rime instant "t" and a(t + I) is learning rate of next epoch
for a rime insram "t+ I".

5.5.2.3 Training Algorithm

The steps involved in the training process of a full CPN are given below.

Step 0: Set the initial weights and the initial learning rare.
Step 1: Perform Steps 2-7 if stopping condition is false for phase I training.
Step 2: For each of the training input vector pair x: y prescmed, pertOrm Steps .~- S.
Step 3: Make the X-input layer activations to vector X.
Make the Y-in pur layer acrivadons to vector Y.
Step 4: Find the winning cluster unit.
If dor product method is used, find rhe cluster unit Zj wirh rarget IK't in pur: forj = ! top.
II It!

Zmj = L .\'il!ij + L.Ykll'kl

;~I l=l

If Euclidean distance method is used, find the cluster unit z1 whose squared distance from input
vectors is the smallest:

{)_, = L (x; - !l;j)! + L'" \Yt- - II'(/

i= I k= I

If rherc occurs a tie in case of selection of winner unit, the unit with the smallest index is rhe
winnt'r. Take rhe winner unit index as J.
Step 5: Update rhe weighrs over d1e cakulared winner unit Zj.

Fori= 1 ro 11, l'iJ(new) = l'if(old)+ t.l'[x;- 111j(old)]

Fork= 1 to Ill, ~~'.(:/(new}= IVkf(old)+{:l[yk- ll'kf(old)]
Step 6: Reduce the learning rare~.

a(<+ ll =O.Sa(<): fl<+ II =0.5fi(l)

Step 7: Test smpping condition for phase I training.

Step 8: Perform Stt::ps 9-15 when stopping cotldition is false for phase I[ training.

"

-
. L.
5.5 Counterpropagation Networks 173

Step 9: Perform Steps 10-13 for each training inpm pair x ;y. Here a and fJ are small constant values.
Srep 10: Make the X input layer activations to vec~or x. Make rhe Y-in pur layer activations to vector y.
Step 11: Find the winning cluster unit (use formulas from Step 4). Take the winner unit index as J.
Step 12: Update the weights entering imo unit ZJ-

Fori= 1 ton, v;j(new) = Vij(old)+ a[x;- v;l(old)]

Fork= 1 rom, Wkj(new) = w,j(old)+ft lYk- w,j(old)]

Step 13: Update the weights from unit Zj to ~he output layers.

Fori= 1 ron, l]i(new) = ~;(old) + b[x; - lj;(old)]

Fork= I rom, "Jk(new) = "I'(old)+ a[y, - "I'(old)]

Step 14: Reduce the learning rates a and b.

a(t + l) = 0.5 a(t); b(t + l) = 0.5 b(t)

I Step 15: Test stopping condition for phase II training. I
If during training process initial weights are chosen appropriately, then after the completion of phase I of
training, the cluster units will be uniformly distributed. When phase II of training is completed, the weights
to the output units will be approximately the same as rhe weights into the duster unit:.

5.5.2.4 Testtng (Application) Algorithm

A CPN once trained can be used for finding approximations X* andY* to che input-output vector pair X
and Y. The application algorithm for full CPN is as follows:

Step 0: Initialize rhe weights (from training algorithm).

Step 1: Perform Steps 2-4 for each input pair X: Y.
Step 2: Ser X-input layer activations ro vector X.
Set Y-in put layer activations to vector Y.
Step 3: Find the duster unit ZJ that is dosm to the input pair.
Step 4: Calculate approximations m x andy:

J xj = t}i; Yk = Ujlr J

One important variation of rhe CPN is operating it in an interpolation mode after the training has been
completed. Here, more than one hidden mode is allowed to win the competition, i.e., we have first winner,
second winner, third winner, fourth winner and so on, with nonzero output values. On making rhe total
srrengrh of these multiple winners normalized ro l, the coral output will inrerpolare linearly among the
individual vectors. To select which nodes to fire, we can choose all those with wejghr.vectors within a cerrain
radius of the in pur x. The interpolated approximations to x andy are then

x7 = Lzjt_;;; Yk = 'L:zptjlt
j j

By using interpolation, the approximation accuracy is highly increased.

174 Unsupervised Learning Networks

5.5.3 ForwardOnly Counterpropagation Net

A simplified verSion of full CPN is the forward-only CPN. The approximation of rhe function y = /(x) but
not of x = f(y) can be performed using forward-only CPN, i.e., ir may be used if the mapping from x toy
is well defined but mapping from y to xis not defmed. In forward-only CPN only the x-vectors are used to
form rhe clusters on the Kohonen units. Forward-only CPN uses only the x vectors to form ilie clusters on
the Kohonen units during first phase of training.
In case of forward-only CPN, first input vectors are presented to the input units. The cluster layer units
compete with each other using winneHake-all policy to learn the input vecmr. Once entire set of training
vecrors has been presented, there exist reduction in learning rate and the vectors are presemed again, performing
several itemions. Fim, the weights between the input layer and duster layer are trained. Then the weights
between ilie cluster layer and output layer are trained. This is a specific competitive network, with target
known. Hence, when each input vecmr is presemed m the input vector, its associated rarg-!t vectors are
presented to the output layer. The winning duster unit sends its signal to the output layer. Thus each of
the output unit has a computed signal (w;k) and die target value {yk). The difference between these values is
calculated; based on this, the weights between the winning layer and output layer are updated.
The weight updation from input units to cluster units is done using the learning rule given below: For
i:::: 1 to 11,

Vif(new) :::: Vif(old)+ a[x; - Vif(old)] = (1- a)vif(old)+ a xi

The we1ght updarion from cluster units to output units is done using following rhe learmug rule: For
k:::: 1 w m,

w;,(new} = w;>(old) + a[y,- Wjk(old)] = (1 - a)wjk(old) + ay,

The learning rule for weight updarion from the cluster units to output units can be written in the form of
delta rule when the activations of the duster units (zj) are included, and is given as

wp.-(new) :::: Wj.(.(old) + llZ}Jk - Wj.k(old))

where

1 ifj=J
Zj= Q if }oFf
\

This occurs when Wjk is interpreted as the computed output (i.e.,yk = wpJ. In the formulation of forward-only
CPN also, no topological strucrure was assumed.

5.5.3. 1 Architecture
Figure 5-20 shows the architecture of forward-only CPN. It consists of three layers: input layer, cluster
(compe-titive) layer and output layer. The architecture of forward-only CPN resembles the back-propagation
network, but in CPN there exists interconnections becween the units in the duster layer (which are nor
connected in Figure 5-20). Once competition is completed in a forward-only CPN, only one unit will be
active in that layer and it sends signal to the output layer. As inputs are presemed m the network, the desired
outputs will also j-,,. ~re.~ented simultaneously.
 5.5 Counterpropagation Networks 175

(Unsupervized) (Supervized)

X1 , (X1 Y, _y,

x, Yk~

x, \ - - - Ym" ____1!!!
Desired
outpul
a ~~ ~
----------.. Cluster ~
layer

Figure 520 Architecture of fo!"'Nard-only CPN.

5.5.3.2 Flowchart
,. The flowchart helps in depicting the training process of forv-,rard-only CPN and the manner in which the
weights are updated. The training is performed in rwo phases. The parameters used in flowchart and training
algorithm are as follows:
a, fJ :::: learning rate parameters where a::: 0.5 ro 0.8 and fJ = 0 to I. The rypical values of learning
rates may be a= 0.6 and f3 == 1
X= activation vector for input layer units, i.e.,
X= (xJ, ... ,.l:j, .. ,x11 )
\\x- v\\ == Euclidean distance between vectors X and V
Figure 5-21 shows the flowchart for training process of for.vard-only CPN.

5.5.3.3 Training Algorithm

The steps involved in rhe training algorithm of forward-only CPN are as follows:

Step 0: Initialize the weights a~d learning rates.

Step 1: Perform Steps 2-7 when smpping condition for phase I training is false.
Step 2: Perform Steps 3-5 for each of uaining input X.
Step 3: Set the X-input layer activations to vector X.
Step 4: Compute the winning cluster unit (J). If dot product method is used, find the cluster unit z;
with the largest net input:

z; 11j:::: L" x;v;j

i==l

f
176 Unsupervised Learning Networks

--------

Obtain X-input layer activations to vector x

Update weights lor unit z,

V,(new)= V,{old) + a[x1-v,(old)]

:______ < I
Continue )------~
.
'
:_----------< l
Continue )------- ----

l
Reduce learning rate
a{l+1)==0.5a(t)

Input stopping learning rates

a1(/),p1(t)

11
I No (,
a(l+1) < a1{1)

FigUre 521 Flowchart for training of forward-only CPN.

5.5 Counterpropagation Networks
177

"A'

-------<.. For each training input pair x: y

Set Input layer activations x to vector x

also, set output layer activations yto vector y

Calculate winning cluster unit use Euclidean (J)

distance or dot product

:-------( Fori== 1ton )--------:

: I ,
!
''
'''
'
'
1
--------"'\. VVIIUIIUCI / ---------'

\_ I"'Uil\= llU/f/ ;-----:

'' I '
'r:---:-----:--:--:-_!_---:-_ _ _ _---::,:
t Update weights from unit z1to output units t
f wit(new) = w,..(old) + fi[y~-wp.:(old)] '

''
'
~~------- ---------'

! "'' . ~- ",., I
I fInput stopping learning rate
value P, (t)

If
I No (p(r+1) < 1\(t)

Yes

Stop

Figure 521 (contin~d).

 178 Unsupervised Learning Networks

If Euclidean distance is used, find the duster unit ZJ square of whose distance from the input
pattern is smalleSt:

"
Dj= L(x;-v,ji
i=l

If there exists a tie in the selection of wiriner unit, the unit with the smallest index is chosen as
the winner.
Step 5: Perform weight updation for unit ZJ- Fori= 1 to n,

vu(new) ~ vu(old)+a[x;- vu(old)]

Step 6: Reduce learning rate a:

a(t+ I)~ 0.5a(t)

Step 7: Test the stopping condition for phase I training.

Step 8: Perform Steps 9-15 when sropping condition for phase II training is false. (Set a a small constant
value for phase II training.)
Step 9: Perform Steps 10-13 for each training input Pair x:y.
Step 10: Set X~input layer acrivarions to vecwr X. Sec Y-ourpur layer activations to vector Y.
Step 11: Find the winning cluster unit Q) [use formulas as in Step 4}.
Step 12: Update rhe weights into unit ZJ Fori= 1 ton,

v;j(new) = Vij(old)+ re[x; - Vi](old)l

Step 13: Update rhe weights from unit z; m the ourput units. Fork= I to m,

w;,.(new) ~ '"Jk(old)+fi (yk- w;,(old)]

Step 14: Reduce learning rare {J, i.e.,

fi (r+ I)~ 0.5fi (r)

Step 15: Test rhe stopping condition for phase II training.

The stopping condition for both phase I and phase II training may be the reduction in learning rare or number
of iterations to be performed.

5.5.3.4 Testing Algorithm

The testing algorithm used for forward.only CPN is given as follows:

Step 0: Set initial weights. (The initial weights here are the weights obtained during training.)
Step 1: Present input vecror X.
Step 2: Find unit] that is closest to vector X.

;
J:
 .

5.6 Adaptive Resonance Theory Network 179

Step 3: Set activations of output units:

Yk = Wjk

& in the case of full CPN, the forwardonly CPN can a)so be used in the interpolation mode. Here, if more
than one unit is the winner, with nonzero activation value, then
p
:Lzj~I
j=l

Hence the activation of the omput unit is given by

Yk = Z:0wjk
j
r'""
Use of interpolation mode results in increase of accuracy. ,"'
{J 0 \.\.I-'1
"t \.J'J-1'
.,. . '
,v \o"\ {)
5.6 Adaptive Resonance Theory Network
("'"''
,.
1 5,6.1 Theory
~ - - - ------ ---- --
-
------,
The adaptive resonance theory (ART) network, developed by)feven Grossberg and Gail Carpenter (1987),
is consistent with behavioral models. This is an unsupervised learning, based on competition, that finds
cate_s.ories auronomously and learns new categories if needed. I he adaptive resonance model was develOped
to solve the problem of instability occurring in feedforward systems. There are two types of ART: ART 1 and
ART 2. ART 1 is designed for clustering binary vecmrs and ART 2 is designed ro accept continuous-valued
vectors. In both the nets, input panerns can be presented in any order. For each pattern, presented to the
nerwo~~! .::'2. appropriate cluster unit is chosen and rhe w.dgb.ILof the cluster unit are adjusted to lec.the.cluster
unit learn the pattern. This ne!}York controls the degree of similarity of the patterns placed on the same cluster
!-i_':!!@udngrr:;Tn'ing, each training pattern may be presented several rimes. It should &e noted that the mpur
p. ~terns sho~l~ not be presented on the same cluster unit, when it is presented each tim~. On dtc basis. of r r;:
(~ f this, the Stab1h of the net IS defined as fhat wfierem a attern IS not esen e ~~-..; .
Lfhe stability may be achieved by reducing r e lear~. he ability of the network to respond m a new . I(>,
p~ttern equally at any stage oflear<J.ing is calleclaSPlastidrf.'" T ners are designed to possess the properties, '_-
stability and plastiCity. e ey concept o ART is t at t e stability plasticity can be resolved by a system( .J~
in which the network includes bottom-up (input-output) competiriyelearning combined with topdown
(output-input) learning. The instability ofinstar-oumar networks could be solved by reducing the learning
rate gradually to zero by freezing the learned categories. But, at this poim, the net mar lose its plasticity or
the ability to react m new data. Thus it is difficult to possess both stability and plasticity. ART networks are
designed particularly to resolve the srabiliry-plasticiry dilemma, that is, they are stable to preserve significant
past learning but neverthdess remain adaptable to incorporate new information whenever it appe:m:--

5. 6. 1. 1 Fundam tal Architecture

\]"h;' groups of neurons reused to build an ART network. These include:
1. Input processing neurons (FJ layer). xJ \

"
j
 180 Unsupervised Learning Networks

2. Clustering uni[S (f2 layer).

3. Control mechanism (controls degree of similaricy of patterns placed on the same duster).

uster
imerf.ice pomonas l'J
There exiH rwo sets of weighted inrerconnections for controlling the degree of similarity between the units

x
,; '
in the interface portion and the cluster layer. The bottom-up weights are used for the connection from F1 (b)
layer to F2 layer and are represented by bij (tth F1 unit to jth F2 unit). The top-down weights are used for the
connection from F2la: er to F1 (b) layer and are represented by tp (jth F2 unit to ith F1 unit). The'competitive
"V"" , ,

layer in this case is the usrer lay and ili~ cluster uni~ largest net input is rhe via:im to learn t1ie InEfti['
pattern, and the acrivat!gns o other h uruS are rna e I he Interface units combme the data from
input and cluster layer umts. On tlie bas1s of the Similarity bern'een the top-down weight vector and input
-\ veaor, rhe dust~ unit may be allowed ro learn rhe input panern. This decision is done by-~
unit on the basis of fie s1gnaiS it fece1ves (Mrn ;nrerhce fiorrion and input portion of rheF\k}'ef:en
cluster unit is not allowed to learn, it is inhibited and a new cluster unit is selected as rhe vicnm. . . .,

5.6.1.2 Fundamental Operating Principle

In ART network, presentation of one input _pattern forms a learning trial. The activations of aH the units
in rhe net are set ro zero before an input pattern is presented. fJl units in the F2 layer are maafv"~. On
presemaoon oT:i""Pitt~rn, the input sign:llSafe-sem continuously uriUitli"e earmng tn 1s campier d. There
exists a user-defined parameter, called vigilance parameter, which comro s t e egree o Similarity of the
patterns assigned to the same cluster unit. The function of the reset mechanism is ro control the state of each
node jn f, l~r. Each unit in F2 layer, at any time instant, can be in any one of rhe three stares mennoned
below. 0 t,;:{.l.. ,-, .-..~_.V-,. '1 -~ ~ ' .
IJ"--. L~~. I l : v
l. Active: Unit is ON. The activation in tl1iscase is equal to l. For ART l, d..= l andim. .ARib_Q_::,,A.< 1.
2. Inactive: Unit is OFF. The activation here is zero and the unit may be available w participate in competition.
3. Inhibited: Unit is OFF. The activation here is also zero but the unit here is prevented from participating
in any funher competition during the presentation of current inpm vector.

The ART nets can perform their learning in rn'O ways: Fast learning and slow learning. The weight updation
rakes place rapidly in fast learning, relative to the length of time a pattern is being presented on any pmicular-
leaming trial. In fast learnin , the wei hts reach equilibnum m each tna:t:-Oili:heCOnt.rnr:i, in slow learning
the weight change occurs slowly relative r c;....mne...ta or a earnin t;:ial and the weights do nm reach
<qu;lib<~m in each i . Mo<e "'""' have to b< <CS<m<d fo< slow "kWiing compa<ed to clm fm fast ~-
~gjor each learning nal, there occurs only minimum num ~ c ons m sow earnmg. n cas.e ~-> ~
of fast learning, the net is considered robe stabilized when each pattern !:lOses Its correct cluste . i'
The pattern rwork, hence the weights assocJat"'ed-~n ea custer unit stabilize
in the fast learnine: mode. The weight vectors obtain riare or t e e of m urpauerns used
in ART 1. In case of ART 2 network, the weights~ fast learning continue to chan~h time
a p:llie~~~resenred . . The net is found w stabilize only after fe\~60~ing parrern.
It iSnor easy to find equilibrium weights immediately for ART 2 as it is for ART l. In slow learning

.I'
5.6 Adaptive Resonance Theory Network 181

process, the weight changes do not reach equilibrium during any particular learning trial and more trifs
are required before the net stabilizes. Slow learning is gene[a}.ly not adopted for ART l. For ART 2, the
weights produced by-slow learning are far better than those produced by fast learning for panicular types
of da<a. ~-,----....:.__ ___:_ _ ___:.~_:__ _ _.:.___

-----
5.6.1.3 Fundamental Algorithm
This algorithm discovers clusters of a set of pauern vectors. The steps involved in various stages of training
algorithm are as follows:

\ Step 0: Initialize rhe necessary parameters." I

Step 1: Perform Steps 2-9 when stopping condition is false.
Step 2: Perform Steps 3-8 for each input vector.
Step 3: F1 layer processing is done.
S<ep 4: Perform Steps 5-7 when reset condition is uue.
Step 5: Find ilie victim unit to learn the current input pattern. The victim unit is going to be the F2 unit
(th~or inhibited) wirh the largest input.

Step 6: \!:.!)b) units combine rheir in ts from F 1(a) and F2.

Step 7: Test for reset condition.
If reset is true, then ilie current_ victim unit _is rejected (inhibited); go to Step 4. If reset is false,
then die currentV~~~ir-i~ accep~edf~r-l~arni~g; go to next step (Step 8).
Step 8: Weight updation is performed.
/ Step 9: Test for stopping condition. I
The ART network does not require all training patterns to be presented in the same order, it also accepts
if all patterns are presented in the same order; we refer to this as an epoch. The flowchart showing the flow
of trainmg process is depicted separately for ARI I tilid AR'r 2.

I 5.6.2 Adaptive Resonance Theory 1

Adaptive resonance theory 1 (ART 1) network is designed for binary input vectors. As discussed generally, the
ART 1 net consists of two fields of units-input unit (F1 unit) and output unit (F2 unir)-along with the reset
control unit for comrolling the degree of similarity of patterns placed on the same cluster unit. There exist
two sets of weighted interconnection path betwee~l and F2layers. The supplem;:nal unit present in the net
provides the efficient neural conrtoi of die learnmg process. Carpenter and Gross erg have designed ART 1
network as a real-time system. In ART 1 network, it is not necessary to present an input pattern in a particular
order; it can be presented in any order. ART 1 network can be practically implemem~d by analog circuits
governi!! the differential equati.ons, i.tte bottom-up and top-dow.n....weighrs are co.mffilled by d!ffi;rerlliaL
equations RT 1 network runs throug Out autonomously. lt does not require any external control signals
and can n stably with infmite patterns of input data.
ART 1 network is trained usin fast learnm method, in which the wei hts reach e uilibrium during each
learning trial. During this resonance B.lli'se rhe acrivadons ofF1 units do not chan'ge; hence e eqU11 rium
weights can be derermjned exaqly: The ART I network performs weU with perfect binary input patterns, but
it IS senmive to noise in the input data. Hence care should be taken to handlss,~e-~~

[~
~ '
"-'1)1)1.'
._.. ~
It(
.,.
~I t'" \lI. c (.,.(\ ""~
-c-)~---
 182 Unsupervised Learning Networks

5.6.2.1 Architecture
The ART 1 network is made up of twO units:

1. Computational units.
2. Supplemental units. I
In this section we will discuss in derail about these two units.
/ rt,
\ . _r
.I ~
' \.!)~ "1
Computational units
T'
The computational unit for ART 1 consisrs of the following: '-..,, v
1. Input units (Ft unit- boili input portion and interface portion).
2. Cluster units (F2 unit- output unit).
3. Reset control unit (controls degree of similarity of patterns placed on same cluster).
The basic architecture of ART I {computational unit) is shown in Figure 5-22. Here each unit present
in the input portion ofF1 layer {i.e., F1(a) layer unit) is connected to ilie respective unit in the interface
~ p~~.i.e., F1(b) layer unit). Reset control una has connecnons from each of F1 (a) and F,(b)
--.. ::._ unus. Also, each unit in F L(b) layer is connected through twO weighted interconnection pailis to each unit
:: in F2 layer and, the reset control unit is connected to every F2 unit. The X; unit of F1 (b) layer is connected
~....., to Yj unit of F2 layer through bo~eigh:~ (hy) and ~he Y1 unit of F 2 is connected to X; unit of F 1
0 through top-down weights (tji). Thus ART I includes a '29rmm-up comp;ri~i;; :rqjqg system combined
L---.Wilh a to -down oursrar learning system. In Figure 5~22 for simplicity on yghted mrerconnecuons
bij and ~i ares own, t e other units' weighted interconnections are l.~. a similar way. The duster layer (Fz
layer) unit is a competitive layer, where only ~ninhibiced node with the largest net input as nonzero
...a!;:_tivanon. ' - ...-

s,

'
) _,::

~s,)
F1(a) layer
(
J '("".<..
.._

'
,_.
'

F2 1ayer
input portion cluster unit

Figure 522 Basic archirecwre: of ART l.

 5.6 Adaplive Resonance Theory Network 183

Supplemental units
Figure 5-23 shows the supplemental unit interconnection involving two _&in conuol unitS along with one
reset unit. The discussion on supplemental uhits is imporcam based on theoretical point of view.
Difficulty faced by computational units: It is n~cess for these units to res on different! at different
stages of the i,ocess, and these are not su one ari of the .
ihe oilier dPculry is that the operation of the resetfnechanism is nor well defined for irs implementation
i~
The above difficulties are rectified by the introducriolllf two supplemental units (called as gain con-
trol units) G1 and G2, along with the reset contrOl unit F. These three units r_ep:ive signals &om and
send signals to all of the units in inpm lq.yer and cluster Ia er. In Figure 5-23, the excitatory weighted
signals are denoted by"+' an m 1 itory signals are indicated by"-." Whenever any unit in desig-
nated layer is "on," a signal is sent. F1(b) unit and F2 unit receive signal from three sources. Ft(b) unit
can receive signal froffi' either Ft (a) unit or F2 units or Gt unit. In the similar way, F2 unit reCeiVes sig-
nal from either F1(b) unit or~ control unit R or gain control unit G2. An Ft(b) unit or F2 unit
should receive rwO"eicitatory signals for them tobe on. Both F 1(b) uni{;iiaF2 unir can receive sig-
na.Is throu h three possible ways; cllts IS called as rwo-thu& rll1e. The F1(b) unit should send a s1gnal
whenever it receives input rom 1(a) an no 2 no e ts acove er an F2 node has been chosen in
competition, it is necessary that on y 1 umrs w ose mpur s1gnal and top-down signal match remain
constant. This is performed by the rwo gam c um s 1 2, m a mon w1 rwo-thirds rule.
Wh~er h unit is on, G1 unit is inhibited. When no F2 unit is on, each F1 interface unit receives a
signal from G1 unit; here, all of the units that eceive a positive input signal from the inp!J!_vecror pre-
sented fire. In the same way, G2 unit corurols the firing of F2 units, obeying fie two thirds rnle. fhe choice
of parameters and initial weights may also be based on rwo-rhirds rule. On the other hai1'if,tlte vigilance
matching is controlled by the reset control unit R. An excitatory signal is always sen'tro R when any unit
in F1(a) layer is on. The strength of ilie signal depends on how many F 1 (input) units are on. It should
be noted that the reset control unit R also receives inhibitory signals from the F1 interface units that are
on. If sufficient number of interface units is on, then unit "F" may be prevented from firing. When unir
"R" fires, it will inhibit any F2 unit fiat is on. This may fo~ce the F2 layer to choose a new winning
node. --------------- /
\,,'
,,
:}
J

F2 layer
(cluster units)

+
b,l I,,
F,(b) layer
(interface portion)

r J
(, +
+
Q-
F1(a) layer
(input portion)
\
Figure 523 Supplemenral unit of ART 1.
/" f
y

I \,(
--., i

.L /.'-": n \.
184 Unsupervised Learning Networks

5. 6.2.2 Flowchart of Training Process

The flowchart for the training process of ART 1 network is shown in Figure ?? . The parameters u..Sed in
flowchart and training algorithm are as follows:
n = number of components in training input vector
m =maximum number of duster units that can be formed
p =vigilance parameter (0 to 1)
bij = bottom-up weights (weights from Xi unit ofF 1(b) layer ro Yj unit ofFzlayer)
tp;::: top-down weights (weights &om Yj units ofFzlayer ro X; unit ofFr(b) layer)
s = binary input vecmr
Q'acg_vaciort~tor fur p,-futlayer ~
llxll =norm of vecror x that is defined as ilie sum of components of x;(i = 1 ro n)

Initially, binary input vecmr "s" is presented in the F1 (a~r'~~gnals are sent to the corresponding
X layer, i.e., Ft (b) layer. Each F r (b) layer sends the activation to the F2 layer over the weighted interconnection
paths. Each Fz layer unit then calculates the net input. The unit with the largest net input is selected as the
winnefind will hav~n"l,jj the other unitS atti'vation"will be 0. The winning unit is specified by its
index "J." Only this win unit can learn the current input pattern. Then the signal is send from Fz layer
to F, (b) layer over the topdown weights (i.e., sign s ger multiplied with topdown weights). The X units
present in the interface portion F1 (b) layer remain on, only if they receive a nonzero signal from bo_th F~:.,{a) "<:,"'
and F2 layer unirs. S 1/<
... Now we calculate the factor llx[l. The norm of vector x gives the number of components in w~. ~r
topdown weight vector for the winning F2 uiiifJJ.and nut vectors are both l his is ca:tled Match. The '"
ratio of norm of x, llxll, to norm of s, llsll, iS--called Match &no, w 1 r than or equal to vigilance
parameter, then both the top-down and bottomup weights have to be adjusted. This is callef{reset conditioii)

-
That is -~------ --------.. --- ------ -- --- -

If llxll/l\s!\ ?:_p, then weight updation is done. This testing condition is called reset condition.
,.
~

If llxll/llsll < p, then currenU!.!!!!.is rejected and another unit should be chosen. The current winning
cluster unit becomes inhibited, so this unit again cannot be chosen as a unit, on this part~ng
tna1 ,an d rh eacnvauomortne
. . ......,-,- F1 umtSaLe.!eset~!O.--;
. ,;._ \ ' 11..' ' -......._
- ------c_-_- / , .7-f"
This process is repeated umil a satisfactory match is found (units get accepted) or until all-the unilS are
inhibited.

5.6.2.3 Training Algorithm

The training algorithm for ART I nerwork is shown below.

I
I Step
.. . he parameters:
0 Inmahzet e::'
'/'
''a >1 an
d 0< P::S 1 t)
9:

Initialize the weights:

0 < bij(O) '>

?
. -_- J --
d 'i;(O) = 1

-
 5.6 Adaptive Resonance Theory Network 185

Step 1: Perform Steps 2-13 when stopping condition is fa)se.

Step 2: Perform Steps 3-12 for each of the training input.
Step 3: Set activations of all F2 units to zero. St!t the_accivacions ofF1 (a) units to input vectors.
Step 4: Calculate the norm of s:

llsii=I>

Step 5: Send input signal from F1 (a) layer.,to F, (b) layer:

Xj=Sj

u; rur eacu r~ode that is not inhibited, the following rule should hold: If Jj =/=- -1, then
...-+-- 't""'J

'ertorm Steps 8-11 when reset is uue. \ )

Step 8: Find] for YJ ?:.. Jj for all_nodesj. IfYJ = -I, then all the nodes are inhibited and note that this
pauern cannot be clustered.
Step 9: Recalculate activation X ofF1 (b):

--xr= sitfi

Step 10: Calculate the norm of vector x:

llxii=Z:>

Step II: Test for reset condition.

If llxll/llsll < p, then inhibit node], Yl = -I. Go back to step 7 again.
Else if )lx\1/llsll ?:.. p, then proceed to the next step (Step 12).
Step I2: Perform weight updation for node )__([<1$_rlearning):..----~
.. 'I
, b ( ax; \
ij new)= a-1 + ilxll \
L?(ne:2 = x; \ ____ ______.)
Step 13: Test for stopping condition. The following may be the stopping conditions:

a. No c~n weights.
b. No reset of units.
I c. Maximum number of epochs reached. I
When calculating ilie winner unit, if there occurs a tie, the unit with smallest index is chosen as winner.
Note that in Step 3 all the inhibitions obtained from th~ previous learning trial are removed. When YJ = -1,
the node is inhibited and it will be prevented from becoming the winner. The unit x; in Step 9 will be ON
only if it receives both an external signals; and the other signal from F2 unit to F 1(b) unit, tp. Note that tji
is either 0 or 1, and once it is set to 0, during learning, it can never be set back to 1 (provides stable learning
method).

-
 Unsupervised Learning Networks
186

,>v
d

Initialize weights,
0 < bf(O)<-L_,fp(0)=1
L-Hn

--------- Training for each Input vector)----------@

---------- Fori== 1ton ----------~

,------ Forj=1tom
'

For each node that is

not inhibited

'
~----: _____ _ 1---- I

Figure 524 Flowchart for rraining of ART l nerwork.

 5.6 Adaptive Resonance Theory Network 187

-.
p~~t)
reset
False I ---, I

All nodes inhibited and

pattern cannot be clustered

'~
"''-; 0'
~
/ "":-..
~
False

d
True
Weight updalion
_ ax,
b, (new)- a-1+11 xll
I~ (new) == x1

---- Continue ----@

---- ----

Test for
'Stopping conditiori'
False 1. no weight change
n 2. no unils reset
3. more no. of
epochs
.reat;:he

k Figure 524 (<onrinud).

188 Unsupervised learning Networks

The optimal values of the initial parameters are a= 2, p = 0.9, bij = 111 +nand fji == 1. The algorithm
uses fast learning, which uses the fact that the input pattern is presemed for a longer period of time for weights
ro reach equilibrium.

I 5.6.3 Adaptive Resonance Theory 2

Adaptive resonance theory 2 (ART 2) is for continuous~valued input vectors. In ART 2 network complexity
is higher than ART 1 network because much processing is needed in F1 layer. ART 2 network was developed
by Carpenrer and Grossberg in 1987. ART 2 necwork was designed to self-organize recognition categories
for analog as well as binary input sequences. The major difference between ART l and ART 2 networks is
the input layer. On the basis of the stability criterion for analog inputs, a three-layer feedback sysrem in the
input layer of ART 2 network is required: A bottom layer where the input panerns are read in, a rop layer
where inputs coming from the output layer are read in and a middle layer where the top and bottom patterns
are combined together to form a marched pattern which is then fed back to the top and bottom input layers.
The complexity in the F1 layer is essential because continuous-valued input vecmrs may be arbitrarily dose
together. The F 1 layer consists of normalization and noise suppression parameter, in addicion to comparison
of the bottom-up and top-down signals, needed for the reset mechanism.
The continuous-valued inputs presented to the ART 2 network may be of two forms. The first form
is a "noisy binary" signal form, where the information about patterns is delivered primarily based on the
components which are "on" or "off," rather than the differences existing in the magnirude of the components
chat are positive. In this case, fast learning mode is best adopted. The second form of patterns are those,
in which the range of values of the components carries significam information and the weight vector for a
cluster is found to be interpreted as exemplar for the patterns placed-on chat unit. In this type of pattern, slow
learning mode is best adopted. The second form of data is "truly continuous.''

5. 6.3. 1 Architecture
A typical architecture of ART 2 nerwork is shown in Figure 5-25. From the figure, we can notice that F1 layer
consists of six types of units- W, X, U, V, P, Q-and there are "n" units of each type. In Figure 5-25, only
one of these units is shown. The supplemental part of the connection is shown in Figure 5-26.
The supplememal unit "N" between units W and X receives signals from all "W'' units, computes the
norm of vector wand sends this signal to each of the X units. This signal is inhibimry signal. Each of this
(XI, ... , X;, ... , Xn) also receives excitatory signal from the corresponding W unit. In a similar way, there
exists supplemental units berween U and V, and P and Q, performing the same operation as done between W
and X. Each X unit and Q unit is connected to V unit. The connections between P; of the F, layer and Yj of
the F2 layer show the weighted interconnections, which mulriplies the signals transmitted over those paths.
The winning Fz unirs' activation is d (0 < d< 1). There exists normalization between Wand X, V and U,
and P and Q. The normalization is performed approximately co unit lengch.
The operations performed in F2layer are same for both ART 1 and ART 2. The units in Fzlayer compere
with each other in a winner-rake-all policy co learn each input pattern. The testing of re.set condition differs
for ART 1 and ART 2 networks. Thus in ART 2 network, some processing of the input vector is necessary
because the magnirudes of the real valUe-d-input vectors may vary more than for che binary input vectors.

5.6.3.2 Algorithm
A derailed descripcion of algorithm used in ART 2 network is discussed below. First, let us analyze the
supplemental connecrion between W; and X; units.
5.6 Adaptive Resonance Theel)' Network 189

(Reset unit)

a,
/

bf(q;)

I
Input
u, ...---. v, units

I
~x,)

w, ....__.;r x,

s,(input pattern)

Figure 525 ArchitectUre of ART 2 nerwork.

x,
Figure 5-26 Supplemental pan of connection berween Wand X.
190 Unsupervised learning Networks

Supplemental connection between W; and X; units

~discussed in Section 5.6.3,.1, there exist supplemental connections bern'~en Wand X, U and V, and P and
Q. Each of the Xi receives signa,l from w; units. After receiving, it will calculate ilie norm of w, 11 wll and then
sends that signal to each of the X units. Normaliz.adon is done in the F1 units from W to X, V to U and P to
Q. Each of the X; units are connected to V; and Q: units are also connected to Vi. The weights a, b, c shown
in Figure 5~25 are fixed. The weights on the connection path indicate the transformation taking place from
one unit to oilier (no multiplication takes place here), i.e., u; is transformed to au; bm not multiplied. When
signals are uansferred from F1 units m F2 units, i.e., from P; to Yj, the multiplication of weiglm is done.
The activation of the F2 unit is "d" which ranges between 0 and 1 (0 < d < 1). It should be noted that these
activations are continuously changing.
Processing ofF1 layer and F2 layer
For understanding the training algorithm of ART 2 network, it is important to know me processing of
F1 and F2 layers. In F 1 layer, the output activation from P; is p and output activation from Q; is q. The
activation vector q, which is the activation of~ units, should be equal to vector p, activation ofP; units that
is normalized approximately for unit length. U; unit performs similar process ofF1(a) layer of ART 1 and
P; unit performs similar process ofF1(b) layer of ART 1 network. The activation function used here is the
functional representation of noise suppression parameter "Q," and is given by

/(x) = lx x?:O
0 x< (}

The noise suppression parameter Q is defined by the user and is used to achieve stability. Stability occurs where
there is no reset, i.e., the same winner unit is chosen in the next trial also. Units x; and Q; apply activarion to
Vii which sUppresses rhe components to achieve stability. Hence Q is used here.
In ART 2 network continuous processing of the input units is done. The continuous-valued input signals
s = (sl, ... , s;, .. ,, s11 ) are sent continuously. For each learning trial, one input pauern is presented. At me
beginning of training, rhe activations are set ro zero, i.e., inactive not inhibit. The computation cycle for a
particular learning trial within F1 layers starrs with 11; which is equal to activation ofV; approximated to unit
length. Unit u; is given by
v;
u;=--
<+ II vii
where "e" is a small parameter for preventing the eli vision by zero when II vii becomes zero. Also q; and Xi are
given by
p; . x;=--
w;
q; = '+ llpll' '+ llwll
The noise suppression parameter is applied only to x; and q;.
The signal will be sent from each unit of u; to w; and p;. The activations of units w; and p; have to be
done. The activation of w; is fie sum of input signal received (s;) and au;:
w; = s; +au;

P; is found to receive signals from u; and top-down weights, i.e., sums u; (activation of u;) and top-down
weight (tp), and is given by
Pi= Uj + dtji
 5.6 Adaplive Resonance Theory Network 191

where dis the activation of winning F2 unit. Before entering into V;, activation function is applied to each of
x; and Qj units. Unit V; sums the signals from x; and Qj which receive signal concurrently:

v; = f(x;) + bf(q;)
Activation function is designed to select dte noise.suppression parameter (user specified "Q''). Accord-
ing to Carpenter and Grossberg, the activations of Pi and ~ (i.e., the outputs) will reach equilibrium
(stable set of weights) only after cwo updates of weights. This completes the phase I or one-cycle pro-
cess of F1 layer. Only after F1 units reach equilibrium, the processing of F2 layer starn (i.e., after three
updates).
F2 layer being a competitive layer uses wiMer.rake-all policy to determine its winner. Dot product method
may be used for the selection of the winner. When the top-down and weight vector remain similar, then that
unit is the winner (active). If for a unit, the wp-down and input vectors are not similar, ilien dlat unit becomes
inhibit. This layer receives signals from P; units via bottom-up weights and P; units in rum send signals to F;
unit. Here only the winner unit is allowed to learn the input pattern S;.
The reset mechanism controls the degree of similarity of the input patterns. The checking for reset con-
dition in ART 2 differs from ART 1 network. The reset is checked every time it receives signal from u;
and P;.
In fast learning mode, the updation of weights is continued until the weights reach equilibrium on each
trial. It requires only less number of epochs, but a large number of iterations through the weight update-FJ
portion must be performed on each learning trial. Here, the placement of patterns on dusters stabilizes, bur
the weight will change for each pattern presented.
In slow learning mode, only one iteration of weight updates will be performed on each learning trial.
Large number of learning trials is required for each pattern, but only little computation is done on each trial.
There is no requirement that me patterns should be presented in the same order or that exactly the same
set of patterns is presented on each cycle through them. Thus it is preferable to have slow lt:arning than fast
learning.
Computatiom for algorithm
The following computations have to be performed in several steps of the algorithm and are referred as
"updation ofF 1 activations." Unit) is the winning h unit after competition is completed. If no winning unit
is chosen, then "d" is zero for all units. The calculations for P; and w;, and x; and q; can be done in parallel.
F1 layer consists of six units; the update F1 activations are given by
v
w- '
,- e+\\v\\; P;=ui+dtji
w;
w; = s;+ au;; x;=--
e+ llwll
q;=-p;
'+ lip II'. v; = j(x;) + bj(q;)
The activation function is given by

j(x) = \X j(x) ?: 0
0 f(x) < e

l
192 Unsupervised Learning Networks

5. 6.3.3 Flowchart
The flowchan for the rrainiJJg process of ART 2 network is shown in Figure 5~27. The flowchan dearly
depicts the flow of the training process of the network. The check for reser in the flowchart differs for ART 1
and ART 2 networks.

( Start )
l
Initialize the parameters
a, b, c, d, e, a,p, q

1
specify
No. of epochs of training nep
No. of learning iterations - nit

/
l
For no. of epochs nep ------@

For each input vector -----~

------<= Fori~1ton )------

l
Update activations ot F, unit

I u,= a I ~
L::C...::cJ
I q,= a I
I P,~ I
0

Figure 527 Flowcharr for rraining of ART 2 network.

5.6 Adaptive Resonance Theory Network 193

Update of F, uni\5 activalions again

w,
x~--
:. e+llwll
I w,- s1+au, 1
p,
q, ~ e+li>ll
1 P, ~ u,+ar, 1 lv,~ l(x)+bl(q)ll

,-----
' '--~,--_/
'

tnhibitj,
False Jthat pallern
for will not be
reset clustered

u~-v,_
' e+llvll
P,=u,+dt,

True
k
Figure 527 (continu(d).
 Unsupervised Leaming Networks
194

e:
w,= s,+ au,
x=____::i_
' e+l\wll
p,
q,= e+l\pl\
v,= f(x1) + bf(q)

:--<-.For no. of learning iterations nit/--,

' '
::'' I Update weights for learning unit j
t1 =adu,+{!+ad(d-!)}t,
' bii= adu1+ {l+ad(d-1)} b1

Update F, activations

x=_3_
' e+l\wll
lw1= s,+ au;j
[P, u + dt,)
1

Test for

False

-- -

-- -

@---- ----@

:Q)
F 1
a se
<ftstopping
Test for
condition
lor no. of
epochs

True
QiOD
Figure 527 (continued).

_
 5.6 Adaptive Resonance Theory Network 195

5.6.3.4 Training Algorithm

The training algorithm of ART 2 ne(Work is shown below.

I Step 0: Initi~lize the f~ll~~ing parameters: a, b, c, d, e, a, p, e. Also, specifY the number of epoch~ of I
training (nep) and number of learning iterations (nir).
Step 1: Perform Steps 2-12 (nep) rimes.
Step 2: Perform Steps 3-11 for each input veaor s.
Step 3: Update F1 unit activations:

u; = 0; wi = s;; P; = 0; q; = 0; v; = f(x;);
s;
x;=--
e+ 1\rl\
Update F1 unit activations again:
v
u; = --'-
e+llvll' w,-=s;+.au;;
w; .
P;=u;; x;= e+Uwll'
p;
q; = e+ 1\pll" V; =J(x;) + bf(q;)
In ART 2 networks, norms are calculated as the square root of the sum of the squares of the
respective values.
Step 4: Calculate signals to Fz units:

Jj= Lbijpi
i=l

Step 5: Perform Sreps 6 and 7 when reset is rrue.

Step 6: Find Fz unit Yj with largest signal 0 is defmed such that Jj ~ Jj,j = 1 ro m).
Step 7: Check for reset:

v _'::'.';_:,+_:cP:.'.;c-c
,- __
u- . p r;= -
e+ 'II vii' i = u;+dtj;; e+ 1\ul\ + cilpl\
If llrll < (p -e), then YJ = -I (inhibit J). Reser is true; perform Step 5.
If I\ ell ~ (p -e), rhen

Wj=s;+au;; X; = ___!'!!__;
e+ 1\wl\
p;
q;= e+llpll' v; = J(x;) + bf(q;)

Reset is false. Proceed to Step 8.

Step 8: Perform Steps 9-11 for specified number of learning iterations.
I

_1_
196 UnsupeNised Learning Networks

Step 9: Update the weights for winning unit J:

tp =adu;+ {[l+ad(d-l)]}tp
bu=adu;+{[l+ad(d-I)}Jbu

Step 10: Updare F1 acrivarions:

";
u;=--: w; = Si +au;;
'+ lldl
w;
Pi= u; + dtji:
x;= '+JiwJJ'
P;
q;= '+ Jlpll' "; = f(x;) + bf(q;)

Step 11: Check for the stopping condition of weight updarion.

Step 12: Check for the stopping condition for number of epochs.

In the above algorithm, at resonance period, reset will not occur and new winning unit cannot be chosen.
Since in slow learning number of learning iterations is l, Step 10 in training algorithm need not be processed.
Perform Step 8 until the weight changes are below some specified tolerance. If slow learning is performed,
then repeat Step 1 until the weight changes are below some specified tolerance. If fast learning is adopted,
then repeat Step 1 until the patterns placement on the cluster units do nm change from one epoch to the nexL

5.6.3.5 Sample Values of Parameter

The sample values of the parameters used in ART 2 network and their role in effective uaining process are
mentioned beloW.
11 = number of F1 layer input units
m = number ofh layer duster unirs
a, b = fixed weights present in the F1 layer. The sample values are a= 10 and b = 10; when a= 0
and b = 0, rhe net becomes instable
c = fixed weight for resting of reset. The sample value is c = 0.1. For small c, larger effective range
of the vigilance parameter is achieved
d = activation of winning h unit. Sample value is d = 0.9. The values of c and d should be selected
satisfying the inequality cd/1- d :5 1. The value of cd/1- d should be closer to l, so that
effective vigilance could be achieved
e = a small parameter included to prevent division by zero error when the norm of vector is zero.
8 = noise suppression parameter. A sample value of nqise suppression parameter is 8 = 1/ .jn. The
components of the normalized input vector, which are less than this value, are ser to zero.
a= learning rate parameter. In both slow and fast learning methods, a small value of a slows down
the learning process.
p = vigilance paiameter used in reset condition. Vigilance parameter can range from 0 to 1. For
effectively comrolling the number of clusters, a sample value of 0.7-1 may be allowed. The
range of p may also be affected by the values of c and d.
tj;(O) == inirial top-down weights. The initial weights of this weight vectors are given by -9;(0) = 0.
5.6 Solved Problems 197

bij(O) = initial bonomup weights. These should be chosen ro misfy rhe inequality bij(O) ::= 1/( I -d) jJi.
High values- of b;j allow the net to form more clusters.
On rhe basis of all these roles of parameters and their sample values, care should be taken in selecting their
values for effective training of the network.

I 5.7 Summary
Unsupervised learning networks are widely used when clustering of units is performed. In case of unsupervised
learning nerworks, the information about rhe ourpm is nor known; when the weights of a net remain fixed
for rhe entire opemion then it resuhs in fixed weight competitive nets. Fixed weight competitive nets include
Maxnet, Mexican hat and Hamming net, In case of Hamming net, Maxnet is used as a subnet. The most
importam unsupervised learning nernrork is the Kohonen self-organizing feature map, where clustering is
performed over rhe training vectors and the nernrork uaining is achieved. An extension ofKSOFM. Kohonen
self-organizing mOtor map is also included. An unsupervised learning network with targets known is the
learning vector quantization (LVQ) network. A study is made on LVQ net with its architecture, flowchart for
training process and training algorithm. The variants ofLVQ net are also included. The compression nernrork
discussed in this chapter is the counrerpropagarion network (CPN). The two ()'pes of counrerpropagation
networks- full CPN and forward-only CPN- are discussed. The resting algorithms for these nerworks are also
given. Anmher important unsupervised learning nerwork is rhe adaptive resonance theory (ART) nef'.'lork.
In this chapter, ART I and ART 2 networb wirh all relevant information are discussed in derail.

I 5.8 Solved Problems

l. Construct a Maxne1 with four neurons and = /10.3- 0.2(0.5 + 0.7 + 0.9)]
inhibitory weight c = 0.2, given the initial acti- =/(O..l- 0.421 = /"(-0.12) = ()
vations (input signals) as Follow~:

a]{O) = 0.3: 112(0) = 0.5: 11,1(0) = 0.7: a, II I =/[a,IO)-< L a;-(01]

tlq(O) = 0.9 l:r;

Solurion: Update rhe Jcrivarions For each node, i.e ..

= /}0.5- 0.2(0.3 + 0 7 + 0.9)]
= /10.12) = 0.12

a1(newl = /[a 1(oldl->" L. a,(old)] a,( II =/[a.\(0)-< L"<(O)]

{j:.J
lr#j
The acrivarion function is given by = /}0.7- 0.210.3 + 0.1 + 0.91]
= /10.36) = 0.36
.f(x) =
l
x ifx> 0
0 ifx :=::_ 0

Fim itmllion: a,ll I= /[a.,IOI-FL. a;-101]

'" + 0.5 + 0.7)}
= /10.9- 0.2(0..1
a(II = /[a,(O)-< L. a,(O)]
1
= /(0.9- o..ll = /(0.6) = o.6
k,Pj
l98 Unsupervised Learning Networks

Second iteration: Fourth iteration:

a,(2) = f[a,(l)-< E(l)] n1(4) = f[a 1(3)-< I:n,(3)]

hFj
= /[0 - 0.2(0.12 + 0.36 + 0.6))
= /(0- 0.216) = 0 = /[0- 0.2(0 + 0.1152 + 0.4608)] = 0

a,(2) = f[a,(l)-< I:Ol] a,(4) = {,(3)-< ~a,(3)]

lr'f;j
= /[0. \2- 0.2(0 + 0.36 + 0.6)) = /[0 - 0.2(0 + 0.1152 + 0.4608)] = 0
= /( -0.072) = 0

J(2) = f[a,(l)-< I:(l)]

a,(4) = f[a,(3)-< ~a,(3)]
lr#j
= /[0.1152 - 0.2(0 + 0 + 0.4608))
= /[0.36 - 0.2(0 + 0.12 + 0.6))
= /(0.02304) = 0.02304
= /(0.216) = 0.216

a,(2) =/[a,( I)-< I:(l)]

k#=j
a,(4) = f [Oh ~a,(3)]
= /]0.6- 0.2(0 + 0.12 + 0.36)] = /10.4608- 0.2(0 +0 + 0.1152)]
= /(0.504) = 0.504 = /(0.43776) = 0.43776
Third ileration: Fifth iteration:

a, (3) = f[a,(2)-f L ,,.(2)]

a,(5) = /[ 1 (4)-< I:(4)]
"""-' ki'J
=flO- 0.2(0 + 0.216 + 0.504)]
= /(0- 0.144) = 0 = /[0 - 0.2(0 + 0.02304 + 0.43776)] = 0

a,(3) = /[a 2 (2)-< L !(2)] a,(5) = /[a,(4)-< Ll(4)]

k#j k=h
= /[0- 0.2(0 + 0.216 + 0.504)] = /10- 0.2(0 + 0.02304 + 0.43776)] = 0
= j(O- 0.144) = 0

a3 (3) = /[.\(2)-< L a,(2)] a3 (5) = f[a 3 (4)-< ~a,(4)]

h:.j
= /[0.0234- 0.2(0 + 0 + 0.43776)]
= /[0.216- 0.2(0 + 0 + 0.504)]
= /(0.1152) = 0.1152 = j( -0.0645) = 0

ru,(3) = /[(2)-e I:a1(2)] a,(5) = f [ ru,(4)-e ~a,(4)]

k#j
= /[0.504- 0.2(0 + 0 + 0.216)] = f [0.43776 - 0.2(0 + 0 + 0.02304)]
= /(0.4608) = 0.4608 . = /(0.433152) = 0.433152

L
 5.8 Solved Problems 199

Even though if further iterations afe made, the value

of tLj (5) remains the same, since ali 'the other values 0.5]
=2+[-l - l l - l] -0.5
a, (5), a,(5) and a,(5) are equal to zero. Thus tho -0.5
[
convergence has occurred. -0.5

2. Construct and rw rhe Hamming network to = 2- 0.5 + 0.5 - 0.5 + 0.5 = 2

duster four vectors. Given Ute exemplar vecrors,
Thevaluey;n\ = 2isthenumberofcom~
e(l)=[l-1-l-l]; e(2)=[-l-l-ll] ponents at which the inpmvector Xi and
e(l} agree. Now
rhebipolarinputvecwrsarex 1 = [-1-11-l];
x:z=[-1 -Ill]; XJ=[-1 -1 -I 'rl]; y;,a = b2 + L:x; w;z
x, = [l l - l - 1].

Solution: Number of components in input vector -0.5]

n=4. = 2 + [-l - l l - l) -0. 5
Number of exemplar vecmrs m == 2. [ -0.5
. +0.5
Setting the inirial weights to -112 of the exemplar
vectors, we get = 2 + 0.5 + 0.5 - 0.5 - 0.5 = 2

The value Yin2 = 2 is the number of com~

Wij==(~) ~)) ponents at which the input vector XJ and
e(2) agree.
where Step 3: Initialize the activations of Maxner as

e(l)=[l-l-1-1]: e(2)=[-l-l-ll] y, (0) = 2: y,(O) =2

Step 4: Since Yl (0) = yz(O), Maxner will find
/ Step 0: The weights are given by -- I the unit with the smallest index as rhe
best march exemplar for input XJ =
0.5 -0.5] [-'- 1 -1 1 -1] or in some cases both
-0.5 -0.5
Wij = -0.5 -0.5 j may be chosen as best match exemplars. /
[
-0.5 0.5
Second inpm vector.
Sening the bias to n/2, we obtain

n 4
ISre~: -;or x2 = [-~ --1 1~], ~erf~rm ~rep~ J

I b,=b,=-=-=2
- 2 2 I 2-4.
Step 2:
First input vector.
Yi11\ = b1 + 'L: x; w; 1
/ Stepl: For;q = [-1 -1 1 -1], perform l
Steps 2-4.
+0.5]
Stop 2' = 2+[-1 - I I I] =~:;
[ -0.5
]in\ = h1 + Lx;w;1
= 2- 0.5+ 0.5-0.5- 0.5= l

L
200 Unsupervised Learning Networks

m2:::::: b2 + Lx;Wi2 Fourth input vector:

I Step I: For X<l = [.I 1 - I - I], perform Steps j

-0.5] 2-4 .
= 2+ [-I - I I 1\ =~:; Step 2:
[
0.5
= (2 + 0.5-0.5 + 0.1 + 0.5) = 3 Yinl = bt + Lx;wn

I
Step 3: lnitiali7.e the activations of Mv:ner as
-00.5-J
5
y,(OI = 1: _,101 = 3 = 2+ \I I - I - I] _ :
05
Step 4: Since )'2(0) > .l'l (0), Max net will find
L-o.5
that the unit )'2 has rhe best march exem- = 2 + 0.5 - 0.5 + o.s + 0.5 = 3
1 plar for input vector X! = \-1 - I 1 \[. \ }i11l = b2 + Lx;w;2

Third input veclor. -0.5]

I s-:p ~For.\]=~~ -1 -I ~.p~Or~St~ps! =2+[11-1-1] =~;
[
2-4. 0.5
Step 2: = 2-0.5-0.5 + 0.5-0.5 =I

.Yml = b1 + L x, ll'il Step 3: Initialize rhe activations of Maxner as

Yl (0) = 3: ,(O) ~ '

= 2 + I -I - I - I I I -o::
11.1]
-05
[ -0.)
Step 4: Since ]I (0) > J2(0}, Max net will find
rhar rhe uniry1 is rhe b~sr march exemplar
J for the input vecror :q = \I I - I - 1]. 1
= 2-0.1 + 0.'5 + 0.5-0.5 = 2
The architecture for the Hamming net tOr this
.Yin2 = b2 + L x; 1Vi2 problem is given by Figure I.

-0.5]
-0.5
=2+1-1-1-11\
-0.5
[
0.5
= 2 + 0.5 + 0.5 + 0.5 + 0.5 = 4

Step 3: Initialize rhe acrivarions ofMaxner as

J'l (0) = 2; y,(O) = 4

j
Step 4: Sincey!(O) > yl(O). Maxnerwill find rhe
unit .Yl a.~ the be.~r march exemplar for 3.
f 03 figure 1 Hamming net architecture.

nsrru~r a K~honen self-orgamzmg map to dus-

.
/ inpurvecwrxJ:::::\-l-1-11]. /' rerthe~ourglvenvectors,[OOll],\1000],
- \0 I I O]and(OOO I].Thenumberofdustersto

I
L
 5.8 Solved Problems 201

L'
be formed is two. Assume an initial learning rate
2
ofO.S. D(2) = \W,1 - x;)
i=l
Solution: The number of input vectors is four and
= (0.9- o)' +
(0.7- o)'
number of dusters tO be formed is two. Thus, n =
4 and m ::::. 2. The architecture of the Kohonen + (0.5 - 1) + (0.3 - 1)2
2
self-organizing fearure map is given by Figure 2. = 0.81 + 0.49 + 0.25 + 0.49
= 2.04
y, Y,
Step 3: Since D(1) < D(2), therefore D(l) is
minimum. Hence the winning duster
unitisYJ,i.e.,J:::. l.
Step 4: Update the weights on the winning
duster unit J = 1.

X, X, ~ = wq(old)+ a[x;- wq(old)]

W;J(new)::::. Wit (old)+ 0.5 [x;- w;t~?ld)].
x, X, x, W\1 (n) = w11 (0) + 0.5 [XJ - W\1 (Qjj
"'
Figure 2 Architecture ofKSOFM.
= 0.2 + 0.5(0 - 0.2) = 0.1
W:ZJ(n) = w;z,(O) + 0.5 [xz- W:ZI(O)]
= 0.4 + 0.5(0- 0.4) = 0.2
r Step 0: Initialize the weights randomly between
0 and 1.
I W3J (n) = w, 1(0) + 0.5 [x, - IUJl (0)]

= 0.6 + 0.5(1 - 0.6) = 0.8

w41(n) = w41(0) +0.5[x,- W4J(O)]
0.2 0.9]
_ 0.4 0. 7 ; R = 0; a(O) = 0.5 = 0.8 + 0.5(1- 0.8) = 0.9
Wi]- 0.6 0.5
[
o.8 o.3 ,,, I The updated weight matrix after presen-
I tation of first input pattern is

Fim iuput vector: 0.1 0.9]

0.2 0.7
I Step 1: I
U);::::
For x = [0 0 I 1}, perform Steps 2-4. I) 0.8 0.5
[
Step 2: Calculate the Eudid~n distance: 0.9 0.3

D(j)'= L (IVij- x 1)
2
Seco11d input vector.

,. St~ l.:for x ~ [1 0 0 0], perform Steps 2-4.

D(1) = L' (wil - x;)
2
Step 2: Calculate the Euclidean distance:
/

i=l
= (0.2 - o)' +(0.4 - o)' D(j) = L (wij- x;)'
+ (0.6- 1) 2 + (0.8- 1) 2
=
=0.4
0.04 + 0.16 + 0.16 + 0.04
D(1) = L' (w;,- x;)'
i=l

L
 Unsupervised Learning Networks
202
2 Step 2: Calculate the Euclidean disca.nce:
=(D. I - !)2 +(D.2-D)
+ (D.8 ~ D) 2 + (D.9 - D)
2
D(j) = L (wij- x;)2
= D.81 + D.D4 + D.64 + D.81
=2.3
D(J) = L' (wi! - x;) 2

D(2) = L' (w,-,- x;) 2 Fl

=(D. I- D) 2 + (D.2- 1)
2
Fl
+ (D.8 - 1) 2 + (D.9 - Dl'
=WS-Jf+WJ-~
=D.Dl +D.64+D.D4+D.81 = !.5
+~5-~+W3-~
=MI+W~~+~
=~
D(2) = L' (w,-, - x;) 2

j=o:\

St<p 3, Since D(2) < D(l), therefore D(2) is

2
= (D.95 - Dl + (D35 - o'
2
minimum. Hence the winning cluster + (D.25 -1)2 + (D.J5- D)
unit is Yz, i.e.,]== 2. = (D.9D25) + (D.4225) + (D.5625)
Step 4: Update the weights on the winning + (D.D225) = !.91
duster unit]== 2:
Step 3: Since D(l) < D(2), therefore D(l) is
w;j(new) == Wij(old)+ a[.r;- wq(old)] minimum. Hence rhe winning cluster
tv;2(new) == Wi2(old) + 0.5 [x;- wo. (old)] unit is YJ,i.e . ,]== l.

wn{n) = w,(D) + D.5 [xi - w,(D)] Step 4: Update the weights on the winning dus~
rer unit]== l:
= D.9 + D.5(1 - D.9) = D.95
w22(n) = W:!2(D) + D.5 ["1- w,,(D)] w,J(new) = WI}( old)+ a[x; - w;j{old)]
= D.7 + D.5(D- D.7) = D.35 !ViJ (new) == w;1 + 0.5 [x; - w; 1(old))
(old)
w32(n) = W,2(D) + D.5 [x,- w,,(D)] WJJ (11) = IVJ\ (0) + 0.5 [XL - WJJ(Q))
= D.5 + D.5(D - D.5) = D.25 =D. I+ D.5(D- D. I)= D.D5
w.n(n) = w42(0} + 0.5 [x<~- Ul42(0)1 W2J(n) = W2J(D) +D.5h- w, 1 (D)]
= D.3 + D.5(D- D.3) = D.l5 = D.2 + D.5(1 - D.2) = D.6
WJI (n) = W,i(D) + D.5 (XJ- WJJ(D)]
The updated weight matrix after presen~
= D.8 + D.5(1 - D.8) = D.9
ration of second input panern is
W41 (n) = W<J(D) + D.5 h- W<J(D)]
= D.9 + D.5(D - D.9) = D.45
D.! D.95]
D.2 D35
w-- The weight update after presemation of
'1- D.8 D.25
[ third input pattern is
I D.9 D.t5 I
D.D5 D.95]
D.6 D35
Third input vector: Wij:: 0.9 0.25
[
I D.45 D.l5 I
\ Step 1: For x = [0 1 l 0], perform Steps 2-4.. I
 5.8 Solved Problems 203

Fourth input vector: The final weight obtained after the pre~
semation of fourth input pattern is
lsrep 1: For x == (o 0 0 1], perform Steps 2-4. I
D.D25 D.95]
Step 2: Compute the Euclidean distance: D.3 D.35
Wij = 0.45 0.25
[
D(j) = L' (wr x;) 2
D.475 D.J5

i=l Since all the four given input panerns are

presented, this is end of first iteration or
D(l) = L' {w;I - x;)' l~epoch. Now rhe learmng rate can be
i::l 1fpdared as
= (D.D5 - D) 2 + (D.6 - D) 2
a(t+ I)= D.5a(t)
+ (D.9 - D) 2 + (D.45 - !) 2 a(!)= D.5a(D) = D.5 x D.5 = D.25
= D.DD25 + D.36 + D.8! + D.3D25
Wirh this learning rate yotn:anpfo~
= 1.475
ceed further up to 100 iterations or till

L' (w,o - x;)

radius becomes zero or the weight matrix
2
D(2) = reduces to a very negligible value. The
i=l net wirh updared weights is shown by
= (D.95- D)'+ (D.35 - D) 2 I Rpre~ I
+ (D.25- D) 2 + (D.I5- !) 2
= (D.9D25) + (D.I225) + (D.D625)
+ (D.7225)
= 1.81

Step3: Since D{l)<D(2), therefore D{l) is

minimum. Hence the winning duster
unit is Y1, i.e.,}== I. X,
Step 4: Update the weights on rhe winning dus-
ter unit]== I: Figure 3 Net for problem 3.

f'for a given Kohonen self~organizing feature map

w,J(new) = Wij(old)+ a[x;- wij(old)]
With weights shown in Figure4: (a) Use the square
W;J(new) == w;1 (old)+ 0.5 [x;- w;1 (old)] of rhe Euclidean disrancc to find the duster unit
wn (n) = wn (D)+ D.5 [xt - wn (D)] l] closest to rhe input vector (0.2, 0.4). Using a
= D.D5 + D.5(D - D.D5) = D.D25 learning rate of0.2, find..th~J:l_ew.~@!Js for uniL
lj.'1f)ffor the input vector (0.6, 0.6) with learn~
'"21 (n) = W21 (D)+ D. 5 ["1 - W21 (D)]
~e 0.1, find the winning duster unit and irs
= D.6 + D.5(D- D.6) = D.3 new weights.
WJI (n) = WJl (D)+ D.5 [x, - "'31 (D)]
Solution: (a) For the inpurvector (0.2, 0:4) == (xl, XJ.)
= D.9 + D.5(D - D.9) = D.45 and a= 0.2, the weight vector Wis giVe~'by
W41 (n) = W4J (D)+ D.5 [X".] - W4J (D)]
...
= D.45 + D.5(1 - D.95) = D.475
w= [D.3 D.2 D. I D.8 D.4]
D.5 D.6 D.7 D.9 D.2
...,

j
 Unsupervised Leaming Networks
204

Fori=1to2,
Wil (n) = IV! 1 (0)+ a[x, -
Wi! (0)]
= 0.3 + 0.2(0.2 - 0.3) = 0.28
W21 (n) = W2! (0)+ a[X2 - '"'' (0)]
= 0.5 + 0.2(0.4 - 0.5) = 0.48
The updated weight matrix is given by

x,
w= [0.28 0.2 0.1 0.8 0.4]
0.48 0.6 0.7 0.9 0.2
Figure 4 KSOFM net for problem 4. For rhe input vecror Cx1 ,X2) = (0.6, 0.6) and
a= 0.1, rhe weight matrix is initialized from
Now we find the winner unit using square of Figure 4 as
Euclidean distance, i.e.,
w= [0.3 0.2 0.1 0.8 0.4]
2
0.5 0.6 0.7 0.9 0.2
D(j) = L (wq- x,.)z = (wlj- X\)2 + (W2.f- xz)z Now we find the winner unit using square of
i=:\ Eudidean distance, i.e.,
2
Forj== 1 to5
D(j) = L (wij- x1) 2
= (w 1f- x1 )
2
+ (11Jlj- xz) 1
2 i:=l
D(l) = (0.3- 0.2) 2 + (0.5- 0.4)
= 0.01 + 0.01 = 0.02 Forj=lto5
2
D(2) = (0.2- 0.2) 2 + (0.6 - 0.4)
2 D(l) = (0.3- 0.6) 2 + (0.5- 0.6)
= 0 + 0.4 = 0.04 = 0.09 + 0.01 = 0.1
D(2) = (0.2- 0.6) 2 + (0.6- 0.6)
2
2
D(3) = (0.1 - 0.2) 2 + (0.7- 0.4)
= 0.01 + 0.09 = 0.01 0 \ = 0.08 + 0 = 0.08
D(3) = (0.1 - 0.6)2 + (0.7- 0.6)
2
2 2
D(4) = (0.8 - 0.2) + (0.9 - 0.4)
= 0.36 + 0.25 = 0.61 = 0.25 + 0.01 = 0.26
2
D(5) = (0.4 - 0.2) 2 + (0.2- 0.4)
2 D(4) = (0.8- 0.6) 2 + (0.9- 0.6)
= 0.04 + 0.04 = 0.08 = 0.04 + 0.09 = 0.13
2
D(5) = (0.4- 0.6)2 + (0.2 - 0.6)
Since D(1) == 0.2 is rhe minimum value, the winner = 0.04 + 0.16 = 0.2
unit is J = l. We now update the weights on rhe
winner unit}= I. The weighr updation formula is Since D(2) = 0.08 is rhe minimum value, the winner
unit is]= 2. We now update the weights on the win-
givtn by
ner unit with a= 0.1. The weight updation formula
Wjjlnew) = W;j(old)+ a[x;- W;j(Old)} is given by
wu(new):;:::: wu(old)+a[x;- wu(old)1
Subsrimring] = I in the equation above, we obtain Substituting}= 2 in the equation above, we obtain

w; 1(new) = w; 1(old)+ a[x; - w; 1(old)) w;z(new) = Wi2(old)+a[x;- w;2 (old)1

5.8 Solved Problems 205

Fori=lto2, Now we find the winner unit using square of

Euclidean distance, i.e., .
wdn) = wn(O)+a[x,- W!j(O)]
= 0.2 + 0.1(0.6- 0.2) = 0.24 D(j) = L;<w;- x,) 2

W22(n) = W22(0)+a[X2- W22(0)]

Fori= 1 to 5 and j = 1 to 2,
= 0.6 + 0.1(0.6- 0.6) = 0.6
D(l) =(I- 0) 2 + (0.9- 0.5) 2 + (0.7- 1) 2
The new weight mauix is given by
+ co.s - o.5) 2 + (0.3 - o) 2
w= [0.3 0.24 0.1 0.8 0.4] ' = I+ 0.16 + 0.09 + 0 + 0.09 = 1.34
0.5 0.6 0.7 0.9 0.2
D(2) = (0.3- 0)2 + (0.5- 0.5) 2 + (0.7- 1) 2
+ (0.9 - 0.5)2 + (I - o)2
5. Consider a Kohonen self-organizing net with two ~
Ater units and five input units. The weigh!J' = 0.09 + 0 + 0.09 + 0.16 + I = 1.34
' v~ctors for the duster units are given by . 1>'\ .A we can see, in this caseD(1) = D(2), so the winner
(Y; // unit is the one wiili the smallest judex. Thus, winner
WI = [1.0 0.9 0.7 0.5 0.31 1 unir is yl> i.e.,]= 1. We now updat~ the weights on
W2 = (0.3 0.5 0.7 0.9 1.01 the winner unit with a= 0.25. The weight updation
Use the square of the Euclidean distance ro find formula is given by
ilie winning duster unit for the input pattern wu(new) = w,y(old)+a[x,- wu(old)]
x =[0.0 0.5 1.0 0.5 0.0]. Using a learning rate of
Substituting]= l in the equation above, we obtain
0.25, find the new weights for the winning unit.
Wil (new) = Wil (old)+ a[x; - Wil (old)}
Fori= 1 toS,
w11 (n) = w11 (O)+a [x1 - w11 (0)]
=I+ 0.25(0- I)= 0.75
w21(n) = W2!(0)+a [x2- W21(0)]
= 0.9 + 0.25(0.5 - 0.9) = 0.8
x, w31 (n) = w31 (0)+ a [X3 - w31 (O)]
= 0.7 + 0.25(1 - 0.7) = 0.775
x, x, x, x,
"'
Figure 5 KSOFM ner.
W4! (n) = W4! (0)+ a [X4 - W41 (0)]
= 0.5 + 0.25(0.5 - 0.5) = 0.5
w51 (n) = w51 (0)+ a [xs - w51 (0)]
Solution: The net can be formed as shown in Figure 5.
For the input vector x :::: [0.0 0.5 1.0 0.5 0.0] and = 0.3 + 0.25(0 - 0.3) = 0.225
the learning rate a= 0.25, the weight vecror W is The updated weight matrix for the winning unit is
given by given by

1.0 0.3] 0.75 0.3]

0.9 0.5 0.8 0.5
w = 0.7 0.7 W= 0.775 0.7
[ 0.5 0.9 [ 0.5 0.9
0.3 1.0 0.225 1.0
 '
206 Unsupervised Learning Networks

6. Construct and test an LVQ net with five vectors For j == l to 2,

/Signed to rwo classes. The given vectors along 2 2 2
v with rhe chsses are" shown in Table 1. D(l) = (0- 0) + (0- 0) + (1- 0)
+ (1- 1) 2 = 1
r: "CJ/)~ ~:
Vector Class D(2) = (1 - 0) 2 + (0 - 0) 2 + (0- 0) 2

Mi,
., . )\tj r,)-n (O o l 1] + (0- 1)2 = 2
'J" ' ,. [1 0 0 0] 2
Since D(J) < D(2),D(1) is minimum#~jje
I [Q 0 0 lJ 2
winner unit index is J == 1. Now tha T , e
' [1 1 0 0] 1
weight updation is performed as -----
[0 I 1 0]
Wj(new) = Wj(o1d)-a[x- WJS!Mll
Solution: In the given five vecmrs, first ~tors are
used ~rial weight vectors and the remaining-three w11(n) =WJJ(O)-a[xl -w\1(0)]
vectors are used as input vectors. Based on this, LVQ = 0- 0.1(0- 0) = 0
net is shown in Figure'Ta'long with initi~ weights. W:Zl (n) = W:Zl (0)- a[X2 - w:z 1 (0)]

Y, Y, = 0- 0.1(0- 0) = 0
(
>l WJJ(n) = WJJ(0)-a['3 -1"31(0)]
=1-0.1(0-1)=1.1
' . W<J(n) = W<J(O)-a[x<- W41(0)]
= 1- 0.1(1 - 1) = 1

After the presentation of fmt input panern, the

weight matrix becomes

W= [~1 ~]
'
,, x, ' Second iuput vector
Figure 6 LVQ ner. For [ 1 1 0 0] with T:::: 1, calculate the square of the
Euclidean distance, i.e.,
Initialize the reference weight vectors as
4
WJ =[0 0 1 1]; w:z=[l.Q..O 0] D(j) = L (wij- x;) 2
Ler rhelearning rare be a= 0.~ ;A) ~~ ff ! i==l

Firrt input vector L! D J Forj== 1 to 2,

2
D(1) = (o- 1) 2 + (o -1) 2 + (1.1- o)
For (0 0 0 1] wiili T == 2, calculate the square of the
Euclidean distance, i.e., +(1-0) 2 = 4.21

'I
--- = L(w,y-
-r- \ 2
D(2) = (1-1) 2 +(O -1) 2 + (o -o)
2
! D(j) x,) \ + (0- 0) 2 = 1
I _':'!---_...)
5.8 Solved Problems 207

Since D(2) < D(l),D(2) is mimmum; hence the W:Zl (n) = W:Zl (0)+ a[., - W:Zl (0)]
winner unit index is 1 = 2. Again since T =fo 1. the =0+0.1(1-0)=0.1
weight updation is performed as
'"31 (n) = WJJ (0)+ a['3 - w31 (0)]
WJ(new) = wj(old)- a[x- wj(old)] = 1.1 + 0.1(1-1.1) = 1.09
wn(n) = w\2(0)- a[x1 - w\2(0)] W4J (n) = W4\ (0)+a[X4 - W<J (0)]
= 1- 0.1(1- 1) = 1 = 1 + 0.1(0- 1) = 0.9
w:z,(n) = W:Z2(0)- a[X2- W:Z2(0)]
= 0- 0.1(1- 0) = -0.1 Afr:er the presentation of third input panem, the
weight matrix becomes
w32(n) = '"32(0)- a['3- IU32(B)]

[~-1 -~ 1 ]
= 0-0.1(0- 0) = 0
w42(n) = w42(0)- a[X4- w<l(O)] w= 1.09 0
=0-0.1(0-0)=0
0.9 0
After the presentation of second input pattern, ilie
weight matrix becomes Thus the first epoch of ilie training has been com~
pleted. It is noted that if correcr class is obtained for

-~1]
0 fim and second input patterns, fun:her epochs can be
w- o performed until all the winner units become equal to
[ :1 all the classes, i.e., all T ==].

. . 7.j.'.opsider an LVQ ncr with two input units and

Thtrd mput vector c/ fo~r target classes: q > l1 CJ and q. There exist

For [0 1 l 01 with T = l. calculate ilie square of the 16 classification u~irs, with weight ve~mrs indi~
Euclidean distance as cared by the coordinates on the foHowmg chart~
read in row-column order. For example, the unit
4
with weightvecror (0.2, 0.2), (0.2,.0.6) is assigned
D(j) = L(w,y-x,J' to represem class 1 and th'e ch.ssification units for
i=l
class 2 have initial weight vecrors of {0.4, 0.2),
Forj==1ro2, (0.4, 0.6), (0.8, 0.4) and (0.8, 0.8). The chan is
given in Table 2.
D(l) = (0- 0) 2 + (0- 1) 2 + (1.1- 1) 2
Table2
+(1-0) 2 =2.01
X2
D(2) = (1 - 0) 2 + (-0.1- 1) 2 + (0- 1) 2 -
1.0
+ (0 - 0) 2 = 3.21
Since D(l) < D(2),D(l) is minimum; hence the
winner unit index is 1 == 1. Now that T = ], the
0.8 "
0.6 C[
0.4 "
0.2 C[ '2
CJ
q
,,
"q
CJ
CJ ",,
"
n
weight updarion is performed as

Wj(new) = Wj(old)+ a[x- Wj(old)]

0.0 "
0.0 0.2 0.4 0.6 0.8 1.0 Xi
Updating ilie weights on the winner unit, we obtain

w 11 (n) = w11 (0)+ a[x1 - wu (0)]

= 0+0.1(0- 0) = 0

-
 Unsupervised Learning Networks
208

of r.x = 0.25, show which classification unit Class 3.-

moves where (i.e., determine its new weight Initial weight vector
vector). .
0.2 0.2 0.6 0.6]
Given the input vector of (0.4, 0.35) repre- w, = [ 0.4 0.8 0.6 0.2
senting class 1, using initial weight vector and
learning rate of a= 0.25, note what happens? with target t = 3.
Given the input vecror of (0.4, 0.45), deter-
mine the performance of the net. The input Class4.-
vector represents class 1. Initial weight vector
Solution: The LVQ net for this problem with two 0.4 0.4 0.8 0.8]
input units and four cluster units is shown in Figure 7. w, = [ 0.4 0.8 0.6 0.2
The initial weight vectors for the respective classes are
shown below. with target t = 4.
For the given input vector (u 1 , uz) = (0.25, 0.25)
=
with a= 0.25 and t l, we calculate the square
of the Euclidean distance using the formula
2
D(j) = L (wij- xi = (WJj- XJ )
2
+ (Wlj- X2) 2
i=l

Forj=lto4,
2
D(i) = (0.2- 0.25) 2 + (0.2- 0.25) = 0.005

D(2) = (0.2- 0.25) 2 + (0.6- 0.25) 2 = 0.125

i 2
D(3) = (0.6- 0.25) 2 + (0.8 - 0.25) = 0.425
D(4) = (0.6- 0.25) 2 + (0.4- 0.25) = 0.145
2
"' "'
Figure 7 LVQ net (with two input units, A D{l) is minimum, therefore the winner unit
four cluster units). index is)= 1. Now we update the weights on rhe

Class 1.-
=
winner unit, since t 1 = l, a== 0.25, using the
weight updarion formula
Initial weight vector
Wj(new) = WJ(oid)+a[x- WJ(oid)]

WI= [0.2 0.2 0.6 0.6] Updating the weights on the winner unit, we
0.2 0.6 0.8 0.4 obtain
with target t = l. WJJ (new) = wu (0)+ a[x1 - w 11 (old)]
= 0.2 + 0.25(0.25- 0.2) = 0.2125
Class 2.-
Wll (new) = Ui21 (0)+ a[X"2 - "'21 (old)]
Initial weight veaor = 0.2 + 0.25(0.25 - 0.2) = 0.2125

w2 = (0.4 0.4 0.8 0.8] Therefore, the new weight vecror is

0.2 0.6 0.8 0.4
WI = [0.2125 0.2 0.6 0.6]
with target t = 2. 0.2125 0.6 0.8 0.4

l
5.8 Solved Problems 209

For the given input vector (uJ, u2) = (0.4, 0.35) k D(4) is minimum, therefore in this case also
with a= 0.25 and t = l, we calculate the square the winner unit index is 1 = 4. Since t f:. 1. the
of the Euclidean distance using the formula weight updacion formula used is
2 . . wj(new) = wj(old)- a[x- Wj(oid)]
D(j) = "L., (wij- x;) 2 = (wlj- XJ)2 + (Wzj- Xz)2 .-
i=l Updating the weights on the winner unit, we obtain
Forj= 1 to4, w 14(new) = w 1,(0)-a[xl- W14(old)]
D(1) = (0.2- 0.4) 2 + (0.2- 0.35) 2 = 0.0625 = 0.6 - 0.25(0.4- 0.6) = 0.65
D(2) = (0.2- 0.4) 2 + (0.6- 0.35)2 ~ 0.1025 W,<(new) = W,4(0)- a[....,- Ul24(old)]

D(3) = (0.6 - 0.4)2 + (0.8- 0.35) 2 = 0.2425 = 0.4- 0.25(0.45 - 0.4) = 0.3875

D(4) = (0.6- 0.4) 2 + (0.4- 0.35) 2 = 0.0425 Therefore, the new weight vector is

k D(4) is minimum, therefore rhe winner unit W = [0.2 0.2 0 6 0.65 ]

in~ex is~ = 4. Thus, f~unh unit is the winner ~
1
0.2 0.6, 0.8 0.3875
umt that Is closest ro the mputvector. Smce t #=1, y -
the weight up dation formula used is 8. Cons1"der me
_L c II
ro full CPN shown m
owmg
WJ(new) = WJ(old)- a[x- w1(old)] Figure 8. Using rhe input pair x =[I 0 0 0] and
y = [1 OJ, perform the phase I of training (one
Updating the weights on the winner unit, we step only). Find the activation of the cluster layer
obtain units and update the weights using learning rates
w 14(new) = w14(0)- a[x1 - w 1,(old)] a =fi = 0.2.
= 0.6 - 0.25(0.4 - 0.6) = 0.65
w,4(new) = w,,(O)- a[x,- w,,(old)]
= 0.4- 0.25(0.35- 0.4) = 0.4125

Therefore, the new weight vector is

w [0.2 0.2 0.6 0.65 ]

I = 0.2 0.6 0.8 0.4125

For the given input vector (u 1 , uz) = (0.4, 0.45)

wirh a:= 0.25 and t = I, we calculate the square
of the Euclidean distance using the formula
Figure 8 lnstar model of CPN net.
2
D(j) = L (wij- x;)2 = (wlj- xd2 + (WJ.j- X2)2 Solution: The input pair isx = [1 0 0 0] andy= [1 Ol
i=:l and the learning rates are CY:..: 0.2 and fJ = 0.2.
Forj= 1 to4,
Phase I of trni11ing: The initial weights are obtained
D(1) = (0.2- 0.4) 2 + (0.2- 0.45) 2 = 0.1025 from Figure 8 as
D(2) = (0.2 - 0.4) + (0.6- 0.45) 2 =
2
0.0625
0.6 0.4]
D(3) = (0.6- 0.4) 2
+ (0.8- 0.45) 2
= 0.1625 y = 0.6 0.4 and W = [0.5 0.5]
0.4 0.6 0.5 0.5
D(4) = (0.6- 0.4) 2 + (0.4- 0.45) 2 = 0.0425 [
0.4 0.6
 Unsupervised Learning Networks
210

Now we calculate the square of the Euclidean distance v31(new) = v, 1(0)+a["3- v31 (old)]
w;ing the formula = 0.4 + 0.2(0 - 0.4) = 0.32
V<J (new) = '" (O)+ a[x4 - v41(old)]
' 2
D(j) = L (x;- Vij)
2
+L (y,- w~)
2
= 0.4 + 0.2(0 - 0.4) = 0.32
i=l k=l
The weight updarion between the y-in put and clusrer
Forj=lto2, layer is performed as shown below:

' 2 w,;(new) = w;;(old)+fil.rk- w,;(old)]

D(l) = L (x;- v;!l + L (yk-
2
WH)
2

i=l k=l Fork= 1 to 2 and]= l, we obrai~

= (xJ - '")' + ("Z- ,,)' + ("3- v,,)'
2 W1J (new) = Wll (O)+fi iy 1 - W1J (old)]
(x<- V<J) + (y, - w11l + (y,- W21l
2 2
+
= 0.5 + 0.2(1 - 0.5) = 0.6
2
= (I - 0.6) 2 + (0- 0.6)2 + (0 - 0.4)
""'(new) = W,! (O)+fi (1'2 - W21 (old)]
+ (0- 0.4) 2 + (I - 0.5)
2
+ (0- 0.5) 2 = 0.5 + 0.2(0- 0.5) = 0.4
= 0.16 + 0.36 + 0.16 + 0.16 + 0.25 + 0.25
Thus the updated weigh[S are
D(1) = 1.34
4 2
D(2) = L (x;- v,;l + L (yk- w,)
2 2
V =
0.68
0.48
0.4]
0.4 and W = [0.6 0.5]
i=l k=l 0.32 0.6 0.4 0.5
[
= (x, - v,) 2 + (, - vu) 2 + ("3 - ,,)' 0.32 0.6
2 2
+ (x4 - vd 2 + (y, - w12l + (y, - um) 9. Consider the CPN net shown in Figure 9. Using
= (I - 0.4) 2 + (0 - 0.4) 2 + (0- 0.6)2 the inpm pair x = [0 I 1 01 andy = [0 1), per-
2 form phase I of uaining (one step only). Find the
+ (0- 0.6) 2 + (1 - 0.5) 2 + (0 - 0.5)
activation of the cluHer layer units and update the
= 0.36 + 0.16 + 0.36 + 0.36 + 0.25 + 0.25 weights using learning rates ct ":=: 0.2 and {3 = 0.3.
D(2) = 1.74

Since, D(l) < D{2), therefore rhe winner unit index

is]= l. We now update the weights on the winner
unit.

Weight updatio11: The weight updation between the

x-input and duster layer is performed as shown below:

vu(new) = vu(old)+ ct[x; - v;j(Old)]

Fori= 1 to 4 and]= l, we obtain

V1J (new) = '11 (0)+ a[x1 - '"(old)]

= 0.6 + 0.2(1 - 0.6) = 0.68 Figure 9 Ins tar model of CPN net.
,,(new)= ,,(U)+a["Z- ,,(old)]
Solution: The inpur pair is x = [0 I 1 O],y = [0 1]
= 0.6 + 0.2(0- O.b) = 0.48 and learning rates are ct = 0.2 and {3 = 0.3.
 5.8 Solved Problems 211

Phase I oftraining: The initial weighrs obtained from ,,(new)= '21(0)+a["Z- ,,(old)]
Figure 9 are = 0.7 + 0.2(1- 0.7) = 0.76
'31 (new) = '31 (0)+ a[x:J - '31 (old)]
UQ5]
V= UQ5 Md W= [~Q2] = 0.5 + 0.2(1 - 0.5) = 0.60
~u o2~ '41 (new) = V<J (0)+ a[x4 - '"(old)]
[
Q5U = 0.5 + 0.2(0 - 0.5) = 0.40
Now we calculate the square of the Euclidean distance
The weight updacion between the Y-in put and cluster
using the formula
layer is performed as shown below:

D(j) = L (x;-
'
Vij)
2
+L
2

,.,, (y,- Wlj)

2 w;;(new) = w;;(old)+fil.rk- w,;(old)]
i=l
Fork= 1 to 2 and]= 1, we obtain
Forj= 1 w2,

' 2 W!!(new) = W!!(O)+fi(rl- w 11 (old)]

D(l) = L (x;- Vi!)
2
+L (y;- w.,) 2 = 0.2 + 0.3(0- 0.2) = 0.14
i=l k=J
""'(new) = ""' (O)+fi(n - W21 (old)]
= (0- 0.7) 2 +(I - 0.7) 2 + (1 - 0.5) 2
= 0.2 + 0.3(1 - 0.2) = 0.44
+ (0- 0.5) 2 + (0- 0.2) 2 +(I - 0.2) 2
= 0.49 + 0.09 + 0.25 + 0.25 + 0.04 + 0.64 Thus the updated weights are
D(l) = 1.76
0.56 0.5]
D(2) = L (x; -
'
va)
2
+ L (y,- wd 2
2 V=
[
0.76
0.6
0.5
0.7
and W= [0.14 0.2]
0.44 0.2
i=l k=l 0.4 0.7
= (0- 0.5) 2 +(I - 0.5) 2 + (! - 0.7) 2
10. Consider the forward-only CPN net shown in
+ (0 - 0.7) 2 + (0- 0.2) 2 + (I - 0.2) 2
Figure 10. Using the input pair x = [1 0 0 0]
= 0.25 + 0.25 + 0.09 + 0.49 + 0.04 + 0.64 andy= [1 0], perform phases I and II of train-
D(2) = 1.76 ing and update the weights using learning rates
ct =a= 0.2.
ln this case, D(1) = D(2) = 1.76, i.e., both the dis-
tances are equal. Hence theunirwith thes:mailesriudex
is chosen as the winner and weights are updated, i.e.,
we rake]= 1 and update the weights on this winner
unit.

Weight updation: The weight updatiOn between the

x-input and cluster layer is performed as shown below:

vu(new) = vu(old)+ a[x;- v;;(old)]

,~
Fori= 1 ro4and]= !,we obtain
x-lnput
""(new) = '" (O)+ a[x1 - '"(old)] layer

= 0.7 + 0.2(0 - 0.7) = 0.56 Figure 10 Forward-only CPN network.

l
212 Unsupervised Learning Networks
_i
Solution: The given input pair is x = [1 0 0 0] v,,(new) = .,,(O)+o[x,- v,.(old)) '
andy:::= [1 0] with learning rates a= a= 0.2. = 0.2 + 0.2(0- 0.2) = 0.16
The inicial weights obtained from Figure 10 are
v41(new) = V4l(O)+a[X4- V41(old)]
= 0.2 + 0.2(0- 0.2) = 0.16
0.8 0.2]
V= 0.8 0.2 and W = [O.S 0.51 The updated weight matrix is,
[
0.2 0.8 0.5 o.s
0.2 0.8
0.84 0.2]
Phase I of training: We calculate the Euclidean dis- V= 0.64 0.2
0.16 0.8
tance using the formula [
0.16 0.8
4
D(j) = L (x; - vij)
2 Phase II of training: We ca1culate the Eudidean
distance using the formula
i=-1
4
Forj= l to 2, 2
D(j) = L(x;- v;)
4 i=l
D(J) = L (x;- Vii)
2
Forj= I ro2,
i=l

= (I - 0.8) 2 + (0 - 0.8) 2 + (0- 0.2) 2

D(l) = L' (x; - v;.J 2

+ (0- 0.2) 2 i=l

= 0.04 + 0.64 + 0.04 + 0.04 = (1- 0.84) 2 + (0- 0.64) 2 + (0- 0.16) 2
D(J) = 0.76 + (0- 0.16) 2

D(2)
' (x;- va) 2
=L
= 0.0256 + 0.4096 + 0.0256 + 0.0256
D(l) = 0.4864
f=l
4
=(I - 0.2) 2 + (0- 0.2) 2 + (0- 0.8) 2
+ (0- 0.8) 2
D(2) = L (x; - va) 2
i=l
= 0.64 + 0.04 + 0.64 + 0.64 = (l - 0.2) 2 + (0- 0.2) 2 + (0- 0.8) 2
D(2) = 1.96 + (0- 0.8) 2
Since D( 1) < D(2), rhe winner unit index is J = l. = 0.64 + 0.04 + 0.64 + 0.64
D(2) = 1.96
Weight updation: The weight updarion on ilie winner
unit is given by Since D(l) < 0(2), the winner unit index is]= 1.

v;j(new) = Vij(oldl+ a[x;- v;j(old)] Weight updation on winner unit:

Updating the weights into unit z;:
Fori= 1 to4and}:= I, we obtain
Vij(new) = Vij(old)+a[x;- Vij(old)]
v11 (new) = Vii (O)+ a[x1.- VJJ (old)]
= 0.8 + 0.2(1 - 0.8) = 0.84 Fori= 1 w 4 and]= l, we obtain

11zJ(ncw) = li;!J(O)+a[x2- VzJ(old)l vn (new) = VJJ (O)+ a[x1 - VJJ (old)]

= 0.8 + 0.2(0 - 0.8) = 0.64 = 0.84 + 0.2(1 - 0.84) = 0.872

l
 5.8 Solved Problems 213

v.n (new) = "21 (0)+ a[x-, - v, 1(old)]

~-- Step 0: Initialize the parameters: -~
= 0.64 + 0.2(0- 0.64) = 0.512
.,,(new)= v31(0)+a[x,- v, 1(old)] p= 0.4; a=2

= 0.16+0.2(0 -0.16) = 0.128 Initialize weights:

"41 (new) = "" (0)+ a[X4 - ""(old)] bij(O) = 0.2; tji(O) = I
= 0.!6"+ 0.2(0- 0.16) = 0.128
Step 1: Start compuration.
Updating the weights from unitzj to output layer: Step 2: For the first input vecwr [0 0 0 l],
w;1(new) = w;j(old) +a ly,- Wkj(o!a)] perform Steps 3-12.
Step 3: Set activations of all F2 units to zero. Set
Fork= 1 to 2 and]= 1, we obtain activations ofF! {a) units to input vector
w11 (new) = w11(0) +a 1.1'1 - W!l (old)] '= [0 0 0 1].
= 0.5 + 0.2(1- 0.5) = 0.6 Step 4: Compute norm of s :

wn (new) = wn (0) +a 1.1'2 -'"''(old)] 11,11 = 0 + 0 + 0 + 1 = 1

= 0.5 + 0.2(0- 0.5) = 0.4 Step 5: Compute acrivacions for each node in
Thus, the updated weights are after phase II of theFt layer:
tral'I).ing are
x=[0001]

0.872 0.2] Step 6: Compute net input. to each node in the

V= 0.512 0.2 and W = [0.6 0.5] F2 layer:
0.128 0.8 0.4 0.5
[ 4
0.128 0.8
Jj = LhijXi
J~nstruct an ART 1 nerwork for clustering four i=l
// input vecwrs with low vigilance parameter of 1
Forj= 1 to3,
0.4 into rhree dusters. The four input vectors
are [0 0 0 1], [0 1 0 1]. [0 0 1 1] and [1 0 0 0]. Yl = 0.2(0) + 0.2(0) + 0.2(0) + 0.2(1)
Assume the necessary parameters needed.
=0.2
Solution: The values assumed in this case are p = )'2 = 0.2(0) + 0.2(0) + 0.2(0) + 0.2(1)
0.4, a = 2. Also it can be nQ[ed that ~ 4 and
=0.2
m=3.Hence,
BonOffi~up-weights,(bij@ - 1{1_ + ii9-
1/l 4 = + Yl = 0.2(0) + 0.2(0) + 0.2(0) + 0.2(1)
0.2. . =0.2
Topdownweights ~'/(0) = 1.
Step 7: When reset is true, perform Steps 8-11.
Fori-== 1 to4 andj= 1 to 2,
Step 8: Since all the inpms pose same net input,
there exists a tie and the unit with the
''[0.2 0.2 0.2]
b- 0.2 0.2 0.2 smallest index is the winner, i.e.,] =1.
'l ~- 0.2 0.2 0.2
Step 9: Recompute theF 1 activations {for]= I):
k 0.2 0.2 0.2 4x3
x; = s;tji

and ;;= [:
:L4 X! ='l'Jl = (0 0 0 1][1 1 1 1)
XJ=[O 0 0 l]
 214 Unsupervised learning Networks

Step 10: Calculate norm of x: Step 4: Compme norm of s:

1\xll =I 1\sl\=0+1+0+1=2

Step 11: Test for reset condition: Step 5: Computeactivarionsofeachnodein the

F1 layer:
1\xll _ ~= 1.0 ~ 0.4 (p)
f,jf- I X= [0 1 0 I]
Hence reset is false. Proceed to Srep 12. Step 6: Compute net input to each node in the
Step 12: Update bottom-up weights for a= 2: F2 layer:
~ ax 4
b;J(new) = a-1 ~ 1\xll Yi = LbijXi
2x; 2x; i=l

2-l+llxll = 1+1\x\\ Forj=lto3,

2X0
bu = - - =0; Yl = 0(0) + 0(1) + 0(0) + 1(1) = 1
1+ 1
2x0 )'2 = 0.2(0) + 0.2(1) + 0.2(0) + 0.2(1)
b,l = - - =0;
I+ I =0.4
2x0
' b,l = - - =0; Yl = 0.2(0) + 0.2(1) + 0.2(0) + 0.2(1)
_}., ~ \ I+ I =0.4
j" 2X 1 2
b,i = - - =- = 1
1+ 1 2 Step 7: When reset is true, perform Steps 8-11.
Therefore, the bottom-up weight Step 8: The unit with largest net input is the
matrix bij becomes winner, i.e.,]::::. 1.
Step 9: ReCompute F1 activations (for]= 1):
0 0.2 0.2]
b = 0 0.2 0.2 x; =s;tp= [0 10 1][000 1] = [000 1]
u [0 0.2 0.2
Step 10: Calculate norm of x:
I 0.2 0.2

Update the top-down weights: 1\x\\ = 0+ 0+ 0+ I= 1

Step 11: Test for reset condition:

'Ji (new) = x;

0 0 0 I] 1\x\\ _ ~ = 0.5 2: 0.4 (p)

IJ;=III1 fsil- 2
[I 1 1 1
Hence reset is false. Proceed to Step 12.
Step 12: Update bortom~up weights for a ::::: 2:
Seeps 0 and 1 remain rhe same.
2x;
J Step 2: For the second input vector [0 1 0 1], I bif(n<W) = 1 + 1\x\\
perform Steps 3-12. 2 X 0 = 0;
Step 3: Set acrivations of all F2 ..units to zero.
bu = 1 + 1
Set activations of F 1 (a) units to inpur 2 X 0 = 0;
vecror s = [0 l 0 1]. bzi = 1+ 1
 5.8 Solved Problems 215

2x0 Step 7: When reset is true, perform Steps 8-11.

h),= 1+1 =O;
Step 8: The unit wirh largest net input is the
2X I 2 winner, i.e.,]= I.
b41 = - - = - = I
I+ I 2
Step 9: Recompute F 1 activations (for]= 1):
Therefore, the bouomup weight
matrix b,J becomes x; = s;y; = [0 0 1 1][0 0 0 1]
= [0 0 0 I]
0 0.2 0.2}
b = 0 0.2 0.2
if 0 0.2 0.2 Step 10: Calculate norm of x:
[
1 0.2 0.2
1\x\1=0+0+0+1=1
Update the topdown weights:
Step 11: Test for reset condition:
y;(new) = Xj

\\x\1 _ ~ = 0.5 ~ 0.4 (p)

0 0 0 I]
tji:=l111 fsil- 2
[1 1 1 I
Hence reset is false. Proceed ro Step 12.
Step 12: Update bottomup weights for a= 2:
Steps 0 and 1 remain the same.

1-Step-2:-For-the-third
- input
- -vector
- -[0 - 1 2x;
0 1 1], by(new) = 1 + \\x\1
perform Steps 3-12.
2 X 0 _ 0
Step 3: Set activations of all F2 units to zero. bll=I+l-'
Set activations of F1 (a) units to input
2 X 0 _ 0
vectors= [0 0 I 1]. b21 ::::. 1 +I - '
Step 4: Compute norm of s:
2 x 0 _ O
1\s\\ = 0 + 0 + 1+ 1 = 2
hJJ = 1+1- .
2x 1 _:=1
Step 5: Computeacrivarionsofeach node in the b4l = 1+1- 2
F 1 layer:
Therefore, the borrom~up weight
x=[0011]
matrix hi] becomes
Step 6: Compute net input to each node in the
F2 layer: 0 0.2
0 0.2 0.2]
0.2
4
bq = 0 0.2 0.2
Yj::= LbijXi [
1 0.2 0.2
i=I

Forj= 1 ro3, Update the topdown weights:

Yl = 0(0) + 0(0) + 0(1) + 1(1) = 1

tp(new) =Xi
)'2= 0.2(0) + 0.2(0) + 0.2(1) + 0.2(1)
=0.4 0 00 1]
+
y, = 0.2(0) 0.2(0) + 0.2(1) 0.2(1) + tji=
[11 11 11 11
=0.4

1
216 Unsupervised Learning Networks

Steps 0 and 1 remain the same. Step 12: Update bottom-up weights fora= 2:

I Step 2: For the foui:th i~put vecror [1 0 0 0], I 2x

bij(new) = 1 + ilxll
perform Steps 3-12.
2X1 2
Step 3: Set activations of all F2 units co zero. bll = -.- =- = 1
Set activations of F1 (a) unir.s to input . 1+ 1 2 '
2x0
vecror s = [l 0 0 0].
b:zJ = 1 1 + = O;
Step 4: Compute norm of s:
2x0
llrll=1+0+0+0=1 b,l = 1 + 1 = 0;
2x0
Step 5: Compute activations ofeach node in the b<l=--=0
1+ 1
F1 layer:
Therefore, the bottom-up weight
x=[1000] mauix b;J becomes
Step 6: Compute net input to each node in the
0 1 0.2]
F2 layer: 0 0 0.2
bij= 0 0 0.2
[
'
Jj= LbijXi
1 0 0.2
i=l Update the top-down weights:
Forj= l m3, tji(new) = x;
]J = 0(1) + 0(0) + 0(0) + 1(0) = 0
00 1]
)'2 = 0.2(1) + 0.2(0) + 0.2(0) + 0.2(0) 'Ji = [: 0 0 0
1 1 1
=0.2
Y3 = 0.2(1) + 0.2(0) + 0.2(0) + 0.2(0) Step 13: Test for stopping condition. (This com-
=0.2 1 pletes one epoch of training.) \

Step 7: When reset is true, perform Steps 8-11. The network may be trained for a particular number
Step 8: The unit with largest net input is the of epochs on the basis of the stopping condition.
winner, i.e.,]= 2.

~
onsider an ART 1 neural net with four F1
Step 9: Recompute F1 activations: nits and three F2 units. After some training,
' the weights are as follows:
x;=s;tj;=[1 0 0 0][1 1 1 1]
= [1 0 0 0] Bottom-up weights Top-down weights

Step 10: Calculate norm of x: 00.67 00 0.2]

0.2 [ 1 0 0 0]
bij = 0 0 0.2 fji = 0 0 0 1
[ 1 1 1 1
llxll=1+0+0+0=1 0 0.67 0.2 4x3 3x4
Step 11: Test for reset condition:
Determine the new weight matrices after the
llxll _ ~ = 1 ;,: 0.4 (p l vector [0, 0, 1, I] is presented if

"'"- 1 The vigilance parameter is 0.3.

Hence reset is false. Proceed to Step 12. The vigilance parameter is 0.7.

......
 5.8 Solved Problems 217

Solution: It can be noted that n = 4, m = 3 (clusters) We now t<;St for reset. Since
and a= 2.
Vigilance parameter p = 0.3.
llxll _ ~ = 0.5 :>: 0.3(p)
Bottom-up weight, "'"- 2
we update the weights. Update bottom-up weights -
1 1 (bij).
HOl=-=-=0.2
IJ l+n 1+4 ax
bij(new) = ' (a= 2)
Top-down weight a 1 + llxll
, 2X 0
t;;(O) = 1 bl2 = = 0;
2 I+ 1
Note:_ These are not necessary m bij and tji weights 2x0
are already given. b22:::;:: = 0;
2-1+1
We now compute norm of s = [0 0 I 1]: 2x0
b32= =0;
llrll '<" 0 + 0 + 1 + 1 = 2 2-1+1
2xl
Then we compute the activations ofF,Iayer: b" = = 1
2-1+1
X= [0 0 1 1] Update rop-down weights, t_p(new) = x;.
The new top-down weights are
Now, calculate the net input:

' 1 0 0 OJ
tJ;=OOOI
Jj= Lx;bij
i=l
[1 1 1 1
J'l
'
= Lx;b;~
The new bottom-up weights are

i=J
0.67 0 0.2]
= 0.67 X 0+ 0 X 0+ 1X 0+ 1X 0= 0 0 0 0.2
bij = 0 0 0.2
'
Y2 = Lx;b,-l
[
0 1 0.2
i=l
Vigilance parameter p = 0.7. The in pur vector is
= 0 X 0+ 0 X 0 + 1 X 0 + 1 X 0.67 = 0.67
s = [0 0 1 I). Now calculate norm of s,

Y3 = Lx;ba
' llrll=0+0+1+1=2
i=l
= 0 X 0.2 + 0 X 0.2 + 1 X 0.2 + 1 X 0.2 Set activations of F1 layer as
= 0.4 x=[0011]
Since Y2 is the largest, hence the winner unit is Calculating the net input we obtain
]=2.
Compute F1 activations again, '
Jj= L;x;bij
x,- = s;tji = [0 0 1][0 0 0 1] i=l

= [0 0 0 1]
)'I=
'
Lx,-b,-1
Computing the norm of x we obtain i=l

llxll = O+O+O+ 1 = 1 = 0.67 X 0+ 0 X Q+ 1X 0+ 1X 0= 0

...
 Unsupervised Leaming Networks
218

'
Y2 = Lx;b;2
,, 2x0
~2-_--,1--'+--:2 = O;

i=l 2 X I = 0.67;
= 0 X 0 + 0 X 0 + 1 X 0 +I X 0.67 = 0.67 b"= 2-1+2
2X 1 =0.67
Y3 =
'
Ex;ba
b4)=2-1+2

i=l The updated bonom-up weights are

= 0 X 0.2 + 0 X 0 + 1 X 0.2 + 1 X 0.2
=0.4 0.67
00 00 ]
As Y2 is Ute largest, therefore the winner unit index bij=
[
~ 0 0.67
is]= 2. 0.67 0.67
Recomputing the activations of F 1 layer we get
(j= 2) The top-down weights are given by tj;(new) = x;.
Hence the updated top-down weights are
x;=s;q;=[O 0 1 1][0 0 0 1]
=[0001]
1000]
q;= 0 0 0 I
The norm of x is [0 0 I I
llx\\=0+0+0+1=1
. 13. Co_nsider an ART 1 networ~ with nine- in puc _F1
We now test for reset condition. Sine<! 7 unns
and two duster F1 unm . .After some tram
ing, the bortomup weights bij and top-down
llxll _ ~ = 0.5 < 0.7 (p l weights fji have the following values:
f,i\ - 2
The bmwm-up weights
yz = -1 {inhibit node 2). Therefore, the net inpur
becomes 113 1/10
0 1110
y,=O; y,=-1; JJ=0.4 113 1/10
As the largest is Y3 the winner unit index is}= 3. 0 1110
Recomputing F 1 layer activations, we get bij = \113 1110
0 1110
x;=s;t;;= [0 0 1][1 I I 1] 113 1110
= [0 0 1] 0 1110
113 1110
From rhiswegerthar the normofllxU = 2. Testing
for reset we obtain Top-down weights fji

llxll_~= I> 0.7(p)

f,i\ - 2 [1 0 I 0 I 0 I 0 I]
tji==L1 1 1 I I 1 1 I I
Hence we update the weights. The bottom-up
weigh~ are (x; = [0 0 I 1],] = 3) The pattern (1 1 1 1 0 1 I 1 1) is
ax; presented to the ne[Wctk Compute the action
bij(new) = a -I + llxll of the network if
the vigilance parameter is 0. 5;
2 x 0 _ O
bll= 2-1+2- rhe vigilance parameter is 0.8.
 5.8 Solved Problems 219

Solution: Here n = 9, m = 2. It can be seen that YI > Y2 so the winner unit

\ndex is]= 1. Recomputing the activations ofF 1
Vigilance parameter p = 0.5. The initial weight
layer, we obtain (for j = 1)
inatrices are
113 1110 x; = s;tp = [1 I I 1 0 I I I 1]
0 1110 [I 0 I 0 I 0 1 0 I]
113 1110
X= [I 0 1 0 0 0 1 0 I]
0 1110
bij = 1113 1/10 Calculating the norm of x, we obtain
0 1110
113 1110 llx\\ = 1 +0+ 1+0+0+0+1 +O+ I =4
0 1110
_113 1110- Tesi:ing for reset implies
1 0 1 0 1 0 1 0
tp= ( 1 1 1 1 1 1 1 I 1
I] llxll _ ~ = ~ = 0.5) 0.5(p)
W-8 2
The input parc:ern iss f= [1 1 1 1 0 I 1 1 1].
Calculating the norm of 1, we obtain Reset is false. Hence we update the weights.
Updating bottom-up weights for a= 2 we get
9
\\sll = L'' = 8 bij(new)
ctXj 2x;
i=l a-1 + llxll 2-1 + llxll
We now compute the activations ofF1 layer, 2x;
= 1 +1\xll
X= [I I I I 0 1 I I lj
2(1) - ~. 2(0) = 0;
Calculating the net input, we obtain b"=l+4-5' b21=1+4

9 2(1) - ~. 2(0) = 0;
b31=1+4-5' b"=l+4
Yi == Lx;bij
i=l 2(0) _ . 2(0) = 0;
b"=l+4-. 0 b61=1+4
9
Yl = Lxibil 2(1) - ~ 2(0) = 0;
i=l b71=1+4-5' b,,=l+4
= 1(113) + 1(0) + 1(113) + 1(0) + 0(113) 2(1) - ~
+ 1(0) + 1(113) + 1(0) + 1(113) b"=l+4-5
I 1 I 1 4 The updated bottom-up weights are
=-+-+-+-=-=1.3
3 3 3 3 3
9 2/5
1/10
]2= Lxib,]. 0
1110
i=-1 215
1110
= 1(1110) + 1(1110) + 1(1/10) + 1(1110) 0
1110
+ 0(1110) + 1(1110) + 1(1110) + 1(1110) b,= 1 o 1110
0 1110
+ 1(1110)
215 1/10
8 0 1110
= 10 = 0.8
215 1110

L
 Unsupervised Learning Networks
220

We now update top...down we~ghts using tp (new} Testing for reset we obtain
= x;. The new updated top-down weight are
1!1_4 l
1 0 1 0 0 0 1 0 1] \ls\1 - 8= 2= 0.5 < 0.8
'Ji= [ 1 I 1 1 1 1 1 1 1
Hence Yt = -1 (inhibit node 1). Therefore, the
Vigilance parameter p = 0.8. The input pauern
dot products become
~ s = [I 1 1 1 0 1 1 1 1]. Calcula<ing
the norm ofsusing the formula as in (1), we obWn
y,=-l; y,=0.8
\ls\1 = 8
We now compute the activations ofF 1 layer,
Since Y2 > YI, ilie winner unit index is J = 2.
Recompming activations ofF 1 layer (for] = 2)
X= [1 I 1 1 0 I 1 1 1] gives
Calculating the net input, we get x; = s;t}i = [1 1 1 1 0 1 1 1 l]
9 X [1 l 1 l l l l 1 l]
Yi = Lx;b;j X= [1 l 1 l 0 l l l !]
i=l
9
Again computing the norm of x, we get
Yl = Lx;bil
i=l \lx\1 = l + l + l + l + 0 + 1 + 1 + 1 + l = 8
= 1(113) + 1(0) + 1(113) + 1(0) + 0(113)
+ 1(0) + 1(113) + 1(0) + 1(113) Testing for reset gives
1 I 1 1 4
= - + - + - + - + - = 1.3
3 3 3 3 3 \lx\1 _ ~ = 1 > 0.8
9
"'"- 8
yz = Lx;bi2
Hence we update the weights.
i=l
Update bouom-up weights, for a= 2, using the
= 1(1/10) + 1(1110) + 1(1110) + 1(1110)
formula
+ 1(1/10) + 1(1110) + 1(1110)
+ 1(11!0) + 1(1110) + 1(1110) ax;
bij(new) = a-l + \lx\1
8
= - = 0.8 2x; 2x;
10
2-l +\lx\1 l + \lx\1
h can be seen that Yl > Y2 Hence the winner unit
index is J = l. For all x; = I (where i = 1 to 4 and 6 ro 9, and
Recomputing Ute activations ofF 1 layer, we obtain
J= 2),
(for J = l)
2 X l 2
x; = s;tj; = [l l l l 0 l l l l] bij(new) = l +8 = 9
[l 0 l 0 1 0 l 0 l]
x= [I 0 1 0 0 0 1 0 l] For x; = 0 (where i = 5 and]= 2)

Computing the norm of x, we have

2 X0 _ 0
bij(new) = 1 + 8 -
\lx\1 =I + 0 + 1 + 0 + 0 + 0 + 1 + 0 + l = 4

1
5.8 Solved Problems 221

The updated bottom-up weights are v = f(x;)+ bf(q;)

= [(0.717, 0.697)
l/3 219
0 219 v = (0.72,0)
l/3 219
0 219 Here the activation function is
bij = ll/3 0
0 219 f(x) = {X f(x) ?_ e
l/3 219
0 f(x) < e
0 219
Since()= 0.7, the output v = (0.72,0). Update Ft
113 2/9 activations again:
Updated top..down weights can be calculated
- -v- -
U (0.72,0)
- - --- -( l,Q)
using the formula t;;(new) = x;.
The new updated top-down weights are
e+l\v\1 0.72
w= s+au= (0.7!, 0.69) + lO(l, 0) = (!0.7!, 0.69)
l 0 l 0 1 0 l 0 l] p=u+dlj=(l,O)
,,,= [ l l 1 l 0 l l l l
w (!0.7!, 0.69) I
X= --11-II
t+ w
= 10.7
32 = 0.998, 0.064)
14. Consider an ART 2 network with two input
units. (n = 2). Show that using () = 0.7 will _ _P_ _ (l,O) _ l
force the input patterns (0.71, 0.69) and (0.69, q-e+\IP\1- l -(,O)
0.71) to different clusters. What role does the v; =fix;)+ bf(q,)
vigilance parameters play in this case? (Do not
calculate the weights, stop wilh checking of reset
= [(0.998, 0.064) !Of(l,O)+
condition.) Assume the necessary parameters.
= (0.998,0) + lO(l,O)
v; = (!0.998, 0)
Solution: The parameters are assumed to be
Calculating signals to F2 uitits, we get
a= b = 10, '= O.l, d = 0.9, e = 0, a= 0.6,
p = 0.9, e = 0.7, n = 2, 'i = (0, 0), YJ='E bqp;= (7,7) X (l,O) = (7,0)
I
bj = ( ~ "' (7.0, 7.0) The winner unit is j = I, since y 1 has a larger value,
l- r -
i.e. 7, than Y! = 0.
111 pattern: Check for restt:
s= (0.71,0.69) v (!0.998, 0)
v u= e+ 1\u\1 = !0.998 = (l,O)
u= - - =(0,0)
e+ \lu\1
p = u + dlj = (l,O)+ 0.9(0, 0) = (l,O)
w = s+ au= (0.71, 0.69)
u+cp (l,O)+O.l(l,O)
p = u+dtj= (0,0) I'= =
e+\lul\+e\lp\1 0+!+0.1 x l
w (0.7!, 0.69)
X= - -- = (0.717, 0.697) (l.l, 0)
e+ 1\w\1 0.99 =--=(1,0)
l.l
(where \\w\1 = Jr(0-.7-1)-:-2 -+~(-0.-69-)2 = 0.99)
Computing norm of r, we get
_ _P_ _ (OO)
q-e+\IP\1-' \ldl = l > (p -e)= l > 0.9
 Unsupervised Learning Networks
222
Thewinnerunitis]= 2,sinceJ2> Yl [i.e., 7> 0]
This implies
Check for reJtt
w ~'+au~ (0.71,0.69) + 10(1,0)
v (0, 10.998)
~ (10.71,0.69) u~ e+l\v\1 ~ 10.998 ~(0, 1 )
w
x~ e+llwll ~(0.998,0.0b4) p ~ u + dt; ~ (0, 1)
u + cp (0,1) + 0.1(0,1)
p
q ~ e+l\p II ~ (l,O) r~
e+l\ul\+cl\p\1 0+1+0.1x1
v ~ f(x) + bf(q) ~ (10.998, 0) (O,l.l) (
~--~ 0,1)
l.l
2nd pattern llrll ~ 1 > (p -e)~ 0.9.

'~ (0.69, 0.71) Then,

v
u~--~o w ~'+au~ (0.69, 0.71) + 10(0,1)
e+ 1111
~ (0.67,10.71)
w ~>+au~ (0.69,0.71)
w (0.69,10.71)
p~u+dc;~(O,O) X= e+ 1\wll ~ _ ~ (0.064,0.998)
10 732
w (0.69,0.71)
x~ e+ llwll ~ _ (0.697,0.717) _ _P_ _ (O 1)
0 99 q- e+IIPII- '
p
q~ e+liPli ~(O,O) v ~ j(x) + bf(q) ~ (0,10.998)

v ~ j(x) + bf(q) ~ /(0.697,0.717) Thus the vigilance parameter assumed p = 0.9 does
~ (0,0.717) ~ (0,0.71) not affect the solutions for first and second pattern.
lr activates the same for borh the in pur pauerns.
Update F1 accivarions again: 15. Consider an ART 2 network to cluster the
input vectors (0.6, 0.8, 0.0) and (0.8, 0.6, 0.0)
v (0, 0.72) together? When will it place (0.6, 0.2, 0.0)
"~ - - ~ - - ~ (O.l)
'+ 1111 0.72 together with (0.0, 1.0, 0.0)? Use the noise sup
w ~>+au~ (0.69, 0.71) + 10(0, 1) pression parameter value e = lJ3 = 0.577
~ (0.69, 10.71) and consider different values of the vigilance
and different initial weights. Assume necessary
p ~ u+dc;~ (0,1)
parameters.
w (0.69, 10.71)
X~ --11-II ~ (0.064, 0.998) Solution: Case (i) Taking p:::: 0.9 , presenting (0.6,
'+ w 10.732
0.8, 0) and (0.8, 0.6, 0)
p (0,1)
q~ -- ~ -~(0,1)
e+ liP II 1 a~ 10,b ~ 10,c ~ O.l,d~ 0.9,e ~ O,p ~ 0.9,
v ~ j(x) + bf(q) ~ /(0.064, 0.998) + 10/(0,1) 1 1
e ~ o.577. bJ = ~~=
(1 - d).,;n (l - 0.9)-/3
~ (0,0.998)+ 10(0,1)
v ~ (0, 10.998) ~ (5.0, 5.0, 5.0), 'i ~ (0, 0, 0), a~ 0.6

Calculating signals to Fz units, we get 1st'pttttem:

'~ (0.6, 0.8, 0.0)

Yi ='E bijPi~ (7,7) x (0,1) = (0, 7)

L
 5.8 Solved Problems 223

v Check for reset:

u ~ - - ~ (0, 0, 0)
e+ 1\v\1 v (6.6, 8.8, 0)
w ~'+an~ (0.6, 0.8, 0.0) u~ e+llv\1 ~ O+ll ~(0.6,0.8,0)

p ~ u+ dt; ~ (0,0,0) 'p(=u;+dtj

. = (0.6, 0.8, 0) + 0.9(0, 0, 0) ~ (0.6, 0.8, 0)
_ _ w_ _ (0.6, 0.8, 0.0) ( )
x- e+ 1\w\1- 0+ 1 6 0.8 ,0.0
0., r ~ -~u;~+__:c~p;~
p
e+ \lull +clip II
q~ e+ liP II~ (O,O,O) (0.6, 0.8, 0) + 0.1(0.6, 0.8, 0)
0+1+0.1x1
v; ~ f(x;) + bf(q;) (0.66, 0.88, 0) ( 6 )
0. ,0.8, 0
=/(0.6, 0.8, 0) + 10/(0, 0, 0) l.l

e ~ o.55 ] Computing norm of r, we get

[ j(x) ~ \x, f(x) 0:: 0

~ 1 > (p-e) ~
0, f(x) < e 1\r\1 0.9
p (0.6, 0.8, 0)
v; ~ (0.6, 0.8, 0) q ~ -1-1- ~ (0.6, 0.8, 0)
e+ PI\ 0+1
Update F1 activations again: w
w=--
e+ 1\w\1
v; (0.6, 0.8, 0) ~ (0.6,0.8,0) + 10(0.6,0.8,0) ~ (6.6,8.8,0)
u ~ e+ 1\v\1 ~ + ~ (0.6,0.8,0)
0 1 (6.6, 8.8, 0)
w
w ='+au= (0.6, 0.8, 0) + 10(0.6, 0.8, 0) x~ e+l\w\1 = O+ll ~(0.6,0.8,0)

= (6.6, 8.8, 0) v ~ j(x) + bf(q) ~ (6.6, 8.8, 0)

p ~ u + dt; ~ (0.6, 0.8, 0) Updation ofweights: Weights are updated for winning
unit}= 2.
x ~ _w_ ~ (0.6, 8.8, 0) = (0.6 0.8 0
e+l\w\1 O+ll ' ') <;;(new) ~adu; + ll+ad(d- l))cfi(old)
p (0.6, 0.8, 0) ~ 0.6 X 0.9 X 0.8
q~ -- = (0.6, 0.8, 0)
e+ liP II O+ 1 + 11 + 0.6 X 0.9(0.9- 1)} X 0
V; = j(x;) + bf(q;) ~ 0.432
b;;(new) ~adu;+ {1+ad(d-1))bij(old)
~ /(0.6,0.8, 0) + 10/(0.6,0.8,0)
= 0.6 X 0.9 X 0.8
~ (Oc6, 0.8, 0) + (6, 8, 0)
+ {1 + 0.6 X 0.9(0.9- I)} X 5.0
V; = (6.6, 8.8, 0) ~ 5.162

Calculate the signals to F2 units: 2nd pattern:

'~ (0.8, 0.6, 0)
Yi ~'E bij Pi~ (5.0 X 0.6, 5.0 X 0.8, 5.0 X 0)
v
~--~(00)
~ (3.0, 4.0, 0) e+ 1111
w ~'+au ~ (0.8, 0.6, 0)
The winner unit index is}= 2, since the largest net
p ~ u +dij ~ (0,0)
input is 4.0.

L
224 Unsupervised Learning Networks

w
x= _w_ = (0.8,0.6,0.0) X = - -- = (0.8, 0.6, 0)
<+llwll . e+ 11 wl 1
p v =f(x) + bf(q) = (8.8, 6.6, 0)
q= e+IIPII =(O,o,o)
v =j(xi) + bf(qi) Updiltion ofw~ights: Weights are updated for winning
unit]= 1.
= /(0.8, 0.6, 0.0) + 0 = (0.8, 0.6, 0)
'fi(new) =dui + {l+ad(d- l))t;i(old)
update F1 activations again:
= 0.6 X 0.9 X 0.8
v (0.8, 0.6, 0) + {1 +0.6 X 0.9(0.9 -I)) X 0
u = -- = = (0.8 0.6 0)
e+llvll 0+1 ' ' = 0.432
w = s + qu = ((}.8, 0.6, 0) + 10 (0.8, 0.6, 0) bij(new) =adui + {l+ad(d- I)) bij(old)
= (8.8, 6.6, 0) = 0.6 X 0.9 X 0.8
p = u + d'J = (0.8, 0.6, 0) +(I+ 0.6 X 0.9(0.9- I)) X 5.0
w
X= - - - = (0.8, 0.6, 0) = 5.162
e+ 11 wll
p Thus from the results for presenting the patterns
q = --11- = (0.8, 0.6, 0)
e+ Pll (0.6, 0.8, 0) and (0.6, 0.8, 0) with the assump-
v = j(x) + bf(q) tion of p = 0.9, even though the winning clus~
= /(0.8,0.6,0) + 10/(0.8, 0.6. 0) rer unit is differen~, due w the componems of
input vector, output weights remains same. Thus
= (0.8, 0.6, 0) + (8, 6, 0)
they are placed together-since both the weights are
v = (8.8, 6.6, 0) same, both the patterns will be placed at the same
location.
Calculate signals w h units:
Case (ii): Taking p = 0.7, preseming (0.6, 0.8, 0) and
Yi = E bijPi= (5, 5. 0) x (0.8, 0.6, 0) = (4, 3, 0) (0, I, 0).

The winner unit index is J = 1, since the largest net a= IO,b= IO,e = O.l,d= 0.9,e= O,p= 0.7,
in put is 4.0. 9 = 0.577, a= 0.6, bi = (6.0, 6.0, 6.0),
Check for met: 'f = (0, 0,0)

v (8.8, 6.6, 0) lst pattern

u=e+llvll= 0+11 =(0.8,0.6,0)
s = "(0.6, 0.8, 0.0)
Pi = u + dt; = (0.8, 0.6, 0)
v
u;+cp; u= - - =(0,0,0)
'i = e+ !lull+ ellp II = (0.8, 0.6, 0) e+ llvU
w = s +au= (0.6, 0.8, 0)
Compming norm of r, we get p = u+dtj= (0,0,0)

lldl = I > (p -e) = 0.9

x= _w_ = (0.6, 0.8, 0.0)
e+IIOII
p (0.8, 0.6, 0) _ _P_ -(0 0 0)
q = e+ liP II = 0 +I = (0.8,0.6,0) q-e+IIPII- ' '
w = s+ au= (8.8,6.6,0) Vi = flxi) + bf(qi) = (0.6, 0.8, 0)
5.8 Solved Problems 225

Update F1 accivarions again: bij(new) =adui + {1+ ad(d- I)) bif(old)

v = 0.6 X 0.9 X 0.8

u = , + II vii = (0.6, 0.8, 0)
+{I+ 0,6 X 0.9(0.9- 1)) X 6.0
w= s+ au= (6.6,8.8,0)
=6.108
p = u + dtj = (0.6, 0.8, 0)
w 2nd pattern:
x= e+ llwll = (0.6,0.8,0)
p s=(O,I,O)
q = , + liP II = (0.6, 0.8, 0)
u- v
- e+llvll = (0,0,0)
v = f(x,) + bf(qi) = (6.6, 8.8, 0)
w = s+ au= (0, 1,0)
Calculate the signals to F2 units:
p = u+ dtj = (0, 1,0)
Jj=l::b,j"pi= (6 X 0.6,6 X 0.8,6 X 0) w
X= - -- = (0.0, 1.0, 0.0)
= (3.6, 4.8, 0) e+ 1lwll
p
The winner unit index is J = 2, since the largest net q = - - = (0.0, 1.0, 0.0)
e+ liP II
input is 4.8.
v = j(xi) + bf(qi) = (0.0, 1.0, 0.0)
Check for met:

v (6.6, 8.8, 0) Up dare F1 activations again:

u = e+ II vii= O+ 11 = (0.6,0.8,0)
v
u = - - = (0.0, 1.0, 0.0)
Pi = u + d'J = (0.6, 0.8, 0) e+ II vii
u+cp
'= _' + __:...:~-
llwll + ellp 11 = (0.6, 0.8, O)
w = s +au= (0, I, 0)+ 10(0, I, 0) = (0, II, 0)
p = u+dt;= (0, 1,0)
11'11 = I > (p -e) = 0.7 w
p x= e+llwll =(0,1,0)
(0.0, 0.8, 0) = (0.6, 0.8, 0)
q= e+IIPII 0+1 p
w = s +au= (6.6, 8.8, 0)
q = e +liP II = (0, I, 0)
w v = f(x) + bf(q) = (0, 1,0) + (0, 10,0)
X= - - = (0.6, 0.8, 0.0)
e+ 1111 = (0, 11,0)
v = j(x) + bf(q) = (6.6, 8.8, 0)
Calculating signals to h units, we get
Up dation ofweights: Weights are updated for winning
unit]= 2.
Yi =l:: bij Pi= (6.0, 6.0, 6.0)(0.0, 1.0, 0.0)
lji(new) =adui+ {l+ad(d-l))l]i(old) = (0.0, 6.0, 0.0)
= 0.6 X 0.9 X 0.8 + 0
The winner unit index: is] :::: 2, since the largest net
= 0.432 input is 6.0.
 Unsupervised learning Networks
226

Check for met: v = f(x) + bf(q) = /(0, 1,0) + 10/(0, 1,0)

v (0,11,0) =(0,11,0)
u = c+ II vii= O+il = (0, 1,0)
Updation ofweights: Weights are updated for winning
p = u+dtj= (0,1,0) unit]= 2.
u+cp _ (0, 1,0)+0.1 (0, 1,0)
r
e+llull+cllpll- 0+1+0.1 tp(new) =adu; + {l+ad(d- 1)}1j;(old)
(0.0, 1.1,0.0) = 0.6 X 0.9 X 1 + 0 = 0.54
r 0 + l.l = (0:0, 1.0, 0.0)
b;j(new) =adu; + {l+ad(d- l)} bij(o1d)
llrll = 1 > (p -c)= 0.7
p = 0.6 X 0.9 X 1
q = r+ liP II = (0.0, 1.0, 0.0)
+ {1 + 0.6 X 0.9(0.9- 1)} X 6 = 6.216
w ='+au = (0.0, 1.0, 0.0) + 10 (0.0, 1.0, 0.0)
= (0, 11,0) Thus ilie WO inputs may be clustered mgether only
w when their weights become equal, this can be achieved
x= e+llwll =(0, 1,0) by proper selection of initial weights.

I 5.9 Review Questions

l. What is meanr by unsupervised learning? 14. Write the principle involved in learning vector
quantization.
2. Define exemplar vector or code book vecmr.
15. What is rhe purpose ofLVQnet?
3. List rhe fixed weight comperirive ners.
16. How are rhe initial weights determined for LYQ
4. Draw rhe architecture of Mexican hat and stare
ner?
irs activation fUnction.
17. With architecture, describe how LVQ nets are
5. What is winneHakes-all or clustering principle
rrained.
or competitive learning?
18. List the variants of LVQ ner.
6. Why inhibitory weights are used in Max ret?
19. Srate Kohonen's learning rule and Grossberg
7. What are "mpology preserving" maps?
learning rule.
8. Define Euclidean distance.
20. Discuss rhe applications of counrerpropagation
9. Briefly discuss about Hamming ner. nerwork.
10. How is competition performed for clustering of 21. How many srages are needed for training a CPN
rhe vectors? network?
11. State the application ofKohonen sdf~orga.nizing 22. Mention the importance of in-star model and
maps. out-Har model.
12. With neat architecture, explain the training 23. Sketch the architecture of full Counter Propaga-
algorithm of Kohonen self~organizing fearure tion Network.
mnps. 24. How are CPN nets used for function approxi-
13. Discuss the important fearures ofKohonen self- mation?
organizing maps.

 5.10 Exercise Problems 227

25. Write rhe training algorithms and testing alga~ 38. Differentiate fast learning and slow learning.
rithms used in full counrerpropagation network. 39. List the advantages and disadvantages of ART
26. Compare full counrerpropagarion network and network.
forward-only counterpropagarion network. ; 40. What are the applications of ART networks?
27. What is the principle strength of competitive 41. Sketch the architecture of ART 1 network and
learning? discuss its training algorithm.
28. State the merits and demerits of Kohonen self- 42. Stare the significance of ART 2 network.
organizing feature maps.
43. Why more complexity is involved in the F1 layer
29. What are called as similarity maps? of ART 2 network?
<
30. Define stability and plasticity. 44. How slow learning and fast learning is. achieved
31. Differentiate between ART networks and CPN in ART 2 network?
networks. 45. What is the activation function used in ART 2
32. List the type of input patterns given to ART 1 networks?
and ART 2 nerwork. 46. List the characteristics of ART network.
33. Mention the three main components of an ART 47. Why reset mechanism is essemial in ART net-
network. works?
34. Define bottom-up weight and top-down weight. 48. W.ith neat architecture, explain the training
35. What is vigilance parameter and noise suppres- algorithm used in ART network.
sian parameter? 49. State the assumptions made in ART 2 network.
36. IHustrate with neat figure, the rwo basic units of 50. Mention the limitation of ART 1 network and
an ART 1 network. how is it overcome in ART 2 network.
37. Discuss the importance of supplemental units in
ART 1 nerwork.

I 5.10 Exercise Problems

I. Construct a Max net with four neurons and 0.05). Using learning rate of 0.25, find the new
inhibitory weights E = 0.25 when given the weights.
initial activations (inpm signals). The initial acti~
vations are llt(O} = 0.1, a2(0) = 0.3, tl)(O) = 1~
0.4, a4(0) = 0.7. 0.5

2. Construct a Kohonen self-organizing feature

map to duster four vectors [0 0 1 1], [1 0 0
1}, [0 1 0 1}, {1 1 1 1}. The maximum num-
ber of clusters to be formed is 2 and assume
learning rate as 0.5. Assume random initial
weights.
Figure 11 KSOFM net.
3. Given a Kohonen self-organizing map with
weighrs as shown in the following figure, use
square of euclidean distance to find the clus- 4. Repeat the preceding exercise problem for input
ter unit that is close to the input vector (0.35, vector [0.4, 0.4] with a= 0.15.

L
228 Unsupervised learning Networks

5. Consider a Kohonen net with two cluster units a =0.4, showwhichclassification unit moves
and five input unirs. Th~ weights vectors for the where.
cluster units are Presem an input vec[Qr of (0.6, 0.75) repre
seming class 1. What happens to the network
WI = (l.O 0.9 0~7 0.3 0.2) performance?
W2 = (0.6 0.7 0.5 0.4 l.O) Present the vector (0.4, 0.55) representing
class 1. Note what happens.
Use the square of the Euclidean distance to find
the winning cluster unit for the input pattern 8. lmplemem a coumerpropagation ne[Work for
x = (0.0 0.2 0.1 0.2 0.0) . Usi~g a leaming approximating the functions:
rate of0.2, find the new weights for ilie winning
' f(x) = 11x
unir.
6. Construct an LVQ net to cluster five vectors
[(x) = 7l-
assigned to two classes. The following input 9. Consider che following full CPN net:
vectors represem rwo classes 1 and 2.

Vectors Class
(I 0 0 1)
(1 1 0 0) 2
(0 1 1 0) 1
(1 0 0 0) 2
(0 0 1 1)

Perform only one epoch of training.

7. Consider an LYQ net with t'HO input units and
four target classes: CJ, q, "3 and q. There are 16
classification units, with weight vectors indicated
by the coordinates on the following charr, read X,
in row-column order.
Figure 12 Full CPN Nee.
X2
l.O Using the input pair x = (1, 0, 0, O},y = (0, 1},
0.9 q c:z perform che first phase of [faining (one step
",,
,, ",,
0.7 c:z,, '4 only). Find the activation of che cluster layer
0.5 c:z units. Update the weights using a learning rate
0.3 c:z ,, q " of0.25.
0.0 " 10. Repeat Problem 9 using x = (0 1 1 1) and
0.0 0.3 0.5 0.7 0.9 l.O Xi
y = (1, 0) with a learning rate of0.3.
11. Modify Problem 9 to implement forward-only
Using the square oF the e'uclidean. distance,
CPN.
perform the following.
12. Consuuct an ART 1 network to cluster four
Presenr an input vector of (0.35, 0.35) rep vectors (1, 0, 1, 1), (1, I, l, 0), (1, 0, 0, 0)
l
resenting class 3. Us'ing a learning rate of and (0, l, 0, l) in at most three clusters using

lI
l
5.11 Projects 229

very low vigilance parameter. Assume necessary values:

parameters.
Bonom-up weights hij Top down weights tji
13. Consider an ART 1 neural net wich four F 1 112 1/8-
units and chree F2 units. After some train~'ng, . 0 1/8
che weights are as follows: 1/2 1/8
0 118 1 0 1 0 1 0 1 01
Bonomup weighrs bij Topdown weighrs tp 1/2 118 [ 11111111

["" . "l
0 0 0.2 [00 I] 0
0
112
1/8
1/8
0
0
0.37 0.2
0.37 0.2
1I I
1 0 0 1
1
0 1/8

The pattern [1 1 1 0 0 1 1 1} is pre-

semed w the network. Compme the action of
the network if
Determine the new weight matrices after the
vector (0, 0, 1, 1) is presented if IDe vigilance parameter is 0.3;
the vigilance parameter is 0.7.
the vigilance parameter is 0.4;
the vigilance parameter is 0.8. 15. Consider an ART 2 network with two input
units {n = 2). Show that using () = 0.7 will
14. Consider an ART l network with eight input force che input patterns (0.61, 0.59) and (0.59,
(FJ) units and rwo duster (F2) units. After 0.61) to different clusters. What role does vig-
some training, the bottom-up weight (bij) ilance parameter play here? Determine the new
and top-down weights (~r;) are rhe following weights.

I 5.11 Projects

l. Wrire a computer program w implement Kobo- 5. Write a program w approximate the function
nen self-organizing map. Take suitable applica J(x) = 7/x using forward-only counrerpropaga-
tion. Usc 2 input units and 25 cluster units and tion net.
a linear topology for the duster unirs. Perform 6. Let the digits 0, 1, 2, ... , 7 be represented as
20 epochs of training.
2. Write a computer program ro implement the 0: 0 0 0 0 0 0 0
LVQ net absorbed in Problem 7. Train the net
1: 0 0 0 0 0 0 I 0
with several sets of data. Experiment with dif-
ferent learning rares and different numbers of 2: 0 0 0 0 0 1 0 0
classi ficarion units. 3: 0 0 0 0 1 0 0 0
3. Write a program for coumerpropagarion net- 4: 0 0 0 1 0 0 0 0
work to approximate che function J(x) = 1/x.
5: 0 0 1 0 0 0 0 0
4. Implement coumerpropagation network for per- 6: 0 I 0 0 0 0 0 0
forming data compression. Take data sets like
heart disease data, cancer clara and credit card
7: I 0 0 0 0 0 0 0
dar a.
230

Use forward~only and full counterpropagation

nets ro map digits to their binary represenmions
0: 0 0 0
Unsupervised !.earning Networks

7. Writer a computer program to implement full

oounrerpropagation network for approximating
the functionf(x) = 3x+ ~-
8. Build a computer program ('oimplementART 1
I
1: 0 0 1
neural ner.
2: 0 1 0 9. Write a computer program ro implement rhe
3: 0 1 1 ART 2 neural network, allowing for either fast
4: 1 0 0 learning or slow learning, based on the number
s: 1 0 of epochs of training and the number of weight
update iterations performed on each learning
6: 1 1 0
trial.
7: 10. Write a compute program ro implement ART 2
Assume rhe necessary parameters involved. network for Problem 15.
I Special Networks 6
Learning Objectives - - - - - - - . - - - - - - - - - - - - - - ,
The other special networks apan from super- The feature of cascade correlation nei:Work
vised learning, unsupervised learning and asso- to fluid its own architecture during training
ciation networks. progresses.
A simulated annealing nePNork. A cognitron and neocognitron ners with their
How Bolrz.mann machine can be used to solve basic features.
optimization problems. Apart from these, cellular NN, logicon NN,
An introduction to Cauchy and Gaussian STCNN are also discussed.
machine. List of neuroprocessor chips that are currendy
A probabilistic neural network. tn use.

I 6.1 Introduction
In this chapter, we will discuss some specialized networks in more derail. Among the networks to be discussed
are Bolrzmann network, ctscade correlation net, probabilistic neural net, Cauchy and Gaussian net, cognitron
and neocognitron nets, spatia-temporal nei:Work, optical neural net, simulated annealing network, cellular
neural ncr and logicon neural ncr. Besides, neuroprocessor chip has also been discussed for rhe benefit of rhe
reader. Bolumann nePNork is designed for optimization problems, such as traveling salesman problem. In this
netv.ork, fixed weights are used based on the constraints and quantity to be optimized. Probabilistic neural net
is designed using rhe probability theory to classify the input data (Bayesian method). Cascade correlation net
is designed depending on the hierarchical arrangement of the hidden units. Cauchy and Gaussian net is the
variation of fixed weight optimization net. Cognitron and neocognirron nets were designed for recognition
of handwritten digits.

I 6.2 Simulated Annealing Network

The concept of simulated annealing has it origin in the physical annealing process performed over merals
and other substances. In metallurgical annealing, a metal body is heated almost to its melting point and then
cooled back slowly to room temperature. This process eventually makes the metal's global energy function
reach an absolute minimum value. If the metal's temperature is reduced quickly, the energy of the metallic
lattice will be higher than this minimum value because of rhe existence of frozen lattice dislocations that
would othe!"'Nise disappear due to thermal agitation. Analogous to the physical annealing behavior, simulated
annealing can make a system change irs state to a higher energy srare having a chance to jump from Ideal

l
 f
~.,

232 Special Networks ~

minima m global minima. There exists a cooling procedure in the simulated annealing process such that rhe ~
j
t
system has a higher probabiliey of changing to an increasing energy state in the beginning phase ofconvergence.
Then, as time goes by, the system becomes stable and always moves in rhe direction of decreasing energy stare f

I
as in the case of normal minimization produce.
With simulated annealing, a system changes its sme from the original state SN 1d to a new state SA"cw
with a probabiliry P given by

p= 1
1 + exp(-/l.E/T)

where !:::.. = old - new (energy change = difference in new energy and old energy) and Tis the non
negative parameter {acts like temperature of a physical system). The probability P as a function of change
in energy (6.) obtained for different values of the temperature Tis shown in Figure 6-1. From Figure 6-1,
it can be noticed that the probability when/).> 0 is always higher than the probability when (). < 0 for
any temperature.
An optimization problem seeks to find some configuration of parameters X= (XJ, ... ,Xn) that minimizes
some function f(k} called cost function. In an artificial neural necwork, configuration parameters are associated
wiili the set of weights and the cost function is associated with the error function.
The simulated annealing concept is used in statistical mechanics and is called Metropolis algorithm. As
discussed earlier, this algorithm is based on a material that anneals into a solid as temperature is slowly
decreased. To understand this, consider the slope of a hill having local valleys. A stone is moving down the
hill. Here, the local valleys are local minima, and the bonom of the hill is going to be the global or universal
minimum. It is possible that the stone may stop at a local minimum and never reaches the global minimum.
In neural nets, this would correspond to a set of weights that correspond to that oflocal minimum, but this
is nm the desired solution. Hence, to overcome this kind of situation, simulated annealing perturbs the stone
such that if it is trapped in a local minimum, it escapes from it and continues falling till it reaches its global
minimum (optimal solution). At that point, further perturbations cannot move the stone to a lower position.
Figure 6-2 shows the simulated annealing between a stone and a hill.

p, _ _,
oE
Hexp (-b.EIT)

T=a r-
T=O

T=1

Figure 61 Probability "P" as a function of change in energy (6.) for different values of temperature T.
 6.3 Boltzmann Machine
233
Stone
(
)

I local
minimun

Figure 62 Simulated annealing-stone and hill.

The components required for annealing algorithm are the following.

1. A basic system configuration: The possible solution of a problem over which we search for a best (optimal)
answer. {In a neural net, this is optimum sready-state weight.)
2. The mov(:'m: A set of allowable moves that permit us to escape from local minima and reach all possible
configurations. ._.
3. A cost fonctlon associated with the error function.

4. A cooling >Chdule.- Stacring of the cost function and mles to detetmine when it should be loweted and by
how much, and when annealing should be terminated.

Simulated annealing networks can be used to make a net\Vork converge to irs global minimum.

~ Boltzmann Machine
The early optimization technique used in artificial neural nernrorks is based on the Boltzmann machine.
When the simulated annealing process is applied w the discrete Hopfield nernrork, it becomes a Boltzmann
m"hine. The netwmk is configuted as the vector of the states of the units, and the stares of the units are binary
valued with probabiliscicstate transiriom. The Boltzmann machine described in this section has fixed weights
wij. Dn applying the Boltzmann machine to a constrained optimization problem, the weights represent the
constraints of the problem and the quantity to be optimized. The discussion here is based on the fact of
maximization of a consensus function (CF).
The Boltzmann machine consists of a set of units {X, and~)
and a set ofbi-directional connections betWeen
pairs of units. This machine can be used as an associative memory. If the units X; and~ are connected, then
wy f. 0. There exisrs symmetry in the weighted interconnections based on the directional narure. It can be
represented as Wij = wp. There also may exist a self-connection for a unit (w;;). For unit X;, irs State x; may
be eirher 1 or 0. The objective of the neural net is to maximize the CF given by

CF = LL WijXiXj
i J5.i
234 Special Networks

The maximum of the CF can be obtained by letting each unit auempr to change its state (alter between" l"
and "0" or "0" and "1 '').The change of state can be done either in parallel or sequential manner. However,
in this case all rhe description is based on sequential manner. The consensus change when unit X; changes irs
state is given by

/>CF(z) = (1 - 2x;) ( Wij +I: WijX;)

j#i

where x; is rhe current srate of unit X;. The variation in coefficient (1 - lx;) is given by

(1 _ 2x;) == I+ 1,
-1, X;
X; ~s currently off
IS currently on

If unit X; were to change its activations then the resulting change in ilie CF can be obtained from the
information that is local to unit X;. Generally, X; does not change irs smre, bur if the smtes are changed, then
this increases the consensus of rhe net. The probability of the network that accepts a change in the state for
unit X; is given by
l
AF(i, T) = l + exp[ ,; CF(z)IT]

where T {temperature) is the controlling parameter and it will gradually decrease as the CF reaches the
maximum value. Low values ofT are acceptable because they increase rhe net consensus since the net accepts
a change in stare. To help the net not to stick with dte local maximum, probabilistic functions are used widely.

I 6.3.1 Architecture

The architecture of a Boltzmann machine is represented through a two-dimensional array of the units in
Figure 6-3. The units within each row and column are fully interconnected. The weights on the imercon-
nections are given by -p where (p > 0). Also, there exists a self-connection for each unit, wirh weight b > 0.
Unit Xij is the common unit on which our discussion is based. The weights present on clte interconnections
are inhibitory.

I 6.3.2 Algorithm

6.3.2.1 $effing the Weights of the Network

The weights of a BoltZmann machine are fixed; hence there is no specific training algorithm for updation of
weights. (For a Boltzmwn machine with learning, there exists a training procedure.) With the Boltzmann
machine weights remaining fiXed, rhe net makes its transition toward maximum of the CE
As shown in Figure 6-3, each unit is connected to every other unit in the same row and same column by
the weights -p(p > 0). The weights indicate the penalties obtained due to the violation that occurs when
more than one unit is "on" in each row and column. There also exists a self-connection for each unit given by
weight b > 0. The self-connection is a bonus given to the unit to rurn on if it can do so withom causing more
than one unit to be on in each row and column. The net function will be desired if p >b. If unit Xij is "off"
and no Qthcr unit in its row or column is "on" then changing the status of Xij to on increases Ute consensus
of lhe o'et by amount b. This is an acceptable change because this increases the consensus. A:, a result, the net
accepts it instead of rejecting it.
 6.3 Boltzmann Machine
235
-p

-p

b
-p

Figure 63 Architecrure of Boltzmann machine.

On the other hand, if any one unit in each row or column of X;j is on rhen changing the state of Xy to on
effects a change in consensus by (h- p). Hence (b- p)< 0 makes p > b, i.e., the effect decreases the consensus.
The net rejects this siruation.

6.3.2.2 Testing Algorithm

It is assumed here that the units are arranged in a two-dimensional array. There are n 2 units. So, the wcighrs
bet\veen unit X;,j and unit XI,J are denoted by w {i, j: I,]).

w(i,j: /,]) = { -~ if i =I or j = J (bur not both)

otherwise

The testing algorithm is as follows.

Step 0:
Initialize the weights represeming the constraints of the problem. Also initialize comrol parameter
T and activate the units.
Step 1: When stopping condition is false, perform Steps 2-8.
Step 2: Perform Steps 3--6 ,?-rimes. (This forms an epoch.)
Step 3,
Imegers I and] are chosen random values berween 1 and n. (Unit U!,f is the currem victim to
change its state.)
Step 4: Calculate the _change in consensus:

i>CF = (I - 2Xr.J) [w(!,j: !,}) + L L w(z~j: /,j)XI.J]

iJ# 1,}

l
 Special Networks
236

Step 5: Calculate the probability of acceprance of the change in state:

AF(T} = 111 + exp[-(t.CF/T)]
Decide whether to accept the change or not. Let R be a random number berween 0 and I. If
Step 6:
R < AF, accept the change:
X1J = 1- X1J (This changes the srate U1,j.)
If R~AF. reject the change.
Step 7: Reduce d-.e control parameter T.
I(new) = 0.95 I(old)

Step 8: Test for stopping condition, which is:

If the temperature reaches a specified value or if rhere is no change of stare for specified number
of epochs then stop, else continue. l
The initial temperature should be taken large enough for accepting the change of state quickly. Also
remember that the Boltzmann machine can be applied for various optimization problems such as traveling
salesman problem.

-
I 6.4 Gaussian Machine
Gaussian machine is one which includes Boltzmann machine, Hopfield net and Q[her neural nerworks. The
Gaussian machine is based on the following three parameters: (a) a slope parameter of sigmoidal function cr,
(b) a rime step b.t, (c) remperawre T.
The steps involved in the operation of the Gaussian net are rhe following:

J Step 1: Compute the net input to unit X;:

N
net; = L WijVj+9; + E
j=l

where(}; is the threshold and E the random noise which depends on remperamre T.
Step 2: Change rhe activiry level of unit X;:

!J.x; _-~+net;
-;;; - t

Step 3: Apply rhe activation function:

v; = j(r;) = 0.5(1 + ronh(r;)]
where the binary step function corresponds to a= 00 (infiniry).

The Gaussian machinewiclJ T = 0 corresponds the Hopfield net. The Bolrz.rr:ann machine can beobmined
by sening b.t =r = 1 to get
b.x; = -x; + net;

l
6.6 Probabilistic Neural Net 237

N
or x;(new) = net;= Z:: Wijvj+(}; + E
j=l

The approximate Boltzmann acceptance function is o~ciined by integrating the Gaussian noise distribution
00'
1 (r-2) 1
J
0
- - - exp - -2 ' dx"" AF(t, T) = -;-c:----,--=
../2rr a' 2a 1 + exp(-r;/T}

where x; =!J.CF(t). The noise which is found"'to obey a logistic rather than a Gaussian distribution produces
a Gaussian machine that is identical to Boltzmann machine having Metropolis acceptance function, i.e., the
output set to 1 with probability,

1
AF(i, T) I+ exp( r;/ T)

This does nor bother about the unit's original state. When noise is added to the net inpm of a unit then using
probabilistic state transition gives a method for extending the Gaussian machine into Cauchy machine.

I 6.5 Cauchy Machine

Cauchy machine can be called list simulated annealing, and it is based on including more noise to the net
input for increasing the likelihood of a unit escaping from a neighborhood of local minimum. Larger changes
in rhe system's configuration can be obtained due to the unbounded variance of the Cauchy distribmion. Noise
involved in Cauchy distribution is called "colored noise" and the noise involved in the Gaussian distribution
is called "white noise."
By setting !J.t =t = l, the Cauchy machine can be extended into the Gaussian machine, to obtain

!:!,.xi = -x; +net;

N
or x;{new) = net;= L Wijvj+(;J; + E
j=l

The Cauchy acceptance function can be obtained by integrating the Cauchy noise distribution:
00

J
0
1 Tdx
; T2 + (x-x;) 2 =
I 1
2 ("')
+;arctan T = AF(i, T)

where x; =!J.CF(z). The cooling schedule and temperature have to be considered in both Cauchy and Gaussian
machines.

I 6.6 Probabilistic Neural Net

The probabilistic neural net is based on the idea of conventional probability theory, such as Bayesian
classification and other estimators for probability density functions, to construct a neural net for dassifi
cation. This net instantly approximates optimal boundaries between categories. It assumes that the training
 Special Networks
238

~< Xl

Hidden
layer 1

Figure 64 Probabilisdc neural nerwork.

data are original representative samples. The probabilistic neural net consists of two hidden layers as shown in
Figure 6-4. The first hidden layer contains a dedicated node for each uaining p:mern :md the second hidden
layer contains a dedicared node for each class. The two hidden layers are connected on a class-by-class basis,
that is, the several examples of rhe class in the first hidden layer are connected only to a single mar..:hing unit
in rhe second hidden layer.
During training process, the probabilistic neural net uses rhe training pauerns for estimating rhe class
probabiliry disuibutions; each new input is classified according to rhe weigh red average of the training sample
which is very closer. The probabilistic neural net avoids the iterative process by simply storing the training
patterns. Owing to this, probabilistic neural ner learns very fast, but large networks are needed for large
data sets.
The algorithm for the construction of the net is as follows:

Step 0: For each training input pattern x(p),p::;:: \ toP, perform Steps 1 and 2.
Step 1: Create p:urern unir zr. (hidden-layer-1 unit). Weight vector for unit Zl is given by
'"I~ .<(p)

Unit Zk is either z-class-1 unit or z-dass-2 unit.

Step 2: Connect the hidden-layer-1 unit to the hidden-layer-2 uniL
If ;..{p) belongs to class l, then connect the hidden layer unir z~. ro rhe hidden layer unit F1.
Otherwise, connect pattern hidden layer unit Zk to the hidden layer unit Fz.

The net can be used for classification when an example of a pattern from each class has been presented
toiL The net's ability for generalization improves when it is trained on more examples.

I
-- 6.7 Cascade Correlation Network
Cascade correlation is a network which builds its own archirecrure as the training progresses. This algorithm was
proposed by Fahlman and Lebiere in 1990. Figure 6-5 shows the cascade correlation architecture. The network
begins with some inputs afld one or more output nodes, bm it has no hidden nodes. Each and every input
 6. 7 Cascade Correlation Network 239

'

~
Hidden node 1 (z!J
<

Input
h

Bias
+1 c ~

Figure 65 Cascade archirecrure after two hidden nodes have been added.

is connected to every output node. There may be linear units or some nonlinear activation function such as
bipolar sigmoidal activation function in the output nodes. During training process, new hidden nodes are
added to the network one by one. For each new hidden node, the correlation magnitude berween the new
node's output and the residual error signal is maximized. The connection is made to each node from each
of rhe network's original inputs and also from every preexisting hidden node. During the time when the
node is being added to the nerwork, the input weights of the hidden nodes are-frozen, and only the output
connections are trained repeatedly. Each new node thus adds a new one-node layer ro the network.
In Figure 6-5, the vertical lines sum all incoming activations. The rectangular boxed connections are frozen
and "0" connections are trained continuously. In the beginning of the training, there are no hidden nodes,
and the netwnrkis trained over the complete training set. Since there is no hidden node, a simple learning rule,
Widrow-Hofflearning rule, is used for training. After a certain number of training cycles, when there is no sig~
' nificant error reduction and the Hnal error obtained is unsatisfactory, we try to reduce the residual errors fun her
by adding a new hidden node. For performing this task, we begin with a candidate node that receives trainable
input connections from the network's external inputs and from all pre-existing hidden nodes. The output of
this candidate node is nor yet connected to the active network. After this, we run several numbers of epochs
for the training set. We adjust the candidate node's input weights after each -epoch to maximize C which is
defined as

c~ L: IL: (vj- V)(Ep- E,) I

' 1

where i is the network output at which error is measured,} the training pattern, u the candidate node's output
value, E0 the residual output error ar node o, V the value of 11 averaged over all patterns, ; the value of Eo

1
240 Special Networks

averaged over all panerns. The value "C" measures the correlation be Ween the candidate node's output value
and the calculated residual output error. For maximizing C, the gradient ac!Ow; is obrained as

ac = "~a; (EJ.i- -E;)~lmJ

/Jw
' 1'.
where u; is the sign of the correlation between the candidate's value and output i; ~ the derivative for pattern j
of the candidate node's acrivarion function with respect to sum of its inputs; lmJ the input the candidate node
receives from node m for pattern j. When gradient aclaw; is calculated, perform gradient ascent to maximize
C. As we are uaining only a single layer of weights, simple delra-learning rule can be applied. When C stops
improving, again a new candidate can be brought in as a node in the active nerwork and its input weights are
frozen. Once again all the outpur weights are trained by the delta learning rule as done previously, and the
whole cycle repeats itself until the error becomes acceptably small.
On the basis of this cascade correlation network, F~lman (1991) proposed another uaining method for
creating a recurrent network called the recurrent cascade correlation network. Irs structure is same as shown
in Figure 65, bur each hidden node is a recurrent node, i.e., each hidden node has a connection to itself.
Cascade correlation network is mainly suitable for classification problems. Even if modified, it can be used
for approximation of functions.

I 6.8 Cognitron Network

Cognitron nerwork was proposed by Fukushima in 1975. The learning hypothesis pur forth by him is given
in the following paragraphs.
The synaptic strength from cell X to cell Y is reinforced if and only if the following rwo conditions are
true:
l. Cell X- presynaptic cell fires.
2. None of the postsynaptic cells present near cell Y fire stronger than Y.
The model developed by Fukushima was called cognitron as a successor to the percepuon which can
perform cognizance of symbols from any alphabet after training. Figure 6-6 shows the connection between
presynaptic cell and postsynaptic cell.
The cognitron net\vork is a self-organizing multilayer neural net\vork. Irs nodes receive input from the
defined areas of the previous layer and also from units within irs own area. The input and output neural
elements can rake the form of positive analog values, which are proportional to the pulse density of firing
biological neurons. The cells in the cogniuon model use a mechanism of shunting inhibition, i.e., a cell is
bound in terms of a maximum and minimum activities and is driven toward these extremities. The area &om
which the cell receives inpur is called connectable area. The area formed by the inhibitory cluster is called
the vicinity area. Figure 6.7 shows the model of a cognirron. Since ilie connectable areas for cells in the same
vicinity are defined to overlap, but are not exactly the same, there will be-a slight difference appearing between
the cells which is reinforced so that the gap becomes more apparent. Like this, each cell is allowed to develop
its own ch;:rracterisrics.
Cogniuon network can be used in neurophysiology and psychology. Since this network closely resembles
the natural characteristics of a biologi~ neuron, this is best suited for various kinds of visual and auditory
information processing systems. However,_amajor drawbac~ of cognitron net is that it cannot deal with the
problems of orientation or disrorrion. To overcqme this drawback, an improved version called neocognitron
was developed.

---
 6.9 Neocognitron Network
241

Synaptic
connections

--~-
Poslsynaplic
cell

@cell
Figure 66 Connection between presynaptiC cell and posrsynaptic cell.

t 0 0

Nodes in
0 r
'7 r./\
l>;v'j
1\IUUeS in
the connectable the vicinity
area area

Figure 67 Model of a cognitron nerwork.

I 6.9 Neocognitron Network

Neocognirron is a multilayer feed~fonvard net\York model for visual pattern recognition. It is a hierarchical net
comprising many layers and there is a localized pattern of connectivity between the layers. It is an extension of
cognitron net\vork. Neocognirron net can be used for recognizing hand~wrinen characters. A neocognitron
model is shown in Figure 68.
The algorithm used in cognitron and neocognitron is same, except that neocognicron model can recognize
panerns that are position-shifted or shapedistoned. The cells used in neocognitron are of t\Yo types:

1. Scdl: Cells that are trained suitably to. respond to only certain features in the previous layer.

g-oo-oo -----.... ... -

Input
layer
.
Modu1e .1 .iviociule
. '
2
DO IVIUUUie n

FigurP 68 Ncocogniuon models.

--
 Spacial Networks
242

~~
c.''
6 UJ
\

0
Outpu i 1
2
3
4
5
6
7
I IB
9

Figure 69 Spreading effect in neocognitron.

2. C-ce!L A C-cell displaces the result of an S-cell in space, i.e., son of "spreads" the features recognized by
the S-cell.
Neocognitron net consists of many modules with the layered arrangementofS-cdls and C-edis. The S-cells
receive the input from the previous layer, while Ccells receive the input from the S-layer. During training,
only the inputs to the S-layer are modif1ed. The S-layer helps in rhe detection of spccif1c features and their
complexities. The feawre recognized in the 5 1 layer may be a horizomal bar or a venical bar but the feature in
the 5, layer may be more complex. Each unit in rhe C-layer corresponds to one relative position independent
1
feamre. For the independent feature, Cnode receives rhe inputs from a subset of S-layer nodes. For instance,
if one node in C-layer detects a vertical line and if four nodes in the preceding S-layer detect a verricalline,
then these four nodes will give the input to the specific node inC-layer to spatially distribute the extracted
features. Modules present near the input layer (lower in hierarchy) will be trained before the modules that are
higher in hierarchy, i.e., module 1 will be trained before module 2 and so on.
The users have to fix the "receptive field" of each C-node before training starts because the inputs to C-
node cannot be modified. The lower level modules have smaller receptive fields while the higher level modules
indicate complex independent features present in the hidden layer. The spreading effect used in neocognitron
is shown in Figure 6-9.
The S-layers are trained to respond to a particular panern or group of patterns. The C-arrays then combine
the results &om related S-arrays ~d correspondingly thin out the number of units in each array. Training
is found to progress layer by layer. The weights from the input units to the first layer are firSt trained and
then frozen. Then the next trainable weights are adjusted and so on. When the net is designed, the weights
between some layers are fixed as they are connection parterns.

I
-- 6.10 Cellular Neural Network
The cellular neural nerwork (CNN), introduced in 1988, is based on cellular auromata, i.e., every cell in the
network is connected only to its neighbor cells. Figures 6-1 O(A) and (B) show 2 x 2 CNN and 3 x 3 CNN,
 6.12 Spatia-Temporal Connec\ionisl Neural Network 243

(A) (B)

Figure 610 (A) A 2 x 2 CNN; (B), 3 x 3 CNN.

respectively. The basic unit of a CNN is a cell. In Figures 6-IO(A) and (B), C(l, l) and C(2, 1) are called
as cells.
Even if the cells are not direccly connected with each other, they affect each other indireccly due ro
propagation effects of the necwork dynamics. The CNN can be implemented by means of a hardware
model. This is achieved by replacing each cell with linear capacirors and resistors, linear and nonlinear
controlled sources, and independent sources. An electronic circuit model can be constructed for a CNN. The
CNNs are used in a wide variery of applications including image processing, pattern recognition and array
computers.

I 6.11 Logicon Projection Network Model

Logicon projection network model (LPNM) is a learning process developed by researchers at Logicon. This
model combines supervised and unsupervised training during the learning phases. When the unsupervised
learning is used, the network learns quickly but nor accurately. On the other hand, with supervised learning,
it learns slowly bm the error is minimized. The learning phase uses a feed-forward nccwork with a hidden
layer in between input and outpur layers. At the beginning of the learning phase, an unsupervised method
such as Kohonen or ART is used ro quickly initialize the weights of the network to some gross values, and
then a supervised method like BPN may be used to finerune weight values. As the supervised method starts
from "almost acceptable" solution, the network is claimed to converge quickly to a global minimum. Logicon
claims that LPNM method is best than other methods. This network does not have to be reinitialized if
more knowledge is to be added. Also, a network with some knowledge can be added ro another nerwork with
different knowledge ro obtain rhe sum of both.

I 6.12 Spatio-Temporal Connectionist Neural Network

Spatia-temporal connectionist neural network (STCNN) characterizes connectionist approaches for learning
input-output relationships in which the dara is distributed across space and time-spatio-temporal patterns.
An STCNN is defined as a parallel disuibuted information processing strucrure which is capable of dealing
with input data presenn!d across both time and space. In STCNN, input and output patterns vary across time
as well as space. For analyzing the network's performance, it is useful to discretize the temporal dimension by
sampling at regular intervals. The system considered here produces the response when the time proceeds by
intervals of b..t. Symbol "t" may be used to re~resent a panicular point in rim. Here b..t tan be considered

.J
 ~1-H'GIOI Networks
244

as the unit measure for quantity tor some'small variations in t. In STCNN, even a ~.:onrinuous lime system
is converted into a set of firstorder difference equations, making it ro be in the form of discrete time
systems.
The time dimension in STCNN differs from ilie spatial dimensions in conventional connectionist net-
works. Components of an input pattern diStributed across space can be accessed at the same time. However,
only the current components of patterns distributed along the time are accessible at any given instant. The
input vector for an STCNN at instant tis denoted by input vector X{t). This vector is supplied to STCNN
at cime t by setting the activation values of the input units of the STCNN m ilie components of the vector.
Hence, input vector can be considered as a stimulus.
The conventional and spatia-temporal networks are equipped with memory in the form of connection
weights denoted by one or more matrices depending on the number of layers of connections present in
the network. These are updated after each training step and constitute a memory of all previous training.
Assuming this memory exrends back past the current input pattern, all ilie way to the first training step, we
refer to ilie weights as long-term memory. After a connectionist network has been successfully trained, this
long-term memory remains fixed during the operation of the network.
Along with these weight matrices, some networks a1so use other trainable parameters. These parameters
may represent either of the three mentioned below:

1. connectivity scheme of the network;

2. rypes of transmission delays associated with connections;
3. initial activation values of the internal processing elements.
These parameters also form a part of the long-term memory. We define" \\7" to denote an n-ruple representing
all the adaptable parameters of rhe nerwork. This 11-ruple includes one or more weight matrices, an_9.-aJso may
contain connectivity scheme parameters, transmission delay parameters and initial activation values depending
on the type of the ne[\vork.
The STCNNs a1so include a shorr-term memory. This memory allows these networks to deal with input
and output patterns that vary across rime and thus defiiles them as STCNNs. Conventional connectionist
networks compute the activation values of all the nodes at time t based only on the input at rime t. On the
other hand, in STCNNs the activations of some nodes at timet are computed on the basis of the activations
at time (t- l) or earlier. These activations serve as shorr-term memory. The state vector 5(t- 1) is used here
to represent the activations at rime (t - 1) of those nodes that are used to compute the activations of orher
nodes at a rime t, i.e., state nodes. The long-term memory is stored in connection weights (which are updated
only during training) while the short-term memory is represented by node activations (which are computed
with each time step even after training).
STCNNs encode their output responses in the activations of a special set of units called output units.
The output of STCNN is represented by vector J(t). Mosr connectionist nemorks learn by computing the
difference between their response and desired (ideal) response and adjuscing their long-term memory suitably.
The desired response is denoted _by ]J(t). The difference between the desired output vector and the actual
output vector is the error vector E(t) = ]J(t)- j(t), and the total network error,, is defined as the one-ha1f
of the square of the magnitude of this vector, i.e.,

e= L H\\(r)II'J
r::-_0

The tota1 error, given bye, is the measure of overall performance. It is this quancity that is minimized via gradi
em descent during training. STCNNs are applicable to dynamical system identification and control, syntactic

L
6.13 Optical Neural Networks 245

pattern recognition and grammatical induction. There are various taXonomies that are being developed for
STCNNs.

I 6.13 Optical Neural Networks

Optical neural networks interconnect neuronswiih light beams. Owing to this imerconnection, no insulation
is required between signal paths and the light rays can pass through each other without interacting. The path
of the signal travels in three dimensions. The .transmission path density is limited by the spacing of light
sources, the di.vergence effect and the spacing, of detectors. A$ a result, all signal paths operate simultaneously,
and true data rare results are produced. In holograms with high density, the weighted strengths are stored.
These stored weights can be modified during training for producing a fully adaptive system. There are two
classes of this optical neural necwork. They are:

1. electro-optical multipliers;
2. holographic correlators.

I 6.13.1 Electro-Optical Multipliers

Elecrrooptical multipliers, also called electro-optical matrix multipliers, perform mauix multiplication in
parallel. The network speed is limited only by the available electro-optical components; here the computa-
tion time is potentially in the nanosecond range. A model of electrooptical matrix multiplier is shown in
Figure 6-11.
Figure 6-11 shows a system which can multiply a nine-element input vector by a 9 X 7 marrix, which
produces a seven-element NET vector. There exists a column of light sources that passes its rays through
a lens; each light illuminates a single row of weight shield. The weight shield is a photographic film where
transmittance of each square (as shown in Figure 6-11) is proportional to the weight. There is another

Lens
j
/

' ,w.
.
w17/ /
- - Light lor the
/
suspective
/ columns
'
'
/
/
'
'
suspective /
rows

/
/
/
/

"
"' w..,
'
/
' ,,
'
Figure 611 Electrooprical multiplier.
246 Special Networks

lens that focuses the light from each column of the shield m a corresponding phoroelc!cror.-The NET is
calculated as

NETk = I: W;kXi

where NET k is the net output of neuron k; w;lt the weight from neuron i to neuron k; x; the input vector
component i. The output of each photodetector represents the dot product between the input vector and a
column of the weight matrix. The output vector set is equal to the produce of the input vector with weight
matrix. Hence, matrix multiplicacion is performed parallely. The speed is independent of the siie of the array.
So, the network is sealed up without increasing the cime required for computation. Variable weights may be
designed for use in the adapcive system. A liquid crystal light valve instead of photographic film may be used
for weights. This makes the weights to get adjusted electronically. This type of electro~optical multiplier can
be used in Hopfield net and bidirectional associative memory.

I 6.13.2 Holographic Correlators

In holographic correlators, the reference images are stored in a thin hologram and are retrieved in a coherencly
illuminated feedback loop. The input signal, either noisy or incomplete, may be applied ro the system and can
simultaneously be correlated optically with all the srored reference images. These. correlations can be threshold
and are fed back to the input, where the strongest correlation reinforces the input image. The enhanCed image
passes around the loop repeatedly, which approaches the stored image more closely on each pass, until the
system gets stabilized on the desired image. The best performance of optical correlators is obtained when they
are used for image recognition. A generalized optical image recognition system with holograms is shown in
Figure 6~ 12.
The system input is an image from a laser beam. This passes through a beam splitter, which sends it to
the threshold device. The image is reflected, then gets reflected from the threshold device, passes back to the
beam splitter, then goes to lens 1, which makes it fall on the first hologram. There are several stored images

Mirror A

Figure 612 Oprical image recognition system.

'
 6.14 Neuroprocessor Chips 247

in fim hologram. The image then gers correlated with each stored image. This correlation produces light
patterns. The brighmess of the patterns varies with the degree of correlation. The projected images from lens
2 and mirror A pass through pinhole array, where they are s'patially separated. From iliis array, light panerns
go to mirror B through lens 3 and then are applied to the second hologram. Lens 4 and mirror C then produce
1
superposition of the multiple correlated images onto the back side of the threshold device.
The front surface of the threshold device reflectS I_nOSt strongly that pattern which is brightest on its rear
surface. Its rear surface has projected on it the set of four correlations of each of the four stored images with the
input image. The stored image that is similar to the input image possesses highest correlation. This reflected
image again passes through the beam splitter and re~enters the loop for further enhancemenr. The system gets
converged on the stored patterns most like the input pattern.
Here we have discussed the basic operaciOn of the holographic optical image recognition system. Employing
hologram correlaror, we can design Hopfield network. Optical neural networks are more advantageous in terms
of speed and interconnect densicy. Theycan virtually construct any network architecture.

I 6.14 Neuroprocessor Chips

Neural networks implemented in hardware can take advantage of their inherent parallelism and run orders of
magnitude faster than software simulations. There exists a wide variecy of commercial neural network chips
and neurocomputers.
The probabilistic RAM, pRAM~256 Very Large Scale Integrated (VLSI) neural nerwork processor, was
developed by the Electrical Engineering Department ofKing's College, London. pRAM has 256 reconfigurable
neurons, each with six inputs. Irs on~chip learning unit utilizes reinforced learning where learning can be global,
local or competitive. The external static RAM of pRAM stores the synaptic weights. The p~RAM possesses
both stochastic and nonlinear aspects of biological neurons in a cypical manner, which allows exploitation of
hardware.
The Neuro Accelerator Chip (NAC) was developed in 1992 by the Information Defence Division, Aus~
tralian Defence Science and Technology Organization. his made up of an array of 16~componem IO~bit
inreger processing elements that can be cascaded in two dimensions with necessary comrol signals. Each
processing element multiplies irs input by one of 16 weights preloaded in dual port registers and accumulates
the results to 23~bit precision at a rate of 500 million operations per second. The NAC can be hard wired to
implement various neural networks.
Neural Network Processor (NNP), developed by Accurate Auromation Corporation, uses a multiple
instruction multiple data architecture capable of running multiple chips in parallel without performance
degradation. Each chip houses high~speed 16~bit processor with on~chip storage for synaptic weights. Only
nine assembly language instructions are executed by the processor. Communication among multiple NNPs
is performed by inrerprocessor. NNP can be programmed to implement any particular neural nel:\vork train~
ing algorithms. Irs performance is 140 MCPS for a single chip and up to 1.4 GCPS for a lO~processor
system.
The CNAPS system, developed by Adaptive Solutions, is mainly based on CNAPS~l064 digital paral~
lei processor chip that has 64 sub~processors operating in SIMD mode. Each sub~processor can emulate
one or more neurons, and multiple chips can be ganged together. The CNAPS/PC lSA card uses 1. 2
or 4 of new CNAPS~l016 parallel processor chips or two of the 1064 chips to obtain 16, 32, 64 or
128 CNAPS processors. Learning algorithms can be programmed. It can be noted that back propagation
and several other algorithms come in the Build Net package. Here, back propagation feed~forward per~
forms 1.16 billion multiply/accumulates/second and 293 million weight updates/second with 1 chip and

:rL
248 Special Networks

5.80 1. t"6 billion multiply/accumulates/second /1.95 million weight updates!second, respectively, with four
chips.
The IBM ZISC 036 is a d!gital chip with 64-component Bbit inputs and 36 radial basis function neu
rons. Multiple chips can be easily cascaded to create networks of arbitrary size. Here input vector V is
compared to store prototype vector" for each neuron. It takes 3.? J.LS to load 64 elements and another
0.5 J.Ls for the classification signal to appear. Learning processing of a v.ector takes about another 2 J.LS beyond
4 JLs for loading and evaluation. Its performance at 16 MHz. 4 J.LS classification of a 64componem Sbir
vecror.
The INTEL 80170NX Electrically Trainable-Analog Neural Network (ETANN) is one with 64 inputs
(D3v), 16 imernal biases, and 64 neurons with sigmoidal transfer functions. Two layer feed-forward necworks
can be implemented wirh 64 inputs, 64 hidden neurons, and 64 output neurons using the two SO X 64
weight matrices. Hidden layer omputs are docked back through second weight matrix to perform output
layer processing. Instead of this, a single 64-layer network with 12S inputs can be implemented using both
matrices and clocking in two sets of 64 inputs. Weights possess 6-bit precision and are stored in nonvolatile
floating gate synapses. There is no on-chip learning. Emulation is performed in software and the weights are
downloaded to rhe chip. In this case, about S ~propagation time is taken for a two-layer network. This is
equivalent to roughly 2 billion multiply/accumulates per second.
MCE MT 19003 Neural Instruction Set Processor is a digital processor chip using signed 12-bit internal
neuron values, with 16-bit multiplier and 35-bit accumulator. Network inputvalues, bias values, synapse
and neuron values are held in off-chip memory. The network processing is also guided by a given program
in off-chip memory using seven-element inmuction set. Neuron values can be sealed by a transfer function
using four available tables. This processor also has no on-chip learning. Its performance is 1 synapse per clock
cycle.
RC Module Neuro Processor NM6403 is a high-performance microprocessor with super scalar architecture.
The architecture includes comrol unit, address calculation and scalar processing units, node to support
vector operations with elements of variable bit lengrh. There is no on-chip learning in this processor. Its
performance is

1. Scalar opmuiom: SO MIPS, 50 MIPS for 32 bit data.

2. Vector opemriom: 1.2 billion multiplications and additions/second.

Nestor NllOOO is a nerwork wid1 Radial Basis function neurons. During its learning, prototype vectors
are stored under the assumption that they are picked randomly from original parent distributions. Here up to
I024 prototypes can be stored. Each prototype is then assigned to a given middle layer neuron. This middle
layer neuron is assigned to an output neuron that represents the particular class for that vecror. Ali middle
layer neurons that correspond to rhe same class are designed to same output neuron. In recall stage, an input
vector is compared to each prototype parallely, and if the distance berween them is above a given threshold,
it fires, leading to firing of the corresponding output, or class, neuron. Here two on-chip learning algorithms
ue available:

1. Probabilistic neural net (PNN);

2. Restricted Coulomb energy (RCE).

Also, microcoding can be modified for user-defined algorithms. Its performance is 40K, 256 element
patterns per second. There is Narional Semiconductors NeuFwlCOPS Microcontroller processor, which
uses a combination of neural network and fuzzy logic software to generate code for National's COPS
6.16 Review Questions
249

microcontrollers. Neural network can be used to learn the fuzzy based rules and membership functions.
There exist several packages of it. Some are listed below:

1. NroFuz Learning Kit (NF2- C8A- Kit): A neural network PC/AT software (2 inputs, 1 output) and
fuzzy rule and membership function generator (nlax 3 membership functions), COPS code generator and
COPS assembler/linker.
2. NeuFuz 4 (NF2- CBA), Neurn.l network PCIAT.sofrware (4 inputs, 1 ourpur) and fi=y rule and
membership function generator (max 3 member functions), COPS code generator and COPS assm/linker.
3. Neu.Fuz 4 DeveWpment System (NF2- CBA}: Neural network PC/AT software (4 inputs, 1 omput) and
fuzzy rule membership ftmcrion generators (max 3 member functions), COPS code generator, COPS
assembler/linker and COPS in-circuit emufaror with PROM programming.

Learning performed here is only software learning. Apart from the above lisred chips, there are several
other neuroprocessor chips. Besides, a wide variety of research is going on for further devdopment of neural
network hardware.

16.15 Summary

In this chapter, we have discussed certain specific networks based on their special characteristics and per-
formance. The nerworks are designed for optimization problems and classifications. Cerrain nets discussed
use Bayesian decision making method and hierarchical arrangement of units. The variations of Boltzmann
machine, which include Gaussian and Cauchy nets were also discussed. Besides, our discussion focused
on the cognirron and neocognitron networks, which are used for recognition of hand wrirren characters.
Other networks discussed include spatia-temporal neural nernrork, annealing nerwork, optical neural nets,
cellular neural nets, and l.ogicon neural ners. To give the reader an idea of neural network hardware, a fC"N
neuroprocessor chips have also been listed.

I 6.16 Review Questions

1. List a few special neural networks designed for S. Discuss the algorithm used in probabilistic
rypical applications. neural network.
2. What is the principle behind simulated anneal- 9. How does cascade correlation network build its
ing network? network as the training progress?
3. How is Bolumann machine used in constrained 10. Justify that cascade correlation network is a
optimization problems? hierarchical network.
4. With a neat architectural diagram explain 11. Compare and contrast presynaptic and postsy-
the application procedure used in Bolumann naptic cells in cogniuon model.
machine.
12. Explain the working principle of cognirron
5. Write short note on Gaussian and Cauchy network.
machines.
13. What is the drawback of cogniuon net?
6. What is the importance of probabilistic neural
network? 14. Write short nore on neocogniuon model, staring
how does it overcome the drawback of cogniuon
7. With an architectural diagram, explain the prob- model.
abilistic neural network.
 Special Networks
250
18. State the prirtciple of optical neural neMorks.
15. Describe the working methodology in cellu1ar
neural network. 19. Briefly explain the concept involved in
electro-multiplier networks and holographic
16. In whar way are the supervised and unsuper
vised learning methods combined to obtain high correlators.
performance in Logicon projection network? 20. Mention a few latest neuroprocessor chips.
17. Discuss in derail the spatia-temporal connec-
tionist neural net"Hork.

.~

~
0
,'-'
.
.'l --- \./1
\

~ .~~\~
'j ~ r::;
~ _, '':
" s.:?-,..,;
<:._, \"""' .......
<::;
........-:: - <:\)
\~~

'
~-
,, \
!-..
'-".I'-
... ~ .....~
..... ~u~

.J (
.. . _

'
q
:0
-
~
 Introduction to Fuzzy Logic,
Classical Sets and Fuzzy Sets 7
Learning Objectives - - - - ' - ' - - - - - - - - - - - - - - - ,
Definition of classical sets and fuzzy sets. Solved problems performing the operations
The various operations and properties of and properties of fuzzy sers.
classical and fuzzy sets.
How functional mapping of crisp set can be
carried our.

I 7.1 Introduction to Fuzzy Logic

In general, rhe entire real world is complex, and the complexiry arises &om uncermimy in the form of
a~gu_!!x. One should closely look into the real~world complex prohlem,s to find an accurate solution, amidst
die existing uncertaimies, using certain med\odologics. Henceforth, the growth of fuzzy logic approach, to
handle ambiguiry and uncenainry exisring in the complex problems. In general, fuzzy logic is a form of
multi~valued !QJ.C. ro deal wnh reasu~Iing tat is a&wximate r~ther chan precise. This is in conrradiccion
with (!crisp lo ic" that deals with precise va ues. Also, bmary sers have binary or Boolean logic (either 0 or 1),
which nds solution to a parucu ar sec of problems. Fuzzy logic variables may have a trmh value that ranges
benveen 0 and 1 and is not consrrained to the rwo rrurh values of classic propositional logic. Also, as linguistic
variables are used in fuzzy logic, these degrees have ro be managed by spec1hc fllncCions .

As the complexity ofa system increases, it becomes more difficult and evmtually impossible to make a precise
statement about irs behavior, eventual '.J.In.~'u.in tu a poim ofcomplexity where the f..tZZJ' logic method born in
humans is the on! a bl.em-
nginally identified and sec for y Locfi A. Zadeh, Ph.D., University of California, Berkeley)

Fuzzy logic, introduced in t ear 1965 by LotfiA. Zadeh, is a mathemacica1 roo! fordealingwich uncerrainry.
Dr. Zadeh states chat rh Pnnc1p e o camp exiry and imprecision are corre ate : "The closer one looks
at a real world problem, ilie fullier ecomes Its so unon. uzzy og1c offers soft computing paradigm
the imporranr concept of compu~ords. It provides a technique to deal with imprecision and
informacion granularity. The fuzzy theory provides a mechanism for representing linguistic constructs such as
"high," "low," "medium," "tall," "many." In general, fuzzy logic provides an inference structure that enables
appropriate human reasoning capabilities. On the contrary, the traditional binary set theory describes crisp
events, chat is, events that either do or do not occur. It uses probability theory ro explain if an event will
occur, measuring rhe chance with which a iven eve red to occur. The dteory of ILliiY log_~c JS based
upon the nonon o re a 1ve gra e mem ership d so are the fu~of cognirive processes. The utility

._
 252 Introduction to Fuzzy Logic, Classical Sets and Fuzzy Sets

Imprecise and vague data De!JISions

Fuzzy Logic System
.'\

Figure 7-1 A fuzzy logic system accepting imprecise data and providing a decision.

0.
"
f
!'

of fuzzy sets lies m their ability to model uncen:ain or ambiguous data and ro provtde s table decisions as i'Q;~
Ftgure 7-1. _,Q ..... .? "\<2)
":!
Though fuzzy logtc has been applied to many fields, fromilieory to intelligence, it still
remains controversial among most statisticians, who prefer B;yesian logic, and some control engmeers, who
prefer traditional two-valued logic. In fuzzy systems, values are indicated by a number (called a truth value)
ranging from 0 to l, where 0.0 represents absolute falseness and 1.0 re resents absolute truth. While this
range evokes the idea of probability, fuzzy logic an sets o erate quite differen ro a 1 1
Fuzzy sers that represent fi.uzy logic provide means to model the unce associated with va-- eness,
imprecision and lack of information regarding a problem or a p ant or a system, etc. Consider the meaning
of a "short person". For an individual X, a short person may be one whose heigbt is below 4' 25". For other
individual Y, a short person may be one whose height is below or equal to 3'90". The word "short" iS called a
inguisric escn tor. he term "short" provides the same meaning to individuals X and Y, bm it can be seen
at ey oth do not provide a unique definition. The term "short" would be conv edeffectivei on! n
a computer compares th the re-assigned value o s orr". This variable "short"~
called as ingwsnc vana le which represents the imprecision existing in e syscem.
' The basis of ffie Uito:ty hes in making the memhfrship fimccion lie oYer a range of real numbers from 0.0
ro 1.0. The fuzzy set is characterized by (0.0,0,1.0). Real world is vague and assigning rigid values m liiigrrisctC
\~es means that some of the meaning and semantic value is invariably lost. The uncerrainrv is found m
arise from ignorance. from chance and randomness, due to lack of know led
., the fuzziness existing in our narurallanguage. Dr. Zadeh proposed thcVet membmht~ea to make suitable
decisions when uncenaimy occurs. Consider rhe "short" example discussed previously. f we rake "shon" as
a height equal to or less than 4 feet, then 3'90" would easily become rhe member of the set "short'' and 4'25"
will not be a member of the set "shorr." The membership value is "1" if it belongs to the set and "0" if it
is nor a member of the set. Thus membership in a set is found m be binary, that is, either the demem is a 1:,
member of a set or not. It can be indicated as c

'
XA (x) ~
I I'
0,
xEA
x~A ,, r,\:
where XA (x) is the membership of element x in the seifA and A is the ennrc set on ilie lll]jVetst.J
If it is said that rhe height is 5'6" (or i68 em), one might rhink. a bit before deciding wherhcr ro consider
it as short or not shan (i.e., rail). Moreover, one might reckon it as short for a man bur rail for a woman. Ld~
make rhe statement "John is short", and give it a truth value of0.70. lf0.70 represented a probability value, ir _
would be read as "There is a 70% chance that John is short," meaning that it is still believed thar John is either V
short or not short, and there exists 70% ance o owm which group he belongs to. Bur fuzzy terminology
acrually uanslates t ohn's degree o mem ers "p m e set o s on p , by which it is meant .~--- ---.i.j'
that if all the (fuzzy sec of) short people are consi ere and lined up, John is posmon Oo/o of the way to the~ ~
s~In conversation, it is generally said that John is "kind of" shan and recognize rha~ there ts ~e /
demarcation between short and tall. This could be stated mathematically as p.SHORT(Russell) = 0.70,
where JL is the membership function.

l
 7.1 Introduction to Fuzzy Logic
253

Tall

Membership

0.5

150 180 210

Height (em)

Figure 72 Graph showing membership functions for fuzzy set "tall."

Fuzzy logic operates on the concept of membership. For example, the statement "Elizabeth is old" can
be translated as Elizabeth is a member of the set of old people and can be written symbolically as J-L(OLD),
where JL is the membership function that can return a value between 0.0 and 0.1 depending on the degree of
membership. In Figure 7-2, the objective term "rail" has been assigned fuzzy values. At 150 em and below, a
person does nor bclo.ng to the fuzzy class while for above 180, the person certainly belong!i to category "tall."
However, between 150 and 180 em, the degree of membership for the class "tall" can be assigned from the
curve nglinearly between 0 an The fuzzy concept "tallness" can be extended into "short," "medium"
ass own m igure 7-3. This is different from rhe pmhahjljcy approach rhar gives rhe ds;gree.of
probabilio/ of an OC'1IPGO-ef.an-eveM-(~g..QJ,_Q,j,Q.~eef.
The membership was extended to possess various "degrees of membership" on the real continuous interval
[0, l]. Zadeh formed fii.ZZJ sets as the sets on the universe X which can accommodate "degreq_Q(membership."
The concept of a fuzzy se~ith the classical concept of a bivalent set (crisp set) whose boundary is
required to be precise, that is, a crisp set is a collection of things for which it is known irrespective of whether
any given rhing is inside it or not. Zadeh generalized ilie idea of a crisp set by extending a valuation set {1 ,0}
(definitely in/definitely out) to the interval of real values (degrees of membership) between 1 and 0, denoted
as [0, l]. We can say that the degree of membership of any particular element of a fuzzy ser expresses the degree
of compatibility of the element with a concept represented by fuzzy set. It y set A contains
an object x to degree a(x), that is, a(x) = Degree(x E A), and the rna :X-)- !Membership" egr~ is called
a set fimction or j_"f!lembership fimct~~rz- The fuzz.y serA can be expressedasA = {(x, a(x))}, x EX; ir imposes an
elastic cori5t"r.iill$f the possible values of elements x EX, called the possibility diStribution. Fuzzy sets rend to
-- ------.; :=:..---

:?~ , Shon Medium Tall

e':---.fi
~' Membership

/1 0.5
c\
CS""
150 180 210
Height (em)

Figure 73 Graph showing membership functions for fuzzy sets ''short," "medium" and "rail."
 254 Introduction to Fuzzy Logic, Classical Sets and Fuzzy Sets

X -universe ol discourse

p
-<
b ~

f! if

~
I,

'f
iu c
;?J
-,J ~~"'- ' 't

v
"~
'-\ "' 0
"\1'-. ~
;;; ''R'
9'
_i

~~""
/'
<;:

{
'
'1--< }r ' Figure 74 Boundary region of a fuu.y ser.
'
capture vagueness exclusively via membership functions that are mappings from a given universe of discourse
X ro a unit interval containing membership values. It is important to note that membership can take values
between 0 and 1.
FUZ2iness describes the@mbtguiry ofan event andjrandomnes:s describes dle uncertainty in m-;bccurrence of
. ~t can be generally seen m claSSical sets that cllere JS no i.incerraimy, hence they have crisp boundaries,
Om in dte case of a fuzzy set, since uncerrainry occurs, the boundaries may be ambiguously specified.
From Figure 7-4 it can be noted that "a" is clearly a member of fuzzy set P, "c" is clearly not a member
of fuzzy set Pand the membership of"b" is found to be vague. Hence "a" can take membership value 1, "c"
can take membership value 0 and "b" can take membership value between 0 and 1 [0 to 1], say 0.4, 0.7, etc.
T_his is said to be a partial membership of fuzzy set P.
\_The members i funcuon For a set rna s ch elemem of the set to a membershi value between 0
~ um ue escribes t set. The l/a ues an escn e nor elonging to" and e ongmg to a
conventiOn set, respectively; values in betW.eeruepment "fuzziness." Determining the membership function
is subjective to varying degrees depending on the simation. It depends on an individual's perception of the
data in question and does not depend on randomness. T~ncept is important and distinguishes fu~y ser_
t~eo from robability theory. ~--- ~{,:L__ ~}. ______ _
Fuzzy logtc so conststs o '\fuzzy infererice eng~e or [fuiif'"~~l~-~--;m perform approximate reasoning
somewhat similar ro (but mud\ m~rimicive--tf'{a'n) that'ofr~rain. Computing with words seems
m be a slightly futuristic phrase today since only certain aspects of natural language can be represented by
the calculus of fuzzy sets; still fuzzy logic remains one of the mosr practical ways to mimic human expertiSe
in-;, reahsuc manner.' I he fuzzy approach uses a premise that humans don't represent classes of objects
(e.g. "class of bald men" or the "class of numbers which are much greater than 50") as fully disjoint sets but
rather as sets in which there may be grades of membership intermediate between full membership and non-
membership. Thus, a fuzzy set works as a concept that makes it possible to treat fUzziness in a quanritative
manner.
F~rs fnrm ~Re 81:iil8ift!j Blaeh rn 6tny ff=THEN rules which have the general form "IF X is A
THEN Y is B, "where A and B are fuzzy sets. The term "fuzzy systems" refers moscly to systems that are
governed by fuzzy IF-THEN rules. The IF part of an implication is called rhe antecedent whereas the THEN
pan is called a com;;;;ent. A fi.ru:y system is a set of fuzzy rules that conv "ii1oim- to outputs. The basic
configuration of a pure fuzzy system ISs own m igure 7-5. The fuzzy inference engine gon mbines
fuzzy IF-THEN rules into a mapping from fu:z.zy sets in the input space X to fuzzy sets in the output space
 7.2 Classical Sets {Crisp Sets) 255
-
-(
<J
~

,.,
~~sefsi";J-LI ___~____ _J

J/ Figure 7-5 Configuration of a pure fuz.z.y system.

~
A

"
Ybased on fuzzy logic principles. From a knowledge representation viewpoint, a fuzzy IF-THEN rule is a
scheme for capturing knowledge that involves imprecision. The main fearure of reasoning using these rules is
its partial matchingcapabiliry, which enables an inference robe made from a fuzzy rule even when the rule's
condidon is only panially satisfied.
Fuzzy systems, on one hand, are rule-based systems that are constructed from a collection of linguistic
rules; on the other hand, fuzzysysrems,ar~ear mappings ofif!_BUts (stimuli) ro outp~,,~esJ;-th:at
ile;rrain types of fuzzy systems can be wri'rteti as aimpact nonlinear for;.mUlas. The inputs and outputs can
be numbers or vectors of numbers. T~ese rule-based systems can in theory model any system with arbitrary
accuracy, that is, they work as universal approximators. ,
Tl'ie Achilles' heel of a hlzztsystem is us rules; _sfflart rules give s~art sy_stems an~ other rules give less
sman or even dumb systems. the number of rnles tncreases exponennally wuh the d1mension of the input
space (number of system variables)\ Tfiis rUle explosion 1s Cillcd the curse o}dime1JSt01ta!ay and 1s a gene@
problem fur mathematicrl modEls~ tor the last 5 years several approaches based on decomposition, (duster)
merging and fusing have been proposed to overcome this problem.
Hence, fuzzy models are nor replacements for probability models. The fuzzy models are sometimes found
to work better and sometimes they do not. Bur mostly fuzzy logic has evidendy proved that it provides better
solutions for complex problems. . r '"ri'\'
"~~: \.'
Jr''
~ c\ ~- . 1
o'
I c ,
(>(
I 7.2 Classical sets (Crisp Sets)
(._.-! l
u-i
Basically, a set is defined as a collection of objects, which share certain characteristics. A classical ser is a
collection,..ofdisrioq objects. For example, the user may define a classical set of negative integers, a set of
persons with height less than 6 feet, and a set of students with passing grades. Each individual enriry in a ser
is called a member or an element of the set. The classical set is defined in such a way--mai
[he universe of
disc~embers and nonmembers. Consider an object x in a crisp set A. This
object xis either a member or a nonmember of rhe given serA. In case of crisp sets, no partial membership
exists. A crisp set is defined by its characteristic function. ---.._
Let universe of discourse be U. The collection of elements in the universe is called whole ser. The total
number.ofe\ements in universe U is called@nu~ber- denoted bfnv:7Col\ections of elements within
a universe are called sets, and collections of elements within a set are called subsets. ---.....___,_.
W;-know tllat for a crisp set A in universe U: ---

1. An object x is a member of given set A (x E A), i.e., x belongs to A.

2. An object xis not a member of given setA (x (/;A), i.e., x does not belong to A.

l
256 ln_!roduclion to Fuzzy Logic, Classical Sets and Fuzzy Sets

There are several ways for defining a set. A set may be defined using one of rhe following:

1. The list of all the members of a set may be given. Example ,0'
A= [2,4,6,8, 10} ~~'

2. The propenies of the set elements may be specified. Example

'.,,J
Q/
A = {xl.r is prime number< 20}
0 o/'
.(>~'
3. The formula for the definition of a set may be mentioned. Example ,i" :\
~
-t'. \...\j
A=
1
~~-x;_=_x_;7;::_,_;=~~1~'-o_lo_,_w_here 1) ~ / Q~~/1
Xi=

4. The set may be defined on the basis of the results of a logical operation. Example \) ~(.:
A = (xlx is an element belonging ro P AND Q} ~
.fv
--=--=---
5. There exists a membership function, which may also be used to define a set. The membership is denoted
by the letter J.L and the membership function for a set A is given by (for all values of x)

. (1
I'A(x)= 0 ifx~A
ifxEA

The set with no demems is defined as an empcy set or null ser. Iris denoted by symbol. The occurrence
of an impossible event is denoted by a null set, and the occurrence of a cenain event indicates a whole ser.
Th_e ~-Glnsi.sts-af.all._e~~~ble su_b~_ers o_f_~er A_~~~~~~~- p_o~v~~-s_e_~~nd is_~~~~:? as

P(A) = [xjx <;A}

For crisp sets A and B containing some elements in universe X. rhe notations used are given below:

x E A::::} xbelongs roA

x (/; A => x does not belong to A
x E X::::} x belongs to universe X

For classical sets A and Bon X, we also have some nmarions:

A C B =>A is completely contained in B (i.e., if x E A, chen x E B)

A ~ B::::} A is contained in or is equivalent to B
A=B=>A~BandB~A

I 7.2. 1 Operations on Classical Sets

Classical sets can be manipulated through numerous operations such as union, intersection, complemem and
difference. All these operations are defined and explained in the following sections.

1
 7.2 Classical Sets (Crisp Sets) 257

~
Figure 7~6 Union of lWO sets.

7.2. 1.1 Union

The union between rwo sets gives all those.elements in the universe that belong to either set A or set B or
both sets A and B. The union operation can be termed as a logical OR operation. The union of two sets A
and B is given as

AUB= [xjxEAmxE B}

The union of sets A and B is illustrated by the Venn diagram shown in Figure 7-6.

7.2. 1.2 Intersection

The imersection between two sets represents all those elements in the universe that simulraneously belong to
both the sets. The intersection operation can be termed as a logical AND operation. The intersection of sets
A and B is given by

A nB = [x[x E A andx E B}

The intersection of sets A and B is represented by the Venn diagram shown in Figure 7-7.

7.2.1 .3 Complement
The complement of set A is defined as the collection of al!_ele~rs in niverse Xrhat do nor reside in setA,
i.e., the entities that do not belong to A. It is denoted by A and is defined as

A= [xlx ~ A,x EX} .rt.P

where Xis rhe universal set and A is a given set formed from un2rse X. The complement operation of set A
is shown in Figure 7-8.

[00]
Figure 77 Intersection of lWO sets.

~
Figure 7.8 Complement of set A.

1
258 1nlroduclion to Fuzzy Logic, Classical Sets and Fuzzy Sets

~~ (A) (B)

Figure 79 (A) Difference AlB or (A- 8); (B) difference BIA or (B- A).

7.2. 1.4 Difference (Subtraction)

The difference of set A with respect ro ser B is rhe collection of all elements in rhe- universe rharbelong to A
but do nor belong ro B, i.e., rhe difference set consists of all elemems that belong ro A bur do nm belong to
B. It is denoted by AlB or A- Band is given by .Q
1 CJ
A)Boc(A-B) =

The vice versa of ir also can be performed

[xjxEAandx~

--:~~
8
B) =A-(AnB)
I \'
r

BjAoc(B-A) \B- (BnA) J {xlxE Bandx~A)

I~
The above operations are shown in Figures 79(A) and (B).
~\
I 7.2.2 Properties of Classical Sets

The important propcrries rhat define classical sets and show ilieir similarity w fuzzy sets are as follows:

1. Commurariviry

AUB=BUA; AnB=BnA

2. Associariviry

AU(BU C)= (AUB)U C; An (Bn C)= (AnB)n C

3. Disuiburivity

AU~nC)=(AUB)n(AUC)
An~uC)=(AnB)u(AnC)

4. Idemporency
AUA=A; AnA=A

5. Transitivity
xr,-;-;;c:-r~c:u
~
r'
6. Identity
/ \
1
/~u=f'l, An=4>
IAUX=X
! ,
AnX=X
j
7.'2 Classical Sels (Crisp Sets) 259

7. Involution (double negation)

A=A

~'-
-----
8. Law of excluded middle
-~
A uA=X / J~J
~-') t
9. Law of contradiction
/
nil =4>
10. DeMorgan's law
A
CJ -"~ n
r ->JO
IAnBI ;=AUB; lA UBI =AnB

From the properties mentioned above, we can observe clte duality existing by replacing~with
J ~pectively. Iris imponant to know the law of excluded middle and the law of ~ont~~4\cton.

r )
7.2.3 Function Mapping of Classical Sets

Gpping is a rule of correspondence between seNheoretic forms and function theoretic fermi} classical set_.
1s represented by its characteristic function x where xis the element m the umverse.
Now consider an as two different universes of discourse. If an elemem x contained in X corres ends
to an elememycomained in Y. it is called mapping from X to Y, i.e., :X--+ Y. On t e asis of this mapping,
the cnaamrtmcftnctionis-ctefined as

1, xEA
)(A(x) = ( 0,
x~A

where XA is the membership in set A for element x in ilie universe. The membership concept re resems
mapping from an element x in universe X to one of the rwo elemems in universe eit er ro elemenr 0
or I). There exists a funcrion~theoretic set cil e va ue s.e.L . ) for any set A defined on universe X based on
the mapping of characteristic function. The whole set is assigned a membership value l, and rhe null set is
assigned a members5Jp value 0. ---
Let A and B be tw~ universe X The function~theoreric forms of operations performed between
rhese two sets are given as follows: ----------- ._ ________ _
- - - - - - -- v

1. Union (AU B)

XAua(x) =)(A(x)vxa(x) = m~x[XA(x), XB(x))

Here v is the maximum operator.

2. Intersection (A n B)
XAna(x) =)(A(x)I\XB(x) = min[)(A(x), Xa(x))

Here A is the minimum operator.

3. Complement (A)
X;;(x) = 1- )(A (x)

.1
 260 Introduction to Fuzzy Logic, Classical Sets and Fuzzy Sets
1
4. Containment
\
' If A <; B, then XA(x) ::" XB(x)

----
-------------
I 7.3 Fuzzy Sets
Fuzzy sets may be viewed_as an extension and generalization of the basic concepts of crisp sets. An important
property of fu:z.zy set is that it allows partial membership. A fuzzy set is a set having degrees of membership
between 1 and 0. The membership in a fuzzy set need not be complete, i.e., member of one fuzzy set~ a1so
be member of other fu.zz_y sets in .f!!.e same universe. Fu'l.Z}' sets can be analogous to the thinking of mtdligent
people. If a person has m be dwahed as fnenii or enemy, intelligent people will nor resort to absolute
classification as friend or enemy. Rather, they will classify the person somewhere between two exuemes of
friendship and enmity. Similarly, vagu ess is introduced in fuzzy set by eliminating the sharp bol!_Qdaries
that divide members from nonmembers in the group. ere IS a gra u rrans1t1on etween imembership
and nonmembership, not abrupt transition.
A fuzzy set 4 in the universe of discourse U can be defined as a set of orde_red-p~rs and it is given by
. ,, ' ;'
4=l(x,l',(x))ixEUj )-'t/
,- '7-

wher~_~!x)-is.-th e of membership of x in.cl and it indicates the degiCe J;~t x belongs to 4- The degree
\',_.-. of membership J.L~(x) assumesv ues m e range from 0 to 1, i.e., the membership is set to unit imerval [0, I]
f( ')or !',(x) E [0, 1]. -
/_ r .. , There are other ways of representatio'"l ~,f fu sets; all representations al!ow partial membership to be
' ~ > expressed. When rhe universe of discourse'!, U is discrete d finite fuzzy ser-d is given as follows:
~\ ~-
' r .. ,.

~f]
'\..?
4 = ll''(xJl + I',(X2) + 1',("3) + ..
XI
',
X2 XJ
l '= [ t
', ._
1- 1
----.,.
!',(x;)
Xi
l ,
J '(
\
i
.
~

where "n" is a finite value. When the ti6ivers~ofdiscours~~;-;~~:-;-;~d infiqirtJfuzzy ser4 is given by
:8
'<'.',I 4=1r~")l
~J.~,
In the above rwo represematio~~ of fuzzy sets for discrete and continuous universe, the horizontal bar is
not a quotient but a delimiter. The numerator in each representation is the membership value in set -d that
is associated with the d;meiirof the universe present in rhe denominator. For discrete and finite universe of
d~mmell the summation symbol in rhe re rese on..of.fuzzpetA.does..nOt.denote 3Jgebraic summation
but indinres e ~ ection o ea. e emenr. Thus the summation sig-;;_ ("+") us-ea IS not d1e <dgebtaic "idd"
~di;q 5
rian.rheoreuc union. Also, for continuous and infinite universe of discourse
U, the integral sign in the representation of fuzzy set 4 is nor an algebraic integral but is a~
fu 'on-rheoretic union for continuous variables. 6,
~ set i an on y tf the value of the membership function is 1 for all the members
under consideration. Any fuzzy set 4 d~ned on a universe U is 'a subset of that universe. Two fuzzy sets 4 1
:md !J.are s~d m be equal fuzzy sets i~JLd_(x) =J.Lp_(xf!Jr al~- fuzzy set4 is said ~o be emp~-~ set .. --~
1f and only 1f the value of :_he_ membership ltncuon ~~-~~~-p~ib~:ncmbe~__c_?_~sldered.~efiar \ >-
fm.zy set can also be call~w~fUZiy sej -- , ,-.
~
I

l
1 7.3 Fuzzy Sets

The collection of aH fuzzy sets and fuzzy subsets on universe U is calle,

261

ince all rhe

fuzzy sets can overlap, the 6ii"dinaliry of the fuzzy power set, new> is infinite, i.
On the b;LSis of the above discussion we have 1
I
.4 ;;;-(j--:;;;;:11-::::o rvO '6 ~~F'r\lu'\
AJso, for all x E U ~r
.:-=-
1 l'.;(x)- 0: l'u(x)- I ~
I 7.3.1 Fuzzy Set Operations

The generalization of operations on classical sets to operations on fuzzy sets is not unique. The fuzzy set
operations being discussed in this seaion are termed standard fuzzy set operations. These are the operations
widely ust;:d in engineering applications. Let A and B be fuzzy sets in the universe of discourse U. For a given
element xon the universe, the following function theoretic operations of union, intersection and complement
are defined for fuzzy sets 4 and fl on U.

7.3. 1. 1 Union
The union of fuzzy sets 4 and fl, denoted by 4 U fl, is defined as
l',u~(x) = max[111[x), /l~(x)] =111[x) v llQ(x) fot all x EU
where V indicates max operation. The Venn diagram for union operation of fuzzy sets A and fi is shown in
Figure 7-10.

7.3. 1.2 lntersect;on

The imersection of fuzzy sets A and fl, denoted by 4 n !J, is defined by

J.L.:;!nl!(x) = min[,u<!(x), J..LQ(x)] =J.L,:~(x) 1\ JLf!(x) for all x E U

where 1\ indicates min operawr. The Venn diagram for intersection operation of fuzzy sets 4 and fl. is shown
in Figure 7-11.

K.

01 .////_//)<.._//,0_, IX

Figure 7~10 Union of fuzzy sets 4. and!!,.

262 Introduction to Fuzzy Logic, Classical Sets and Fuzzy Sets

0 X

Figure 711 Imersecrion of fuzzy sets 4 and Q..

7.3.1.3 Complement
When J.L~~~ I], the complemem ofJ, denoted as.d is defined by

IJ.";;_(x) = 1-l'c(x) fo"ll X E U

The Venn diagram for complement operation of fuzzy set .d. is shown in Figure 7-12.

7.3.1.4 More Operations on Fuzzy Sets

l. Algebmic sum: The algebraic sum (d + !l) of fuzzy sets, fuzzy sm 4 and !!. is defined as
-
- - - - - ------- ---- ---------......

l'c+(x) =l'kl+ l'~(x) -~t,(x) I'~(~))

---------- -- - ., .
2. Algebraic product: The algebraic product (d !lJ of 1:\VO fuzzy sets 4 and [J_ is defined as

'I'H(x) =)1,(x)/1~(x)

0 X

Figure 712 Complement of fulZ)' mJ.

7.3 Fuzzy Sets 263

3. Bounded sum: The bounded sum (d ffi Ji) of twq fuzzy sets 4 and [i is defmed as

J10m,(x) =min[!, 1'0 (x)+J1~(x))

4. Bounded dijfrrence: The bounded difference (.cl0 fi) of rwo fuzzy sets 4 and Ji is defined as

11c0,(x) = m,;.[o, Jl,(x)-~~(x/1

I 7.3.2 Properties of Fuzzy Sets

Fuzzy ser.s follow the same properties as crisp .4r.s except for the law ofexcluded middle and law of contradiction.
That is, for fuzzy set 4
~ <A )cb - - ifr ' " ~ "'"""
&td<!J J! ,o ---:;'" .1 u .1 ;< U: .1 n.1 ;< <e:-o'l"' ~y)J/ ,
Frequently used properties of fuuy sets are given as follows:
1. Commutacivity

J1Ull=l!UJ1; J1nll=llnJ1
2. Associativity

J1U@U,)=~umu'
J1n@n0=~nmn'

3. Distriburiviry

J1u@n0=~umn~u0
J1n@u0=~nmu~n0

4. ldemporency

J1UJ1=4; dnJ1=4
5. Identity

4U = 4 and 4 U U = U{universal ser)

dn= and <JnU=d
6. Involution (double negarion)

4=4
7. Transitivity

IfJ1 S ll S ,, then J1 S'

8. De Morgan's law

<1 u ll =4 nji;J1 n ll = 4u li
264 Introduction to Fuzzy Logic, Classical Sets and Fuzzy Sets

I 7.4 Summary
In this chapter, we have discussed the basic definitions, properties and operations on classical sets and fuzzy
selS. Fuzzy sets are the tools that convert the concept of fuzzy logic into algorithms. Since fuzzy sets allow
panial membership, they provide computer with such algorithms that extend binary logic and enable it to
take human-like decisions. In other words, fuzzy sets can be thought of as a media through which the human
thinking is nansferred to a computer. One difference becween fuzzy sets and classical sets is that the former
do nor follow the law of excluded middle and law of contradiction. Hence, if we want to choose fuzzy
intersection and union operations which satisfy these laws, then the operations will nor satisfy distriburivicy
and idempotency. Except the difference of set membership being an infinite valued quanricy instead of a
binary valued quamicy, fuzzy sets are treated in the same mathematical form as classical sets.

I 7.5 Solved Problems

1. Find the power set and cardinalicy of rhe given set (b) Intersection
X= {2, 4, 6}. Also find cardinalicy of power set.
Solution: Since set X comains three elements, so its d n fj = min{/1~(x), /1Q(X))

cardinal number is 0.5 0.3 0.1 0.21

= -+-+-+-
\ 2 4 6 8
nx=3
(c) Complemem
The power set of X is given by
P(X) = {. \2), {4). {6). \2, 4). 0 0.7 0.5 0.8 I
d= l-l'~(x) = Z+ 4 + G + B
{4, 6). \2. 6). {2, 4, 6)) \
The cardinality of power set P(X), denoted by nl'(X) B= 1-I'Q(x)
-
= -0.5
2
0.6 0.9 0 I
+ - + - +-
4 6 8
is found as \

llP(X) =e~ = 23 = 8 (d) Difference

2. Consider two given fuzzy sets (~ \0.5

A\B=AnB'-
0.3 0.5 01
- + - + - +-
I 0.3 0.5 0.21 - - '<_
~: 2 4 6 8
A= - + - + - + -
- \2 4 6 8 ~ \0
B\A-fBnAk - +0.4
- +0.1
- +0.81'
- ,,
--1- -1 2 4 6 8
0.5 0.4 0.1 I I '---._} \
B= - + - + - + -
- \ 2 4 6 8 3. Given the two fully sets \
Perform union, intersection, difference and com~
I 0.75 0.3 0.15 0 I
plement over fully sets d and[}. B, = - + - + - + - + -
- \ 1.0 1.5 2.0 2.5 3.0
Solution: For the given fuzzy sets we have the
1 0.6 0.2 0.1 0 I
following Bz= - + - + - + - + -
- \ 1.0 1.5 2.0 2.5 3.0
(a) Union
find the following:
,1 U fJ. = max\l'~(x), I'Q(x))
I 0.4 0.5 I I (a) [j, UfJ.z; (b) fJ., n fJ.z; (c) fJ.,;
= -+-+-+-
\.2 4 6 8 (d) fu; (c) fJ.J\fJ.z; (f) fJ., u fJ.z;
7.5 Solved Problems
265

(g) [!, n fJ.z: (h) fJ. 1 n fJ.,; (il fJ., u !: Table 1

---
Gain
(j) lb n liz; (k) lb u fJ.z Detection level Detection level
setting of sensor 1 of sensor 2
Solution: For the given fuzzy sets, we have the
following: 0 0 0
10 0.2 0.35
a BUB
( ) _, _, -
_\_I_ 0.75 0.3 0.15 ~~
1.0 + .1.5 + 2.0 + 2.5 + 3.0
20 0.35 0.25
30 0.65 0.8
40 0.85 0.95
b B nB
( ) -' -' -
-\_I_ 0.6 0.2 ~ ~~
1.0 + 1.5 + 2.0 + 2.5 + 3.0
50 1 1

_I
(c) 8 1 =
-
o
1.0
0.25 +0.7
- + -
1.5
-+
2.0
0.85
- + -1 I
2.5 3.0
Now given the universe of discourse X= {0, 10,
20, 30, 40, 50) and the membership functions
for the rwo sensors in discrete form as

d B
( ) -' =
\ o o.4
1.0 + 1.5 +
o.8 0.9 1 I
2.0 + 2.5 + 3.0 [}i= I 0 0.2 0.35 0.65 0.85
o+w+w+3o+40+50
I l
0 0.35 0.25 0.8 0.95 I I
(e) fJ., lfJ.z = [!, n 'f, fh= \ o+10+20+3o+40+5o
0 0.4 0.3 0.15 0 I
= -+-+-+-+- find the following membership functions:
\ 1.0 1.5 2.0 2.5 3.0
(a) I'Q,uJ?>(x); (b) l'lMJ!>(x); (c) I'Q,(x);
-
BU- B\-0- -0.25 -0.7 - 0.85+ -
1I
(f) -'!21- 1.0 + 1.5 + 2.0 + 2.5 3.0 (d) I' (x);
0 (e) I'Q,uQ,(x); (f) I'Q,nQ,(x);

-- = l1.0
(g) fJ., nfJ.,
0
+
0.4
1.5 +
0.8 0.9 1 I
2.0 + 2.5 + 3.0
(g) I'J!>UJ?>;
(j) 1'01 Q, (x)
(h) ILQ,nQ,(x); (i) I'Q,rQ,(x);

- \ 0 0.25 0.3 0.15 0 I Solution: For the given fuzzy sets we have
(h) fJ., nfJ., = 1.0 + ]j + 2.0 + 2:5 + 3.0
(a) 1'/,),UQ,(X)
=max {I'Q,(x),I'Q,(.<))
-
(i) s,us, = \ -I +0.75
- + 0.7
- +0.85
-+- 1I
- - 1.0 1.5 2.0 2.5 3.0 - \ ~ 0.35 0.35 0.8 0.95 _I_ I
- 0 + 10 + 20 + 30 + 40 + 50
- = \ -0 + 0.4
(j) B2 nB - +0.2
- +0.1
-+-0 I
2
- - 1.0 1.5 2.0 2.5 3.0 (b) I'Q,nQ,(x)

= m;n \i'Q,(x),I'Q,\x))
- \ -1 + 0.6
- +0.8
- +0.9 I I
(k) BzUBz=
- -
-+-
1.0 1.5 2.0 2.5 3.0 = \~ + 0.2 0.25 0.65 0.85 + _I_ I
0 10+20+30+40 50
4. It is necessary ro compare two sensors based upon
their detection levels and gain settings. The table (c) I'Q,(x)
of gain serrings and sensor detection levels wirh
a standard item being monitored providing cypi~ = 1-I'Q,(x)
cal membership values to represent the detection - ~~ 0.8 0.65 0.3.5 ~+~I
levels for each sensor is given in Table I. - 0 + 10 + 20 + 30 + 40 50
266 Introduction to Fuzzy Logic, Classical Sets and Fuzzy Sets

(d) JJ, (x) Find the following:

0
= 1-JJ,fb(x) (a) Pl!Pe U Tr<in; (b) Pl.ne n Tr!,in;
= ~~ 0.65 0.75 + 0.2+ 0.05 ~~
0+10+20 30 40+50 {c) Pl!,ne; {d) Tr.!)n;

(e) Pli\fle{Tr.!)n; (f) Pli\fle U Tr.!)n;

(e) ~"Q,u(!, (x)
= max[JJ,Q 1 (x), !"y,(x)} (g) Pli\fle n Tr;\Or.; (h) Pl!,ne U Pl!,ne;
-I~
-
0.8 0.65 0.65 0.85
0+10+ 20 + 30 + 40 +50
.!..) (i) Pl,ene n Pl.ene; (j) Tr!in U Tr~n;

(k) Tr,ein U Tr.!!}n

(f) ~"Q,nQ, (x)
Solution: For the given fuzzy _sets we have rhe
= min[JJ,Q 1(x),JJ,l'1 (x)}
following:
-1~ 0.2 0.35 0.35 0.15 ~~
- 0 + 10 + 20 + 30 + 40 + 50 {a) Pl.@!le U Tr!in

(g) ~"0u0(x)
= max{JJ,fb (x), JJ, 0 (x)}
= max{,U.pJ!ne(x), JlTr!in(x)}

=
1.0 0.5 0.4
-+-+-+--+--
0.8 0.2 I
-I~ 0.65 0.75 0.8 0.95
- 0 + 10 + 20 + 30 + 40 + 50
.!_I 1 train bike boar plane house

(b) Pl!fle n Tr~in

(h) ~"Q,nfb(x) = min{,U.p]!ne(x), /LTr:_in(x)}

I I
l
= min{JJ,fb(x),JJ, (x)} 0.2 0.2 0.3 0.5 0.1
0 = -+-+-+--+--
-I~ 0.35 0.25 0.2 0.05 ~ train bike boar plane house
- 0 + 10 + 20 + 30 + 40 + 50

(i) I"Qd 0 {x) 1

(c) Pl,!ne= 1-J.LpJ!ne(x)
0.8 0.5 0.7 0.2 0.9 I
=t!;nnz(x)
-lil+
-~ r~
= min{JJ,Q 1

0.2 0.35 0.2 0.05 ~I

(x),JJ,n-(x)}
~l
=
I -+-+-+-+--
train bike boar plane house

(d) Tr~n=l-1-LTr~in(x)
-

(j) I"Q,Il! {x)

0 10 + 20 + 30 + 40 + 50
~
I 0 0.8 0.6
-+-+-+--+--
train
o.s 0.8
bike boar plane house
l
.'i =J.Llbri~(x) = min{JLQl(x),JLQL(x)}
{e) Pl~neiTr.!!,in
-~~- 0.35 0.25 0.35 0.15 ~~
- 0 + 10 + 20 + 30 + 40 +50 = Pl,ene n Tr!in
5. Design a computer software to perform image
processing to locate objecrs within a scene. The
two fuzzy sets representing a plane and a train
= min{/-LP!:nc(x),J.Lfr:;in(x)}

=
I 0 0.5 0.3
-+-+-+--+--
train
0.5 0.1
bike boat plane house
I
image are:

0.2
I
0.5 0.3 0.8 0.1
Plane= - + - + - + - - + - -
~ train bike boar plane house
l (f) Pl,ene U Tr!in

= 1- max{JLp[~c(x),JLTf?;in(x)}
1 0.2 0.4 0.5 0.2 l -I 0 + 0.5 + 0.6 0.2 0.8 I
~ I
Train= - - + - + - + - - + - -
train bike boat plane house - train bike bOat + plane + house
 7.5 Solved Problems 267

1 (g) Pllne n Tr<in Solution: For the two given fuzzy sets we have the
following:
=I- min{JLpJ~e(x),JLTr!in(x)} ,

= ~~ + 0.8 + .Q:Z. +
train bike boar
..Q2_ + 0.9
plane house
l (a) 4U!J

= max{l',(x),l'~(x)}
(h) Pli\"e U Pli\fle
= max{J.Lp[~nc(x), JLp[me(x)}
l = I 0 0.75 1 1 0.51
0.64 + 0.645 + 0.65 + 0.655 + 0.66
=
I 0.8
-+-+-+-+~
rrain
0.5
bike
0.7
boat
0.8
plane
0.9
house (b) 4n!j

(i) Pl,ene n Pli!_ne

= min{JLpJ!ne(x), JLp[!nc(x)}
~

=
m;n{JJ,,(x),!'ll(x)}

I 0 0.25 0.75 0.5 0 I

-I.Q2. + .Q1_ + .Ql + ..Q2_ + _Q:!._ I
- train bike boat plane house
0.64 + 0.645 + 0.65 + 0.655 + 0.66

(c) J= 1-JJ,,(x)
(j) Tr.!!,in U [email protected]
= max{JLTr~in(x), JLTr~in(x)}
1.0 0.8 0.6 0.5 0.8 I =
I 1 0.25
-+-+-+-+-
0 0.5 1
0.64 0.645 0.65 0.655 0.66
I
= -+-+-+-+--
1train bike boar plane house

(k) Tr.!!in n Tr!in (d) !i = J- JJ,Q(X)

= min{JLTr~in(x},JlTr!in(x)}

= 1-0- + .Q2
train bike
+ .Q,! + ..Q2_ + ....2:3....
boat plane house
I =
I 1 0.75
-+--+-+--+-
0.25 0 0.51
0.64 0.645 0.65 0.655 0.66

6. For aircraft simulator data the determination of

certain changes in irs operating conditions is made
(e) !J U !i
on clte basis of hard break points in d1e mach = I - max{JJ,,{x),I'Q(x)}
region. We define two fuzzy sets 4 and !}, rep-
resenting rhe condition of"near" a mach number
of0.65 and "in the region" of a mach number of
0.65, respectively, as follows
=
I I 0.25
-+--+-+--+-
0 0 0.51
0.64 0.645 0.65 0.655 0.66

lfl !l n !l
4= near mach 0.65

=
I 0 0.75
-+--+-+--+-
I 0.5 0
0.64 0.645 0.65 0.655 0.66
l =I- min{JJ,,(x),l'a(x)}
1 0.75 0.25 0.5 1 I
!J =in the region of mach 0.65
=
I -+--+-+--+-
0.64 0.645 0.65 0.655 0.66

= I 0 0.25 0.75 1 0.51

0.64 + 0.645 + 0.65 + 0.655 + 0.66 7. For the two given fuzzy sets

For cltese two sets find the following:

I0.1 0.2
A= - + - + - + - + -
0.4 0.6 II
(a) 4 u !J; {b) 4 n !j; (c)' ij;
-

B=
-
0 1

I
2 3 4
I 0.5 0.7 0.3 0
-+-+-+-+-
0 1 2 3 4
l
(d) /i; (e) 4 U !j; (f) 4 n !l
268 Introduction to Fuzzy Logic, Classical Sets and Fuzzy Sets

find the following: [i) <1 nli = min[!L,(x), !L~(x))

(a) <1 u il:

(d) li:
(b) <1 n /j;
(e) <1 u4:
(c) 4:
- -
(f)Anli
.~
}'7
=
! 0 0.2 0.3 0.6
-+-+-+-+-
0 1
11
2 3 4

(g) !! u li: (h) /jn)i; (i) 4 nJi; ~ ['c . (j) 4 U Ji = max[IL<(x), !L~(x))

(j) <1 u
(m) ,jU!J;
li: (k) !Jn4:
(nJ4nli '--'~/P. (\'~
(1) !JU41<9 =
! 0.1
0
0.5 0.4 0.7
-+-+-+-+-
l 2 3
11
4

[k) !! n4 = min[!'ll(x), IL,j(x)}

Solution: For ilie given sets we have:

(a) ,j U !J = max[IL&(x), !'ll(x)} ! 0.9

= -+-+-+-+-
0
0.5
l
0.6
2
0.3
3
o
4
I
=
! 1 0.5 0.7 0.6
-+-+-+-+-
0 1
11
2 3 4
(I) IJU4 = max[!'ll(x),/L;;(x)}

1 0.8 0.7 0.4 o I

(b) ,jn!J= min[IL<(x),l'll(x))
=
! -+-+-+-+-
0 l 2 3 4

=
! 0.1
0
0.2 0.4 0.3 o
-+-+-+-+-
1 2 3 4
I (m) ,j U !J = l - max[IL<(x), J'~(x))

(o) 4 = 1-l',ix)
=
! 0 0.5 0.3 0.4 01
-+-+-+-+-
0 l 2 3 4

0.9 0.8 0.6 0.4 oI (n) 4n li = min[!L;;{x). !L~[x))

= -+-+-+-+-
1 0 1 2 3 4
o +0.5- +0.3
= -
! 0
0.4 o
-+-+-
l 2 3 4
I
(d) = 1-!L~(x)
8. Let U be the tmiverse of military aircraft of
= 1~ + 0.5 + 0.3 + 0.7 + ~I
lo 1 2 3 4 imeresr' as defined below:

(e) d u;;i = max[!L,(x), ILJ(x)} U = [nlO, b52, d30, [2, {9)

10.9 + 0.8 + 0.6 + 0.6 + ~I Ler.d be the fuzzy m of bomber class aircraft:

I
=
l 0 l 2 3 4

(f) d n;;i = min[JL,(x), !L,j(x)) d= !0.3 0.4 0.2 0.1 l

alO + b52 + d30 + {2 + [9

Ler f!. be rhe fuzzy ser of fighter class aircraft:

=
! Ql
0
0.2 Q4 0.4 01
-+-+-+-+-
l 2 3 4
o I
(g) flU= max[!'ll(x), JLix)) !0.1 0.2
!! = alO + b52 + d30 + {2 + [9
0.8 0.7

=
! 1 0.5 0.7 0.7
-+-+-+-+-
0 l
1
2 3 4
I Find rhe following:

[a),j U !J; (b),jn!J; kl4: (d) li:

(h) !l n = min[l'fl(x), !L~(x))
(e) 411! ; (f)!JI!J : (g) <1 u !J;
o +0.5- +0.3 0.3 oI
= - -+-+- (h) 4 n !J; OJ 4u l!: ij) li u <1
0 1l 2 3 4

j
7.5 Solved Problems 269

Solution: We have 9. Consider two fu:zzy sets

(a) ,j U !J = max[!L,(x), !'ll(x)}
I' - !
0.2 0.3 0.4 0.51
A= - + - + - + -
1 2 3 4
= !0.3 0.4 0.8 0.7 I
a10 + b52 + cl30 + {2 + [9
-
0.1
1 !
0.2 0.2 11
B= - + - + - + -
2 3 4
(b) d n!! = min[!L,(x),!'ll(x)} Find the algebraic sum, algebraic product,
0.1 0.2
= \ a10 + b52 + d30 + {2
0.2 0.1 o
+i9
I bounded sum and bounded difference ofthe given
fuzzy sets.
Soludon: We have
[c) 4= 1-!L&lx)

0.7 0.6 0.8 0.9 o

= \ a10 + b52 + d30 + {2 + [9
I (a) Algebraic sum

IL&+~(x) = [J',(x)+!'ll(x)] - [!L,(x)!L~(x)]

0.3 0.5 0.6 0.51

(d) J!= 1-!'ll(x) =
1-1+ -2 + -3 + -4
= ! 0.9 0.8 0.2 0.3 I
a10 + b52 + d30 + {2 + [9
I -
0.02 0.06 0.08 0.51
-+-+-+-
\ 1 2 3 4
(e) ,:! II! = ,j n Ji = min[!L,(x), !L~(x)) = ! 0.28 + 0.44 + 0.52 + ~I

= !0.3 0.4 0.2 0.1 I

a10 + b52 + c!30 + {2 + [9
I (b) Algebraic product
l 2 3 4

l'&~(x) =!L,(x)!L~(x)
(f) !lid= !Jn4 = min[!L~(x),J',jlx)}
o I = !0.02 + 0.06 + 0.08 + 0.51
= !0.1 0.2 0.8 0.7
a10 + b52 + d30 + {2 + [9
(c) Bounded sum
I 2 3 4

,.. \ _, ,.,.-\ 1.
. '
/\
-
\

(g) d U !l = 1 - max[!Lix), !L~(x))

! 0.7 0.6 0.2 0.3 o I l',enixJ_ -

= min[1,J',(x)~ ~
- ,. 0\ I' /

= a10 + b52 + c130 + [2 + [9

(h) d n !J = I - min[IL<(x), !L~(x))

.!
'13-- 0.5 0.6 0.511
=mm l, -+-+-+-
1 2 3 4

= ! 0.9 0.8 0.8 0.9 I

aiO + b52 + d30 + [2 + [9
I =
! 0.3 0.5
-+-+-+-
1
0.6 0.51
2 3 4
(d) Bounded difference
(i) ;;! U )i = max[!L;;(x), !L~(x))
J',o,(x)
0.9 0.8 0.8 0.9 1 I = 1'2._:[0, IL<(x)=~~BJ
1
=1 niO + b52 + d30 + f2 + [9

(j) )iu,j = max[!L~(x),!L,(x))

I
=max 0,
! !-+-+-+-0.1
1
0.1
2
0.2
3
0.511
4
0.9
= \ a10 +
0.8 0.2 0.3 I
b52 + d30 + f2 + /9
=
! 0.1 0.1
-+-+-+-
1 2
0.2 0.51
3 4
270 Introduction to Fuzzy Logic, Classical Sets and Fuzzy Sets

I 0. The discrerized membership functions for a (b) Algebraic product

transistor and a resistor are given below:
l'n(x)
0 0.2 0.7 0.8 0.9 11 =Jl.z{x)JlE(X)
JLT= [ - + - + - + - + - + -
-012345

o 0.1 0.3 0.2 0.4 o.s

iLB= ( - + - + - + - + - + -
I = -
0I o o.o2
+1- +2- + -
3
+-
4
+-
5
0.21 0.16 o.36 o.5)

0 1 2 3 4 5 (c) Bounded sum

Find the following: (a) Algebraic sum; (b) alge- I'I<JE(x)

braic product; (c) bounded sum; (d) bounded = min{I,Jl.z{x)+I'B(x)}
difference.
0 0.3 1.0 1.0
Solution: We have
=min ( I (
, 0
-+-+-+-
I 2 3

(a) Algebraic sum +1.3

- +1.511
-

ILI+B(x)
= [IL.z{x)+ILB(x)J- [IL.z{x)ILB(x)J
o 0.3 1.0 1.0
= [ -+-+-+-+-+-
0 1 2 3
1.0
4
1.0
5
4 5
I
0 0.3 1.0 1.0 1.3 1.51
= ( -+-+-+-+-+- (d) Bounded difference
0 1 2 3 4 5
0 0.02 0.21 0.16 0.36 I'I0B(x)
(-
0
+-+-+-+-
1 2 3 4 = mox{O, l'_nxl-I'B(x))
0.51 0 0.1 0.4 0.6
+- =max ( 0 ( -+-+-+-
5 ' 0 I 2 3
0 0.28 0.79 0.84 0.94 0.5 0.511
= ( -+-+-+-+- +-+-
0 1 2 3 4 4 5

+H 0 0.1
=[-
0 I
0.4
2
0.6 0.5 0.51
+-+-+-+-+-
3 4 5

I 7.6 Review Questions

1. Define dassica.l sets and fully sm. 8. Justify the following sratemem: "Partial mem-
2. Srate the importance of fuzzy sets. bership is allow~d in fuzzy scrs."
3. Wha~ are the methods of representation of a 9. Discuss in derail the operations and properties
classical set? of fuzzy sets.
4. Discuss the operations of crisp sets. 10. Represent the fuu.y sets operations using Venn
diagram.
5. List rhe properties of classical sets.
11: What is rhe cardinality of a fuzzy set? Whether a
6. What is meant by characteristic function?
power set can be forme& for a fuzzy set?
7. Write the function theoretic form representation
12. Apart from basic operations, state few other
of crisp set operations.
operations involved in fuzzy sets.
 ?. 7 Exercise Problems 271

13. Compare and contrast classical logic and fuzzy 15. Describe the importance of funy sets and its
logic. application in engineering sector.
14. Why the excluded middle law does not get
satisfied in fuzzy logic?

I 7.7 Exercise Problems

1. Find the cardinality of the given set: Flow speed Control level 1 Controllevel2

0 0 0
A= {1,3, 5, 7, 9) 20 0.5 0.45
40 035 0.55
2. Consider set X = [2, 4, 6, 8, 10]. Find its 60 0.75 0.65
power set, cardinality and cardinality of power 80 0.95 0.9
set. 100 I
3. Show the following fuzzy sets satisfy DeMorgan's
Given the unive;;rse of discourse i.s X =
law:
(0, 20, 40,60, 80, 100) and me membership
functions
I
(a) I'A(X) = 1+5x
L _ J~ 0.5 0.35 0.75 0.95 + _1_)
112
('. -' - 1o + 20 + 4o + 6o + 8o 1oo
(b) JlB(X) = ( 1.,'5x)

0.= -
0 I 0 0.45
+20- +40- +60
- +so
- +100
-
0.55 0.65 0.9 I l
4. Consider I:WO fuzzy sets

I 0.65 0.5 0.35

A= ( - + - + - + - + -
0 I find the following memberships using standard
set operations:
-

B=
-
2.0 4.0

[-
0
2.0
6.0
0.35
8.0

+-+-+-+-
4.0 6.0
10.0
0.5 0.65
8.0
I
10.0
I (a) llc,u(,(x);

(d) W[i(x);
(b) l'[,nfa(x);

(e) l'(,ufa(x);
(c) l'r,(x):

(f) ILc,nfa(x);
(g) l'[,n[;; (h) l'(,u[-;(x); (i) ll(,u7)x);
Find the following:
(j) l'(,u[-;(x)

(a) <1 u Jl; (b) <1 n ll: (c) 4: (d) Ji; 6. Consider two membership functions as follows:
~

(e) 4 n Ji; (f) 4 u Ji; (g),:! u ll:

1160-xll '
(h),:!nJl; (i)4U4; (j)J1n4;
For fu:u.y set-:1: f.Lt!X() =
8
+1
(k)Jlu; (l)Jln~ 1(40 -xll
()
For fuzzy set /l: Jl~X +I
8
5. We want ~o compare rwo liquid level controllers Find the following:
for their conuolleveis and Aow speed. The fol-
lowing values of Aow speed and liquid control (a),:! U jl; (b),:! n jl; (c) 4: (d) ~;
levels were recorded with a srandard liquid Aow
monitor: (e)d U Jl; (f),:! n ll

j
.272 Introduction to Fuzzy Logic, Classical Sets and Fuzzy Sets

7. Let X be the universe of satellites of interest, as 9. Consider a local area network (LAN) of imer-
delini below' connecred workstations that communicate using
Ethernet protocols ar a maximum rare of 12
X= {ai2, xiS, bl6,f4,f900, vlll} Mbic/s. The two fuzzy sets given below represent
Let 4 be the fu7zy set ofiNSAT~ V satellite: the loading of the lAN:

A (0.2 0.3 I 0.1 0.51 ()= 1-+-+-+-+-

JL'-X
1.0 1.0 0.8 0.2 0.1 i.
\a12 + xl5 +bl6 + f4 + vlll
- =

.Let Jl be the fuzzy set ofiNSAT~B satellite:

0 l 2 5 7
0.0 0.0
+-+-
9 10
l
B=
~
(~ + 0.25 + E.:2_ + 0.7 + ~
al2 r15 bl6 J4 /900
~)
+ vlll ( I-+-+-+-+-
JJ.cx)=
o.o
0.0 0.0 0.5 0.7

l
- 0 I 2 5 7
Find the following sets of combinations for these
0.8 1.0
rwo sets: +-+-
9 10
(a) 4 u!!; (b) 4 n Jl; (c) !j; (d)~; where S,representssilem and Crepresents conges~
(e)4U!J; (f)4n!J; (g) !ju ~; tion. Perform algebraic sum, algebraic product,
bounded sum and bounded difference over the
(n) <! n Jl; (i) 4 Ill; Gl!lk1; (k) <1 u!j; two fuzzy sets.
(l),jn!j; (m) IJU)j; (n) !Jnjj 10. Consider the following rwo fuzzy sets:

8. The discrecized membership functions (in

nondimensional units) for a UJT (uni-juncrion
transi~mr) and BJT (bipolar junction transistor)
X=
- I-+-+-+-+-
0.1
0
0.2
I
0.3
2
0.4
3
0.51
4

are given bdow: Y=

- l-+-+-+-+-
0.5
0
0.4
I
0.3
2
0.2
3
0.1]
4

JLn =
I
0 0.2 0.3 0.6 0.9
-+- +- +- +- +-
-012345
0 0.1
I)

0.2 0.3 0.4 0.7]

Perform the following operations over the given
fuzzy sets:

JLn= (
- 012345
-+-+-+-+-+- (a) ,ru):; (b) ,rn):; (c) K; (d) t
For rhe two fuzzy sets, perform the following (e),rur; (f),rnt; (g)KUX;
calcularions: (hl,rnx; (i),rut (il rul?;
(a)JL]lVJLp; (b)JL_nAJLp; (c)JL]l; (k) algebraic sum; (I) algebraic product;
(d) JLp; (e) JL]l AJLp = JL]l V JL[2 (m) boUnded sum; (n) bounded difference

.J
Classical Relations and
Fuzzy Relations 8
OJ>W w~
~-

Learning Objectives - - - - - - - - - - - - - - - -
Definition of classical relations and fimy Composition of relations - max~min and
relations. max~product composition.

Formulation of Cartesian product of a Description on classical and fuzzy equiva~

relation. lence and tolerance relations.
Operations and propenies of classical rela~ A shan note on noninteracrive fuzzy sets.
tions and fuzzy relations.

1 8.1 Introduction ~ '

Relationships between objects are the basic concepts~olved d~c::~ng
in and other dyn,ic system
applications. The relations are also associated wiili g~ptYtheory, Which has a great impact on designs ana data
mampularions. Relations represem mappings ~and connectives in l~ic. A dassH:ai bmiiy relation
represents the resence or absence of a connecnon or interaction or~ocJatJon between the elements of two
sets. uzzy binary relations are a generalization of crisp binary relations, and they allow various egrees o
rer;;;io~J.hie (~;ia_rion) between elements. In other words, fuzzy relations impart d:grees ~Uengtli[Q. such
connections and ~i61f~ln'""ifuzzy binary relation, the degreeofassociation is represented by memberiliiJ>
grades in the same way as the degree of set membership is represented m a fuzzy set. This chapter discusseS-
the'b~c concepts and operations on fuzzy relations, and the composition between relations is smdied via
the max-min and max-product compositions. The properties and the cardinaliry of fuzzy relations are also
discussed. Other topics discussed include the tolerance and equivalence relations on both crisp and fuzzy
relations.

I 8.2 Cartesian Product of Relation

An ordered r-tuple is an ordered sequence of r-elements expressed in the form (llJ, a2, a3, ... , a,). An unordered
Huple is a collection of r.oelements without any restrictions in order. For r = 2, the r-ruple is called an ordered
pair. For crisp setsAI, Az, ... , A,, thesetofall r-tuples (al, 112,113, ... , a,), wherea1 E A1, a2 E A2 ... , a, E
A,, is called me Cartesian product of AI ,A2 ... ,Ar and,i,s denored by. AI X A2 X ... X A,. The Cartesian
producr of two or more sers js not the same as rhe arirhliietj, product of two or more sets. If all the a,'s are
'ldern!Ciiran.d equal to A, then the Cartesian product A 1 x Az X x A, is denoted as A'.
~
 '
274 Classical Relations and Fuzzy Relations

I 8.3 Cl~ssical Relation

An rary relation over At.Az, ... ,A, is a subset of the Cartesian product At X Az x x A,. When r 2, =
the relation is a subset of the Cartesian product AI x Az. This is called a binary relation fromA 1 roA2. When
three, four or five sets are involved in the subset of full Cartesian product then the relations are called ternary,
quaternary and quinary, respectively. Generally, the discussions are cemered on binary relations.
Consider two universes X and Y; their Cartesian product X x Yis given by

Xx Y={(r,y)lrEX,yE Y)

Here me Cartesian product forms an{O;dered palr}f every X E X with every y E Y. Eve.!Y element in X is
completely related to every element in Y. The characteristic function, denoted by x, gives the men
relationship between ordered patr of elements in each univers;. If it rakes umty as ItS value, then complete
relacionShtp IS found, jj die villue 13 2t16, Chen rnac ) J!Q ldationship. i.e.,
< ---------,
l, (x,y) EXx Y
Xx,, (x,y) = { 0, (r,y) ~ Xx Y

When ilie u!!,iverses or sets are finite, then the relarion is represented by a matrix called relation matrix.
An r-dimensional relation marrix represents an r-ary relanon. l hus, bina1y telitlons are represented by
two-dimensional matrices.
Consider the clements defined in the universes X and Y as follows:

X; [2,4,6); Y= {p,q,r)

The Cartesian product of these two sets leads to

X x Y; {(p, 2), (p, 4), (p, 6), (q, 2), (q, 4), (q, 6), (r, 2), V. 4), (r, 6))

from this set one may select a subset such thar

R; {(p, 2), (q, 4), (r, 4), (r, 6))

Subset R can be represented using a coordinar a ram as shown in Fi re 8-1.

The relation could equivalently be represcnte }:l~]!2S.,a matnx as allows: 1
~--~v

The relation between sets X and Y may also be expressed by mapping representations as shown in
Figure 8-2.
A binary relation in which each element from firsts,erXis nor mapped to more than one element in second
set Y is called a function and is expressed as

~
____ __.
.......__ ..
 8.3 Classical Relation 275

I
,I I
-t----+-

;.
I 6 -f..-
I I
I I
I
-f..- ----+-
'I I
I

\ 2
- I
-t----1-
I
I

p
q r '--1
Figure 81 Coordinate diagram of a relatio .

2 p

4 q

6 -

Figure 82 Mapping representation of a relarion.

Figures 8-3 (A) and (B) show the illumarion of R :X -t Y. Figure 8-3 shows mapping of an unconstrained
relation. A more general crisp relation, R, exists when marches between elements in two universes are con-
strained. The characteristic function is used to assign values of relationship in the mapping of the Cartesian
space X X Y:to the binary values (0, I) and is given by

XR(r,y);
! 1, (r,y) ER
0,
(x,y) ~ R

i
()._,' rr-o'
.i '
276 Classical Relations and Fuzzy Relations

X y X y

Civil Lathe
Time Second
Mechanical Wire

Speed Meier Electrical Transistor

Electronics Soil

Distance rpm Automobile ~ Engine

(A) (B)
-)
Figure 83 Illustrations of R : X -7 Y.
\ rcl
J- r)
Then universal relation (UA) and identity relation (!A) are given as follows: .--1 .
{ (' '.j

~~~~~.~n~~.~~.~~.~0.~~.~~~0l
fA~ {(2,2),(~)---- \, ,,
.. ,.
I 8.3.1 Cardinality of Classical Relation ''\'
( \

Consider n elemcms of universe X being related to m elements of universe Y. When the cardinality of
X::::: TIX and the cardinality of Y = ny, then the cardinality of relation R between the two universes is
\ ~~ --------. ------------
'----------
The cardinaliry of rhe power set P(X x Y) describing rh-erelation is given by

[~:~:;,~,
8.3.2 Operations on Classical Relations

Let RandS be two sepame relations on the Carresian universe X x Y. The null relation and the complete
relation are defined by the relation matrices Rand En. An example of a 3 X 3 form of d1e Rand ER matrices
is given below:

\0= [~ ~ ~]
0 0 0
md '\]i;''T [ 11
"--'
1 1]
1 1
1 1 I
\>-
'0 1:"-
Function-theoretic operations for the two crisp relations (R, 5) are defined as follows: c
1. Union \)

R US---'> XRus(x,y) : XRus(x,y) ~ max [XR(x,y). Xs(x,y))

- {.v
-iJ
2. Intersection

Rn S --->XRns(x,y): XRns(x,y) ~min [XR(x,y). Xs(x,y))

""'
8.3 Classical Relation 277

3. Complement ~ ,' "'

r"~)
R--->X)i(x,y) :xli(x,y) ~ 1-x.(x,y) :..,~. \>-\._ ..
y,(' ~~\
4. Containment
. .
---
~
~ --'_.~
-o- "I
\~
\llcS--->xR(x,y):xk(x,y)Sxs(x,y) I - ,:~ -'
,. . \ ;{-'r__ ,. / -./2)
5. Idenri')'
- '('r
/ ' \"::
-
'0'" . \;
_f/J-+R/ and ER 'f-+ .,\
-:--;\ft-J./!!f:t-v~ {.o'
1 8.3.3 Properties of Crisp Relations 1

8.3.4 Composition of Classical Relations

The opemion executed on two compatible binary relations to get a single binary relation is called com osition.
Let Rbe a relation diat maps elements from universe X to universe an e a relation that maps elemems
from universe Y to universe Z. The two binary relations Rand S are ~e if

-~and(,-~
'----"\
In orher words, the second ser in R r be the same as the first set inS. On the basis of chis explanation, a
relation T can be formed that re at e same e ements o umverse comained in R with the same elements ~
~-f u~iverse Z contained in 5.)-his ty eOfrela:rion can be obtained by performmg the composmon operatiOrl
over the two given relario;I"The composition between the two relations is denoted by R o S. Consider ilie
universal sets given by ---

x~ {a,,a,,a,l: Y~ {b,,b,,b,}: z~ {q,c,,C(}

Let the relacions Rand S be formed as

R~Xx Y~ {(a 1,b,),(a 1,b,),(a2,b,),(a,,b3)}
s~ Yx z~ {(b 1,q),(b,,c3),(b3 ,c,))

Relations Rand S are illuscrated in Figure 8-4. From Figure 8-4, it can be inferred that

T ~ R o S ~ {(a 1, q ), (a,, c,), (a,, c,), (a,. c3)}

The representation of relations RandS in matrix form is given as

b, b, b, CJ (2 C3

a, [1
R=a2 0 I ~]; b, [1 0 OJ
s~b,001
a,
0 0 1 b, 0 1 0
278 Classical Relations and Fuzzy Relations

R s

X
z
B1 b, --t--------,'---

~ -,
a, b,

Figure 84 Illusrrarion of relations Rand S.

Composition T = R o Sis represented in matrix form as

c1 C'2 c3

a,
T=a2 001
[I 0I]
113 0 1 0

This mauix also leads m

T ~ R o S ~ {(a 1, q ), (az, <)),(a,, c,), (a,, 0))

as expected. The composition operations are of rwo rypes:

1. Max-min composition
2. Max-product composirion.

The max-min composition is defined by the function rheoreric expression as

T~RoS

XT (x, z) ~ V, [XR (x, y) A XS (y, z))

1!
The max-product composition is defined by rhe function theoretic expression as

T~RoS

Xr(x,z)=; '.\[XR(x,y)xs(y.z)]

-
yeY:,

The max-product composition is sometimes ~iso referred to as ~t com posicion.

Some propenies of the composition operation are described in Table 8-1.
 8.4 Fuzzy Relations 279

T~ble_ 81 Few properties of composition operation /

Associadve (R o S) oM:::: R o (So M) V
Commutative R o S ;/=So R "f...
Inverse (Ro s)- 1 = s-1 o ] ( I _ /

I 8.4 Fuzzy Relations

FU7.2y relations relate elements of one universe (say X) to those of another universe (say Y) through the
Cartesian product of the two universes. These can also be referred to as fuzzy sets defined on universal s~ts,
which are Cartesian products. A fuzzy relation is based on the concept that everything is related to some exrem
or unrelated. ~ - ---------
~relation is a fuzzy set defined on the Cartesian product of classical sets {XI, Xz, ... : Xn} where tuples
(XJ, XL . , x11 ) may have varying degrees of membership J.LR (XJ,xz, .. , x11) within the relation. That is,

r-;;x"x2 .... ,X"l ~ j JLR(Xl,Xz, ... ,xn)l<xl>Xl--xn), x;EX;

\ X]XX2XXX,.
/
A fuzzy relatloill:ietween rwo sets-xaricl"-yis-a.IledOmary filiiY relatiOilaiiQTS-Clenoted by R(X, Y). A binary
relation R(X, Y) is referred to as bipartite graph when X =f:. Y. The binary relation on a single set X is called
directed graph or digraph. This relation occurs when J( :::= ? and is denoted as R(X,X) or R(X 2). ..
~ ,
}{:::= {xJ,Xz, ... ,x11 } and !:":::= {Y~>Yz, ... ,ym} .\ yYI,-i .
(
I'
-
.
Fuzzy relation 8{}{, [I can be expressed by an 11 x m marrix as follows: l

fLR(XJ,Jl) JLR(XJ,Jl)
lLR(X1,yJ) /LR(xz,yz)
J-LR(XJ,ym)
J-1-R(_~:2,)'m)
\.y. ~ i
f.l(K.YJ =
11R (x,,yl) llR (xn,yz) J-LR(x,,ym)

The matrix representing a fuzzy relation is called fuzzy matrix. A fu1.zy relation B is a mapping from
G._~rresian space X x Y to the interval (0, 1] where the~ smngth is expressed by the membership
function of_the-relation for ordered pairs from the rwo unNWCS [us.i_x,y)}.
A fuzzy graph is a graphical re resem . .narv fu relation. Each element inK and corresponds r
to a noCie in t e z:z.y graph. The connectio..o...li are established between the nodes by the dements of,Xx [
with nonzero membership grades in R X links may also be present in the form of arcs. These links
are labelea-wrd'(" e mem ers ip values as JLf!. (x;,yj). When X =f:. Y, the link connecting the two nodes is an
undirected binary graph called bipartite ~raph. Here, each of the sets X and Y can be represented by a set of
nodes such~hat the nodes corresponding to one sec are clearly differentiated from the nodes represenring rhe
other set. When X:::= Y, a node is connected co itself and directed links are used; in such a case, the fuzz.y
gra_e_h is called directed graph. Here, only one set of nodes corresponding to ser Xis used.
The domain of a binary fuz.z.y relation R(X Y) is the fuz.z.y set, tWm R(X, Y), having the membership
&merion as

' - - - - --- ,.,,. ______

/LclorNinR (X) =-miX7lRcx.J) v r, X E
)

\, ~-- ./\'%s
~---
"-
 280 Classical Relations and Fuzzy Relations

The range ofa bmary fuuyrelauonR(X Y) is the fuzzy set, ran R(X, Y), havingchemembershlp fune.tionas
~-----_ __ - {\! ,::w.
-f"V;J
~
J",;.,.R(y)=max V EY ,
11
~ \0

l
Consider a universe X= [xi>xz,x,,x,} and the binary fuzzy relacion on X as , ,L ~
' :q X2 X3 X4 -{
1
~~<..
6
X! [0.2 0 0.5 0 -cf ~.
Ji(X,X) = X2 0 0.3 0.7 0.8
"' 0.1 0 0.4 0
x,
0 0.6 0 I
The bipartite graph and simple fuzzy graph of E(X,X) is shown in Figures 8-S(A) and (B), respectively.
Ler

{{= [x,,xz,x,,x,} and J::= (yi>Y2YlJ4}

Let Ebe a relation from Kto given by r
0.2 0.4 0.1 0.6 1.0 0.5
R=--+--+--+--+--+--
- (x 1,y,) (x, .n) (xz .n) (xz,yJ) [x,.y,) (x,,y,)
The corresponding fuzzy marrix fur relarion Bis
Yl Y2 YJ
x, [ 0 0.4 0.2]
R=x, 0 0.1 0.6
X3 0.5 0 1.0

The graph of the above relationE= Kx [is shown in Figure 8-6.

X y
0.2

g
0.3

-
Q
x, x, \ (x:J

v.v / 0.7
f1oa
~~ \ ':>C I --ex, \
""
' 0.5 0.1
I
-
0.6
x,e< ' ~
v ... ~~.!C3

)
I 7

x, x, X,
~
u 0.4
0 0.1
~
(A) (B)

Figure 85 Graphical representation of fuu.y relations: (A) Bipartite graph; (B) simple fuzzy graph.
8.4 Fuzzy Relations 281

X3 l.U 0
Figure 86 Graph of fuzzy relation.

I 8.4.1 Cardinality of Fuzzy Relations

The cardinality of fuzzy sets on any universe is infinity; hence the cardinality of a fuzzy relation bet'Neen two
or mer; univer~es is also infinity. This is mainly a result of the occurrence of~til members@!Vm fuiiyser.s-
and fuizy relanons. \ .:r.J' \
, 'Jj? ~_\ \ ~-ti
-----
, '\ s '
I 8.4.2 Operations on Fuzzy Relations ~-' r '(

The basic operations on fuzzy sers also apply on fuzzy relations. Let E and S, be fuzzy relations on the
Cartesian space X x Y. The operations that can be performed on these fuzzy relations are described below:
1. Union

)"~us_ (x,y) = max[)"~ (x,y),J";:(x.y)]

,.'
2. Intersection
--.,. '
/
.) .
J"~r\S_(x,y) = min[)"~ (x,y).!';:(x,y)]
'. ''
3. Complement
.-L--
!")!(x,y) = 1-)"~(x,y)
..
'- .
, ~ ..''\-
4. Containment
L,''<'~~~~~.~ .

'
r is a relation on Y x X defined by

6. Projecuo~zzy relation R(X,Y), le't [R t Y} denote dte projection of R onto Y. Then [R i Y] is a

'in=1"eeGiarion in Y whose membership function is defined by

~ Ji[RI~ (x,y) max)"~ [x,y) -\

.____ ~------_.;..---- ( ,
The projection concept can be extended roan n-ary relation R(x 1,X2, ... ,xn). . '
.~'\\ ~-
~
 282 Classical Relations and Fuzzy Relations

I 8.4.3 Properties of Fuzzy Relations

Like classical relations, the pi'openies of commutativity, associativity, distribucivity, idem_potency and identity
also hold good for fuzzy relations. DeMorgan's laws hold good for fuzzy relations as they do foi classical
relations. The m:.!l relation and complete relation gR are analogous to the null set and the whole
sec .g, respectively, in set rheO~tic form. The excluded middle laws are nor satisfied in fu - relations as for
)
~rs. This is because a fuzzy relation Bis also a tty set, and there ex.im an overlap, between a re anon
1
and its complement. Hence , 1 -;. ., --.;-:1>,
(Q. c' ... ,. z." .J
f .wvU ; J r:,r~- ~-( 0
EUJ(;i<!i (wholeset)o )'"', ;_.
)( 1?:, r-.l '1 c 'f(Q. [' 1'-:>>,(J~ En U! (null set) v ,. 1

12:JU (} i~ , \( -' p( r:<::-r / _(.,

, ~ : ~~
1
P.: 0
I
\ - P.u
8.4.4 Fuzzy Composition
r d' '\ , D
(l._\1 ~ ~
r ' r ; \

Before understanding the fuz.zy composition techniques, let us learn about the fuzzy Carresian product. Let
4 be a fuzzy set on universe X and J!. be a fuzz}r set on universe Y. The Cartesian product over 4 and B results
in fuzzy relation Band is comained within the emire (complete) Cartesian space, i.e., "'

4xJ!=!!

where

EcXxY
The membership function of fuzzy relation is given by

{Jt~.y) =l'~x~(q;:~min[~l ,!
- - . - ~--~- - - - - - -......;..:._-...J---
The Cartesian product is not an operation similar to arithmetic product. Carresian product B= 4 X !},
is obtained in the same way as rhe c~~~t of rwo vectors. For example, for a fuzzy ser-d that has three
elements (hence column vector of size 3 x 1) and a fuzzy ser!}, that has four elemems (hence row vecror of
size 1 x 4), the resulting fuzzy relation Bwill be represented by a matrix of size 3 x 4, i.e., Swill have three
rows and four columns.
Now Ids discuss the composition of fuzzy relations. There are [WO types of fuzzy composition techniques:

1. Fuzzy max-min composition

2. Fuzzy max-product composition
There also exists fuzzy min-max composition method, bur the most commonly used technique is fuzzy
max-min composition. Let Bbe fuzzy relation on Cartesian space X X Y. and be fuzzy relation on Canesian
space Yx Z.
The max-min composition of R(X, Y) a,nd S(Y, Z), deno_d by R(X. Y) o S(Y, Z) is defined by T(X, Z) as
/ '
~tr(x,z) =iLE_ o,t(x,z) ,/{max {!)'in[I'R(x,y).l',t(y.z)])
~ yeY, -
( ,E~'E(x,y)A/l,t(y.z)] Vx EX, z EZ
'-..../
8.5 Tolerance and Equivalence Relations 283

The min-max composition of R(X. Y) and S(Y, Z), denored as R(X, Y) o S(Y,Z), is defined by T(X, Z) as

~tr(x, z) =iLMJx, z) = min {max[I'R (x,y).!L,t (y,z)]} = A [I'E (x,y)V l',t (y, z)] Vx EX, z E Z
y<Y - , y<Y

From theabove definitions it can~b~e~no;t:ed;th:gr;~::-~:;;

r:i .
R(X, Y) o S( Y, Z) - R(X, Y) o S( Y, Z)
...,
The max-min composition is ni~dely used, h"ence the problems discussed in this chapter are iimited to
max-min composition. The max-producr composition of R(X, Y) and S(Y, Z), denoted as R(X, Y) . S(Y, Z),
is defined by T(X, Z) as

~tr(x,z) =iLE,t(x,z) =max [!LE(x,y)l',t(y.z)]

y<Y

= v [I'E (x,y)l',t (y,z)]

yEY

The properties of fuzzy composition can be given as follows:

,A
cr'
Eo~f'~oE r J ~' /
1
<Eo,)T' = ..f'' o 1['
' 1 r
(f! o ~) oJ:1 = !! o ($, oJ:1)

I 8.5 Tolerance and Equivalence Relations

Relations possess various ~~ties. Some of them are discussed in this section. Relations play a major
role in graph theory. The three characteristic properties of relations discussed are: reflexiviry, symmetry and
transitiVicy:"fhe-imronyms of these properties are: ~refiexiviry, a.symJ!letry and nonrransitivi~.
1. A relation is said to be reflexive if every vertex (node) in the graph onginates a single loop as shown in
Figure 8-7.
2. A relation is said to be symmetric if for every edge pointing from vertex ito veerex j, there is an edge pointing
in the opposite direction, i.e., from vertex j to vertex i where i,j = 1, 2, 3, .... Figure 8-8 represents a
symmetric relation.
3. A relation is said to be transitive if for evgr pai~ _ed~es it!.fh~rap
vertex j and the other pointing from vertex; to ve~ b_irtfi I~ 3h NM

n
8
0 0
u u
Figure 87 Three-vertex node- reflexive property.
284 Classical Relalions and Fuzzy Relations

Figure 88 Threevenex node - symmerry property.

Figure 89 Three-vertex graph - transitive property.

Figure 8-9 representS a transitive relation. Here an arrow points from node 1 to node 2 and anQ[her arrow
eKtends from node 2 to node 3. There is also an arrow from node 1 to node 3.

I 8.5.1 Classical Equivalence Relation

Let relation Ron universe X be a relation from X ro X Relation R is an equivalence relation if the following
three properties are satisfied:

1. Reflexiviry
2. Symmetry
3. Transitivity

The function theoretic forms of represemation of these properties are as follows:

1. Reflexivity
'-------------
~;.x;) =/1 or (x;,xi) ER
~-J
2. Symmetry

XR (x;,xjr~-~~-(~,;,j
i.e., (x;, Xj) E R ::::} (Xji"Xi}-E" R

1
 8.5 Tolerance and Equivalence Relations 285

3. Transitivity
XR (x;, Xj) and XR (xi, Xk) =- 1, so XR (x;, Xk) = 1
i.e., (x;,xj) e R(~,xk) E R, so (x;,x~r) E R
The best example of an equivalence relation is the_relacion of similarity among uiangles.

I 8.5.2 Classical Tolerance Relation

1erance re1auon can aJSO oe'C<Illeo, roXJml re1 t

from [Oierance re anon 1 " ... 1 l"'runnn~:~:n ... ~ ,.,;.hin :.~..lt "'
defines R~o here it is X, i.e.
R~-l =R1 oR1 ooR, = R
"-..-' "-v-'
Equiv:llencc
Tol=ncc
rdation
rdation
f'~
\ -----
8.5.3 Fuzzy Equivalence Relation v
Let .fS be a fuzzy relation on universe X, which maps elements from X to X Relation .fS will be a fuzzy
equivalence relation if all the three properties - refle.xive, symmetry and transitivicy - are satisfied. The
membership function theoretic forms for these properties are represented as follows:
I. Reflexivicy
JL~(x;,x;) =I rEX I \
If this is not the case for few x EX, then R(X,X) is said to be irreflexive.
2. Symmetry
/L!. (x;,xi) =IL!. (xj,X;) for all x;,xj EX .rl_'

If this is nor S<uisfied for few x;,Xj EX, then R(X,X) is called asymmetric.
3. Transirivicy
JLf!(X;,Xj) =At and J.lf!(>:J-,Xk) =A2
=> JLE (x;, x;) =/.
where
:i~~min~
;,;, /L[! (x;,xi)d::. max min[JLR (x;,Xj), J..L[! (Xj,Xk)] V(x;, X.(-) E x2
XjEX

This can also be called max-min rransirive. If this is not satisfied for some members of X, then R(X,X) is
non transitive. If the given transitivity inequality is not satisfied for all the members (x;,Xk) E X2 ' then the
relation is called as amirransitive.
The max-product transitive can also be defined. It is given by

~~[JLE(Xi>~~V(x;,x,) EX2
[IL/i(X;,x,) ;>:
---::::---
The equivalence relation discussed can also be called similaricy relation.

1
286 Classical Relations and Fuzzy Relations

I 8.5.4 Fuzzy Tolerance Relation

A binary fuzzy relation that possesses the properties of reflexivity and symmetry is called fuzzy tolerance
relation or resemblance relation. The equivalence relations are a special case of the tolerance relation. The
fuzzy tolerance relation can be reformed into fuzzy equivalence relation in the same way as a crisp tolerance
relation is reformed imo crisp equivalence relation, i.e.,

Rn-l =Rr oR1 ooEr = -...._

__.
~I ~ -

F=y F=y
role ranee o:quiv:den'c
rebtion rdaLion

\"'here "n" is ilie cardinality of the set that defines Rt:

I 8.6 Noninteractive Fuzzy Sets

The i@ependent events in probabilicy theory are analogous ro noninteractive fuzzy sets in fuzzy theory. A
nonimeraaive fuzzy set is defined as follows. We are defining fuzzy set.d on the Cartesian space X= X1 x X2.
4
Set is separable in{~ rwn oaninreracrive fuzz'Uers called ~ions, if and only if

~;(4)~.
---~"~____ I
where

JlOPrx ,~,
"AJ

2 '-'AJ --
JLDPrx1,A, (xJ) = maxJL,J..(XJ,X2) Vx1 EX1

-
'"2EXz
(X2) = maxJ1.{ (XJ,X2) VX2 E X2
;r1EX1

The equations represent membership functions for the orthographic projections of 4 on universes X1 and
x2. respectively.

1 8.7 Summary
This chapter discussed the properties and operations of crisp and fuuy relations. The relation concept is most
powerful, and is used for nonlinear simulation, classification and control. The description on composition of
relations gives a view of extending finziness into functions. Tolerance and equivalence relations are helpful for
solving similar classification problems. The noninteractivicy between fuzzysers is analogo~s to the assumption
of independence in probabiliry modeling.

I 8.8 Solved Problems

1. The elements in two sets A and Bare given as Solution:.'I'he various Cartesian products of these two
given sets are
A~ {2,4} and B~ {a,b,c)
A X B ~ {(2, a). (2. b), (2, c), (4, a), (4, b), (4, c)}
Find the various Cartesian products of these two B x A~ {(a, 2), (a, 4), (b, 2), (b, 4), (c, 2), (c, 4))
sets. Ax A~ A2 ~ ((2,2), (2,4). (4,2), (4.4)}
8.8 Solved Problems 287

Bxs~~~~~~.~w.~~.~~~w.~~. ., Zz Z3

~~~~~~ and ~~JI [ I 0.5 0.3]

]2 0.8 0.4 0.7
2. Consider the following two fuzzy sets:
Obtain fuuy relation'[ as a composition between
0.3 0.7 ll the fuzzy relations.
4~ -+-+-
\ X! Xz X3 Solution: The composition between two given fuzzy
0.4 0.9\ relations is performed in rwo ways as
and 11~ -+-
~ \ n ~ n ,, ~a) Max-min composition
Perform the Canesian product over these given (b) Max-product composition
fuzzy sets.
(a) Max-min composition
Solution: The fuzzy Cartesian product performed
over fuzzy sets 4 and !l results in fuuy relation B
., ., .,
given by B= 4 x lf.. Hence r~tso~~"'[0.6 0.5 0.3]
... .~
' y
' "' 0.8 0.4 0.7

R~
1''
0.3 0.31.
'lA 0.7 - The calculations for obtaining l are as follows:
co.4,_.o.9 ,,
l'r(x,, ZI) ~ max{minii'B (x, .yil.l'.> (y" zdl,
., .. ,. /
The calculation for His as foll.ows: /
min{I'B (x, .]2). 1'.> (n, z,)))
~ max[min(0.6. I), min(0.3, 0.8)1
I'B(x,.y,) ~ min{JL4_(x 1 ),1'~(y!l)
~ max(0.6, 0.3) ~ 0.6
~ min(0.3, 0.4) ~ 0.3
l'r(x,, z,) ~ max{min(0.6, 0.5), min(0.3, 0.4))
I'BCxi.]2) ~ min[l'4_(x,),l'~(n)l
~ max(0.5, 0.3) ~ 0.5
~ min(0.3. 0.9) ~ 0.3
I'[ (xi>Z3) ~ max[min(0.6, 0.3), min(0.3, 0.7))
I'E (x,y,) ~ minl1'4. (x,), I'~ (n)l
~ max(0.3, 0.3) ~ 0.3
~ min(0.7, 0.4) ~ 0.4
ILl" (x,, II ~ max{min(0.2. 1). min(0.9. 0.8))
JLB (x,,]2) ~ minl1'4. (,),I'~ (y2ll
~ max(0.2, 0.8) = 0.8
~ min(0.7, 0.9) ~ 0.7
I'[ ("1, 2l ~ max{min(0.2, 0.5). min(0.9, 0.4)1
I'E (xo,,yii ~ minl1'4. (xo,), I'~ (y!ll
~ max(0.2, 0.4) ~ 0.4
~ min(l,0.4) ~ 0.4
l'r(x-z, z3) ~ max[min(0.2, 0.3), min(0.9, 0.7))
I'E (x,,]2) ~ min[l'4_ (xo,), I'~ (nil
~ max(0.2, 0.7) ~ 0.7
~min(!, 0.9) ~ 0.9
(b) Max-product composition
Thus, the Cartesian product becween fuzzy sets-cl and o
~are obtained. T = E S.
3. Two fuzzy relations are given by Calculations for I are as follows:

]1 ]2 l'r(x,, z,) ~ maxlii'B (XI>JI) I'.> (y"zill.

IS~ Xi [0.60.3] [I'B (XI>]2) I', (n,z,))}

"' 0.2 0.9 ~ max(0.6, 0.24) ~ 0.6
288 Claesical Relations and Fuzzy Relations

JL[(XJ.Z2) ~ max[(0.6 X 0.5), (0.3 X 0.4)) RelationE is obtained as the Canesian product of H..:
~ max(0.3,0.12) ~ 0.3 and I., i.e., -
~
JL[(XJ,Z3)
~
max[(0.6 X

max(O.I8, 0.21)
0.3),(0.3
~ 0.21
X 0.7)]
!i ~ ~ X ~ '')I'\ V J\', u.: \1
20 40 60 80 100 120
JL[(x:z, ZJ) ~ max[(0.2 X I), (0.9 X 0.8))
~ max(0.2, 0.72) ~ 0.72 -,l \ 30 [0.2 0.3 0.4 0.4 0.4 0.2]
.,,,_ ~ 60 0.2 0.3 0.6 0.6 0.6 0.2
JLr(x:z,zz) ~ max[(0.2 x 0.5), (0.9 x 0.4)]
100 0.2 0.3 0.6 0.8 l.O 0.2
~ max(O.l,0.36) ~ 0.36 ., ., 120 0.1 0.1 0.1 0.1 0.1 0.1
JLI(X:Z,Z3) ~ max[(0.2 X 0.3), (0.9 X 0.7)]
Relation S. is obtained as the Cartesian product of I.
~ max(0.06, 0.63) ~ 0.63
and!:f, i.e., . . .-1 , L !t j. ''L; "'VI\ -
The fuzzy relation [by max-product composition is 500 1000 1500 1800
given as ~;I
20 0.2 0.2 0.2 0.2
v 40 0.3 0.3 0.3 0.25
Zj Z2 Z, ., . 60 0.35 0.6 0.6 0.25
r~ 1 [o.6 0.3 0.21] ~~I
~
xN~
~n,j.' 80 0.35 0.67 0.8
x:z 0.72 0.36 0.63 0.25
100 0.35 0.67 0.97 0.25
4. For a speed conuol of DC moror, the membership 120 0.2 0.2 0.2 0.2
fUnctions ofseries resistance, armature current and
speed are given as follows: Relation I is obtained as the composition between
relations BandS, i.e.,
_1 <V

T" ,. , ""l
0.4 0.6+~ ~I 500 1000 1500 1800
~- 30 + 60 100 + 120

1~ I
'
0.2 + o.3 + o.6 + o.8 ~ + o:2J T-Ro -
60 0.35 0.6 0.6 0.25
~ 20 40 60 80 + 100 120 --- s.- 100 0.35 0.67 0.97 0.25
120 0.1 0.1 0.1 0.1

I
!:!~
0.35 0.67 0.97 0.251
500 + 1000 + 1500 + 1800
5. Consider two fuzzy SC[S given by
Compute relation I for relating series resistance
to moror speed, i.e., f?.t: to N. Perform max-min
composidon only. ~ -
A~-+--+
~ Ilow
I 0.2
medium
0.51
high

Solution: For relating series resisra.nce to mawr-

speed, i.e., R., to !:f, we have co perform the following
B~
~ I 0.9
---+--+---
0.4
positive zero negative
0.9 I
operations:: two fuzzy cross-producrs and one fuzzy {a) Find the fuzzy relation for the Cartesian product
composition (max-min):
of-;1 and:, i.e.,E=4 X f!..
(b) lmrod!Jce a fuzzy set{; given by
!i~~x~
S. ~ I, x!:f c~(!!.:!.+~+E_)
- low medium high
rr~ii~r

l
8.8 Solved Problems 289

Find che relation between C and f!. using (d) [0.1 0.1
Cartesian produce, 1.e., fmd oi {; x Ji. 0.1]
(;aS.~ [0.1 0.2 0.7J 1x3 0.2 0.2 0.2
(c) FinO{; o Eusing max-min composition. 0.7 0.4 0.7 3x3
(d) Find b o .,tusing max-min composition. ~ [0.7 0.4 0.7]
Solution: Hence maxmin composition was used to find the
relations.
(a) The Cartesian product' berween 4 and f!. is
obtained as 6. Consider a universe ofa.ircraftspeed near the speed
of sound as X~ {0.72, 0.725, 0.75, 0.775, 0.78)
Jl ~I! x J3_ ~ minlJL<t (x), JL~ (y)] and a fuzzy set on chis universe for the speed "near
mach 0.75" =J:t! where
positive zero negacive

I 0 0.8 I 0.8 0 l
]
0.9 0.4 0.9 M~ -+--+-+--+--
low [ - 0.72 0.725 0.75 0.775 0.78 .
=medium 0.2 0.2 0.2
high o.s 0.4 o.s Define a universe of altitudes as Y = {21, 22,
23, 24, 25, 26, 27} ink-feet and a fuzzy set on this
(b) The new fuzzy ser is universe for rhe altirude fuzzy set "approximately
24,000 feet" = !:f where

c~
- I0.1
-+--+-
0.2
low medium
0;71
high
N- ~~ + 0.2 0.7 +_I_ 0.7
- - 2lk 22k + 23k 24k + 25k
The Cartesian product between {; and
obtained as
Jl is 0.2 o
+ 26k + 27k
I
S.~ (,"x !!.~ min[JL((x),JL~(y)] (a) Construct a relation E= Jyf X /j
positive zero negative (b) For another aircraft speed, say ll:Jl> in the
region of mach 0.75 where

]
0.1 0.1 0.1
low [

I
=medium 0.2 0.2 0.2 0 0.8 I 0.6
M1~ -+--+-+--
high 0.7 0.4 0.7 - 0.72 0.725 0.75 0.775

(c)
f 1 "!. 1?
+ o.~8J
.) ~
",,, ' .9 0.4 0.9]
l,[o' fmd relation ~ = IJ11 o B using maxmin
'l-'
(;a I!~ [0.1
'
composition.
~

o.2 o.7L:\' o.2 0.2 0.2

\~. 0.5 o.s Solution: The two given fuzzy sets are

I
0.4 3X3
j '

~ [0.5 0.4 0.5]

For instance,
M~
- I 0 0.8
-+--+-+--+-
0.72 . 0.725
0 0.2
I
0.75
0.7
0.8
0.775
I
0
0.78
0.7
!:! ~ 2lk + 22k + 23k + 24k + 25k
JL(oE (XJ,yJ) ~-max[min(O.l, 0.9), min(0.2, 0.2),
min(0.7, 0.5)]
\
0.2 0
+ 26k + 27k
l
= mru<(O.l, 0.2, 0.5) ~ 0.5
290 Classical Relations and Fuzzy Relations

(a) Relation E. = l}f X tf is obtained by using If E is a relationship between frequency and

Cartesian product remperarure and s_ represents a relation beCween
remperarure and reliabiliry index of a circuit,

l
ll = min[!Ly (x), 1'/i (Ill obrain the relation between frequency and reli-
ability index using (a) max-min composition and
21k 22k 23k 24k 25k 26k 27k
(b) max-product composition.

0.72 [ 00 00.2 0.7

0.725 0 00.8 0.7
0 0.2
0 00 Solution:
= 0.75
0.775
0.78
0 0.2 0.7 I
0 0.2 0.7 0.8 0.7 0.2 0
0 0 0 0
0.7 0.2 0

{b) Relation .S.=M1 o Sis fOund by using max-min

0 0 ----
(a) Max-min composition is performed as follows.

I= !l o $. =
2
max[min[l'~ (x,y),

4 8 16
1'$. (x,y)]}.
20
composition
9 [0.9 0.6 0.7 I 0.9]
$.= max{min[l'y(x),!L~(x,y)]} 18 0.8 0.6 0.7 I 0.9
= 27 0.6 0.6 0.8 0.9 0.9.
= [o o.8 1 o.6 o],,, 36 0.9 I 0.8 0.8 0.8
~ 0 0 0 0 0 0
0 0.2 0.7 0.8 0.7 0.2 0 (b) Max-product composition is performed as
follows.
0 0.2 0.7 I 0.7 0.2 0
0 0.2 0.7 0.8 0.7 0.2 0
I= Eo$.= max{min[I'E (x,y)X/L(x,y)lJ
0 0 0 0 0 0 o_j5x7
2 4 8 16 20
$.= [o 0.2 o.7 1 o.7 0.2 o],,,
9 [ 0.81 0.5 0.7 1.0 0.9]
18 0.72 0.5 0.7 1.0 0.9
7. Consider [WO rehnions
27 0.4 0.6 0.8 0.9 0.81
-100 -50 0 50 100 36 0.9 1.0 0.8 0.64 0 64
0.5 0.7
9 [ 0.2 Thus the relation berween frequency and reliabil-
18 0.3 0.5 0.7 I 0.9]
0.8
R= ity index has been found using composition tech-
- 27 0.4 0.6 0.8 0.9 0.4 niques.
36 0.9 0.8 0.6 0.4
8. Three fuzz.y sets are given as follows:
and

-100
-50 0.7
2
I
4 8 16
0.8 0.6 0.3 0.1
I 0.7 0.5 0.4
20
P=
- I-+-+-+-+-
0.1
2
0.3
4
0.7
6
0.4
8
0.21
10

$. = 0 0.5 0.6
50 0.3 0.4 0.6
I 0.8 0.8
I 0.9
Q=
-
0.1
0.1 I-+-+-+-+-+-
0.3
0.2
0.3
0.3
0.4
0.4
0.5
0.5
0.21
0.6

100 0.9 0.3 0.5 0.7 T=

- I-+-+-
0.1
0
0.7
0.5
0.31
I
8,8 Solved Problems 291

The following operations are performed over the 9. Which of rhe following are equivalence relations?
fuzzy sets:
No. Set Relation on the set
(a) E=Ex g= min[!Le (x),!Lg (Ill (i) People is the brother of
0.1 0.2 0.3 0.4 0.5 0.6 {ii)People has the same parents as
(iiOPoints on a map is connected by a road to
2 ~I ~I ~I ~I ~I ~I
is perpendicular to
(iv)Lines in plane
4 0.1 0.3 0.3 0.3 0.3 0.2 geometry
6 0.1 0.3 0.3 0.4 0.5 0.2 (v) Positive integers for some integer k; equals
10-" times
8 0.1 0.3 0.3 0.4 0.4' 0.2
10 Ql Q2 Q2 Q2 Q2 Q2 Draw graphs of the equivalence relations.
Solution:
(b) ~= g xI= min[!Lg(r),!Lr(lll (a) The set is people. The relation of the set "is the
0 0.5 brother of." The relation (figure below) is not
equivalence relation because people considered
0.1 0.1 0.1 0.1 cannot be brothers ro themselves. So, reflex~
0.2 0.1 0.3 0.3 ive property is not satisfied. But ~etry and
0.3 0.1 0.3 0.3 transitive properties are satisfied.

~\?
=
0.4 0.1 0.4 0.3
0.5 0.1 0.5 0.3
\
~t:~
0.6 0.1 0.2 0.2

(c) !!f = !l o $. = max{min[!LE (x,y), !L,t (x,y)]l

0 0.5 The figure illustmes that the relation is not an

2 0.1 0.1 0.1 equivalence relation.

4 0.1 0.3 0.3 (b) The set is people. The relation is "has the
same parents as." In this case (f1gure below), all
= 6 0.1 0.5 0.3
the three properties are satisfied, hence it is an
8 0.1 0.4 0.3 equivalence relation.
10 0.1 0.2 0.2

!!f = 8 o $. = max[!'~ (x,y)X!L (x,y)] t) 8

IA
(d)

0 0.5
2 0.01 0.05 0.03
4 0.03 0.05 0.09
= 6 0.05 0.25 0.15 Thus the relation is an equivalence relation.
8 0.04 0.20 0.12 (c) The set is "points on a map." The relation is "is
10 0.02 0.0 0.06 connected by a road to." This relation (figure on
next page) is not an equivalence relation because
Thus the operations were performed over the given the transitive property is nor satisfied. The road
fuzz.y sets. may connect lsr point and 2nd point; 2nd point
292 Classical Relations and Fuzzy Relations

08
and 3rd point; but it may nor connect 1st and 3rd
points. Thus, transitive propercy is nor satisfied.
. t

8
w
&
IL0 The figllre illustrates that the- relation is not an
equivalence relation.

10. The following figure shows three relations on

The figure illustrates that rhe relation is not an ilie universe X ={a, b, c). Are these relations
equivalence relation. equivalence relations?
(d) The set is "lines in plane geometry." The rela-
tion -"is perpendicular to." The relation (figure 0
below) defined here is not an equivalence relation
because both reflexive and transitive properties
are not satisfied. A line cannot be perpendicu-
lar to itself, hence reflexivity is not satisfied. Also
transitivity propeny is not satisfied because 1st
line and 2nd line may be perpendicular to each
other, 2nd line and 3rd line may also be perpen-
(i) {ii)
~
dicular ro each other, but 1st line and 3rd line
will not be perpendicular to each other. However,
symmetry property is satisfied.

8
w
&
I'=8
(iii)
The figure illustrates that rhe relation is not an
equivalence relation. Solution:
(a) The relation in (i) is not equivalence relation
(c) The set is "positive integers". The relation is "for
became transitive properry is not satisfied.
some integer k, equals 10"' rimes." In _this C:tSe
(figure below), reflexivity is not satisfied because (b) The relation in (ii) is nor equivalence relation
a positive integer, for some integer k, equals w"' because transitive property is nor satisfied.
times is not possible. Symmetry and transitivity (c) The relation in (iii) is equivalence relation
propercies are satisfied. Thus, the relation is not because reflexive, symmetry and transitive prop-
an equivalence relation. erties are satisfied.

I 8.9 Review Questions

I. Define classical relations and fuzzy rela- 3. How are the relations represented 10 vanous
tions. forms?
2. Smte the Cartesian product of a relation. 4. Whar is one-ore mapping of a relation?
8.10 Exercise Problems
293

5. Compare constrained relation and non- 13.- Explain the operations and properties over a
conmained relation. fuzzy relation.
6. Give the cardinality of classical relation. 14. Discuss fuzzy composition techniques.
7. Mention the operations performed on c!as'Sical. 15. What are tolerance and equivalence relations?
relations. 16. Describe in derail classical equivalence relation.
8. List the various properties of crisp relations. 17. Write short note on fuzzy equivalence relation.
9. What is ilie necessity of composition of a 18. How are a crisp tolerance relation and a fuzzy
relation? rolerance relation converted to crisp equiva-
10. What are the various types of composition lence relation and fuzzy equivalence relation
techniques? respectively?
11. Define fuzzy mauix and fuzzy graph. 19. Explain with suitable diagrams and examples
12. Give the cardinality of fuzzy relation. fuzzy equivalence relation.
20. What is meam by non-inreractive fuzzy sers?

I 8.1 0 Exercise Problems

1. The elements in two sets X and Yare given as Find the following:
X= {1,2,3}, Y = {p,q,r}. Find the various
Cartesian products of these two sets. (a) !l = .d x f!
2. For ilie fuzzy sets given (b) o\:= f! X (;
(c) I= Ro ~using max-min composition
I
0.5 0.2 0.9\
A= -+-+-
-x,xzX3
l
(d) I= Ro S, using max-product composition

!!= I;+y,+y,
I 0.5 I 5. For two fuzzy sets

find relation Bby performing Cartesian producr

over the given fuzzy sets.
A=
-
I 0.2
LS
~
+ MS + HS
0.7\

3. The fuHy relations arc given as B= I~ + 0.55 0.85\

Zi Z2 - PE ZE+NE
Jl
Y2 Y3 [O S OI]
R=x' [o.1 0.2 o.3], S=~ 0:6 0:9
- X2 0.4 0.5 0.6 - ]3 0.4 1.0
(a) Find !l = .d x !!
Perform composition over ilie two given fuz.zy (b) Introducing a fuzzy set; given by
relations and obtain a fuzzy relation I-
4. Three fuzzy sets are defined as ft;tllows:

A=~~ 0.2
+ 60
+ 0.3
+ 120
0.4\
C=
- I- + - + -
0.25
LSMSHS
0.5 0.75\

- 30

II
I 0.2 0.5 0.7 0.3 0
B= -+-+-+-+-+-
-123456
90
l Findi=f!X (;.
(c) Find 'o S, using max-min composition.
(d) Find 'o Eusing max-min ~Om position.
c = 0.33 0.65 + 0.92 0.21\
- 100 + 200 300 + 400 (e) Find' oB, using max-product composition.
294
Classical Relations and Fuzzy Relations

6. Three elemems for a medicinal research are v, v,

defined as

D=
-
0.3 0.7
1-0 + -I+ -2
II ~=A'
A3Al0.5I 0.5]
0.5
I
-I
l-
0.5 0.75 0.61
20 + 30 + 40
A, 0.5 I

V=
-
I 0.7 0.8 0.51
20+30+40
Find relation,[= ET o ~using
(a) Max-min composition
(b) Max-product composition
Based on these membership functions, find the
following: 9. Two relations are given as
(a) B.=Q X
(b) Max~min composirion of yo fl A [0.8
= 0.1
I
0.2
0.5
0.3
0.1
0.2
0
I 0
0]
(c) Max~product composition of Yo g - 0.1 0.6 0.2 0.7 I 0
0.1 0.4 0.5 0.8 I 0.9
7. An athletic race was conducted. The follow-
ing membership functions are defined based me 0.1 0.2 0.5 0.9 0
speed of athletes: 0.1 0.2 0.5 0.9 0
B= I 0.1 0.2 0.5 0.9 0
~w = I 0 0.1 0.31
100 + 200 + 300 - 0.3
0.3
0.4
0.4
0.7
0.7
0.6
0.6
Medium= -
I
0.5+0.57
- +0.61
- 0.3 0.4 0.7 0.1

I- + - - l
- 100 200 300
Find the relation 4 o fl, using
Hih- 0.8 0.9 1.0
_g - 100 200 + 300 (a) Max-min composition
Find the following: (b) Max-producr composition

(a) B= ~w X Me~ium I 0. Which of the following are equivalence relations?

(b) ~ = Medium x High No. Set Relation on the set
(c) [ = B. o~using m~-min composition. (i) People is the sisrer of
(d) I= B. o ~using max-product composition. (ii) People has rhe same gran dparcms as
8. Two relations are defined as (iii) Lines in plane is parallel ro
geometry
B, B2 B~ B4 (iv) Positive integer For some integer k, equals
e-k cimes
AT
-=A,A3 0.10.3
R 01
0.4 03
0.5 0
0.6] (v) Poimsonamap Is connected by a rail ro
0.4 0.6 0.9
A, 0 0.2 0.9 I Draw graphs of the equivalence relations wiffi
appropriate labels on the vertices.
 'O""'~,..r

Membership Functions 9
learning Objectives - - - - - - - - - - - - - - - - - ,
Scope of membership functions. Different cypes of fuzzification pr.ocesses.
Abour fuz.zification process. Determination of fuzzy membership func-
How membership functions are used to tions using neural networks and generic
define rhe fuzziness existing in the fuzzy algorithms.
ser. Classifications of fuzzy sets.

I 9.1 Introduction
Membership function defines the fuzziness in a fuzzy set irrespective of the elements in the set, which_Ee
discrete or continuous. The membership hlncnons are generally represented in graphical form. There exist
certain limitacions for the shapes used ro represent graphical form of membership function. The rules that
describe fuzzirltss graphically are alSo tuzzy. But stan(Jar(fShipes of the merri.Deiship functions are maintained
over the years. Fuzzy membership functions are determined in practical problem by the opinion of experts.
Membership function can be thought of as a technique to solve empirical problems on the basis of experience
rather rhan knowledge. Available h. rams and other probabilicy information canaiSoneJ- m constructing
the membership function. There are several ways to c aracterize zzmess; In a similar way, there arc several
ways to graphica:Jiy construct a membership function that describes fuu.iness. In this chapter few possibilities
of describing membership functions are dealt with. Also few methodologies have been discussed to build these
membership functions.

9.2 Features of the Membership Functions

(the membership funcnon defmes an ilie mformat1on contamed Ill a fuzzy set; ~ence it is important to discuss
the various fearures of the membership functions. A fuzzy set .cl in the universe of discourse X can be defined
as a set of ordered pairs:

t1 = {(x, 11-d(x)) lx EX]

where 1-',d() is called membership function of.d The membership funccion 1-',d() maps X to the membership
space M, i.e., ~-'.1 : X-+ M The membership value ranges in the interval [0, I), i.e the range of the
membership funcnon 1s a subset: f ffie non-negauve r numbers whose supremum is finit . , .. -
'{ r-;'
'- ,. 1\ co \_:,~)~
;c

' .
\} f
296 Membership Funclions

"(x)

Core
................................. ~

0
.s rt'
:uppo; :. x
t+soundary-\ ~Boundary.J,
Figure 91 Fearures of membership functions.

Figure 9-1 shows the basic features of the membership functions. The three main basic feamres involved
in characrerizing membership function are the following.

l. Core: The core of a membership fimcrion for some fuzzy ser4 is defined as that region of universe char
is characterized by complete membership in the ser.d. The core has elements x of the universe such char
/

!LJ(x) = I

The core of a fuzzy set may be an empty ser.

2. Support: The supporc of a membership function for a fuz.zy ser 4 is defined as chat region of universe chat
is characterized by a ~seed. The support comprises elements x of rhe universe
such char

. 1 0 !'J(x) >
....---......
A fuzzy set whose sup.ggr{_is a single element ~X-wim ttc(x) = i) referred ro as a fuz.z.y si~leron.
3. Boundary: The supJtrr of a membership funct'lon f~r41nlellned as thit region of universe
containing elements that have a no~ur not complete membership. The boundary comprises those
elements of x of the universe such that

0 < ILJ(x) <I

The boundary elemems are those which possess panial membership in the fuzzy ser -!1-

The core, support and boundary are the three main fearures of a fuzzy set membership function. There
are various orher types of fuzzy sets, of which a few are discussed below.
A fuu.y set whose membership function has at least one elementx in the universe whose membership value
is uni_ry h called_ uormal ~ry J(t. The elemem for which the membership is equal to 1 is called pro.lQJYpical
element. A fuZZy set wherem no membershi nction has its value equal m I is called subno_nn4lfo.zzy set.
The normal and subnormal fuu.y sets are shown in Figures 9-2 an , respectively.
A convtxfuzzy set has a menlbership function whose membership values are strictly monotonically increasing
or suiccly monotonically decreasing or strictly monoronically increasing than srrictly monoronicaJly decreaSing
wirh inC'reasmg_~ e(ements in the universe. f\. fuzzy set possessmg chatactensrics opposite ro that r
I
-----
of convex fuzz.y set is called nonconvtxfoizj m, i.e., the membership values of the membership function
I
I
l
J
9.2 Features of the Membership Functions 297

~(x)

~ ~

0 X 0 X
(A) (B)

Figure 92 (A) Normal fuzzy set and (B) subnormal fuz.zy set.
~-

,u(x) ~(x)

,,~ <l

0 ' . . oL-L-----~~-l--~
X1 x2 X3 X X1 X2 X3 x
(A) (B)

Figure 93 {A) Convex normal fuzzy scr <lnd (B) nonconvex normal fuzzy set.

are nor strictly monoronically increasing or decreasing or micrly monoronically increasing rhan decreasing.
The convex and nonconvex normal fuzzy sets are shown in Figures 9-3(A) and (B). respectively.
From Figure 9-3(A), rhe convex normal fuz.zy scr can be defined in rhe following way. For elements XJ, X2
and X] in a fuzz.y set cl- if rhe following relarion berween XJ, X! and holds. i.e.. x,
. ,, '.; L-- 1 '.-'J / , 1
--- ---- ---- ' . \ . j'-
', !_ld(x!l 2: min [J1,1(.q ),Jt<1(.q)] ~ g-r '\r" ~ ,_ rf
--- - -- ------ r(

then d is said robe a convex fuzz.y set. The membership of the elc-menr x~ should be greater than or e~o-/: -~-
the membership of elemenrs x1 and X3 For a nonconvex funy ser, rhe constraint is nor satisfied. _/ \ - r ,
/ \')\.1 /
lltJ(X2) 2: min [J.td(xJ),l-(d(x3)] (, '/-''
1
)~ ,...----- <,
. -~ -----~.1 tp~ ["''
,~) \ Theln[C'rsecrion of t'NO convex fuzzy sets is also a convex fuzzy s::1 The _element in rhe universe for
Which a particular fuzzy set 4 has irs value equal to 0.5 is called crossover point of a membership function.
The membership value of a crossover pomt of a !uu.y set is..~qual to 0.5. i.e., J-l(l(x) ::;::: 0.~. lr is shown in
Figure 9-4. There can be more rhan one crossover point in a 6n:ifr:- ---------- ----
The maximum value 6f the membersh1 funa10n m a fuzz set A is led as the h;!ght of the fuzz.y set.
For a normal fuzzy set, the heig tis equal to 1, because the maximum Yalue of the membership hliitrion
allowed is I. Thus, if the height of a fuzzy set is less than I, then the fuuy set is called subnormal fuzzy set.
When the fuzzy set 4 IS a convex sngle-pomt normal fuzzy set defined on the real nme, then :d-.1s-rP.-rffleCJ as
a fuzzy number --- - " "}l ~ _...--~ / \
\ [, r \\ ...-, ~0\.-- o,')
v ,. -\ \ "' /.J v ' J
/\ ~ f ' --- G>F'"~C)- l
. \ I,
 -:

298 Membership Functions

p(x)

0.5 ~-.------ _,_----- ~

0'''
,J .:!'.
,. '
,, 0

oL---~~--~--~L-~ '"
'c~' ,,~~
x, X, X
~' -.3
\._ ,,.,
Figure 9-4 CrO~o.~er point of a fuzzy ser.
' I
(:-
v '"");;
I 9.3 Fuzzification ('

~flcation : the process of transforming a cl(p ~et w a fuzzy ser or a fuzzy set w a fuzzier set, i.e.,
crisp quantities are converted to fuzzy quantities. This operation rranslares accurate crisp inpur-.valueeil\o
linguistic va?ra~es. In real-life world, the quantities that we consider may be thought of as criW, ac~
,...._ and derermi'nisrtc, but actually they are nor so. They possess uncertainty within themselves. The uncertainty
-.... may arise due ro vagueness, imprecision or uncerrainry; in this case the variable is probably fuzzy and can be
':. '~ represented by a membership IuntfiOn. For example, when one is told rhar rhe temperature is 9 C, the person
translates ilie crisp input value into linguistic variable such as cold or warm according ro on'es knowledge and
~ then makes a decision a0i5Untte need ro wear jacker or nor. If one fails ro fuzzify ilien it js _n_o~possib!:_ ro
~, contin~.e rbe decision process or erro-r deciSion may be reached. -~
1..--- FC?~~l~~~! E X}, a com.~on fuzzificarion algorithm is performed by keepin~ns~ht ~
andb being rramformed ro a fU1Zy"ser-Q(X;}.3epicting the expression about x;. The fuzzy ser _Q(x;) is referred",
to as the~O"lle/ offozzification.J~ set-cl can be expressed as (~
f. 'i ----
6 ',(.,c.=l'-~xr-'-! "V''"':r''"-ko"-'
c
t<l=ll,Q(xJi+!l2Qfxil+--+f'nQ(?] ,, '.
~. where Jile symbol '"" means fuzzified. This process of fuzzification is called support fuzzification
(s-fuzz.ificarion). There is another method of fuzzification called grade fozzification (g-fuzzificarion) where
Xi is kept constant and J.lt_is expressed as a fuu.y set. Thus, using these methods, fuzzificarion is carried out.

I 9.4 Methods of Membership Value Assignments

There are several ways ro assign membership values to fuzzy variables in comparison with the probability
density functions to random variables. The process of m~ value assignment may be by rntuition,
logical reasomng, procedural method or algorithmic approach. The~r-assigning membership value
are as follows:

1. Intuition;
2. inference;
3. rank ordering;
4. angular fuzzy se"rs;
S. neural nerworks;
:,;I

9.4 Methods of Membership Value Assignments

299

6. genetic algorithm;

~ 7. inducr.ive reasoning.
These methods are discussed in detail in the followi~g subsections. Apart from lhese methods, there are oilier
melhods such as soft panitioning, meta rules and fuzzy .statistics, to name a few.

-
I 9.4.1 Intuition
Intuition method is based upon the common intelli ence ofhuman.lt is rhe capacity of the human to develop
membership functions on the basis cif their wn intelligence and understanding capa 1.1 There should be
an in-depth knowledge of rhe application tO whic m v ue assrgnment as to be made. Figure 9-5
shows various shapes of weights of people measured in kilogram in !he universe. Each curve is a membership
function corresponding to various fuzzy (linguistic) variables, such as very lighr, light, normal. heavy and
very heavy. The curves are based on context functions and !he human developing them. For example, if the
weights are referred to range of thin persons we get one set of curves, and if they are referred to range of

I normal weighing persons we get another seC: and so on. The main characteristics of these curves for their usage
in fu~tions are based on !heir overlapping capaciry.
j
I
'

- 9.4.2 Inference

The inference method uses knowledge to perform deductive reasoning. Deduction achieves conclusion by
means of~rward inferen::i)There are various methods for performing deductive reasoning. Here the knowl-
edge of geometrical shapes and geometry is used for defining membership values. The membership functions
may be defined by various shapes: triangular, trapezOidal, bell-shaped, Gaussian and so on. The inference
meiliod here is discussed via uiangula:; ;hape. -. ----
~onsider _a triangle, where X, yand z are th~uch that r~ -~~'and let u be the universe
of rr,Jangles, 1.e., ------

,,. r 0 U= {(X, Y,Z)IX :0:: Y::: Z :0:: O;X+ Y+Z= 180}

\.- '-/\
.\ \_ .~I 11
,);,~,<'~ ru \
(

c. 1 c
}y ''~) Very heavy
Very ligh_t Light Normal Heavy
c:~'" '. r '2. ------,
.... \ \ '
' c ;}
' l

A
tf
"' " ''('>
\.:,- l "
' f- ,'
./J\Qj
' [~ ,,
' 0 20 40 60 80 100 120
Weight in (kg)

Figure 95 Membership functions fur the fuzzy variable "weighr."

 300 Membership Functions

There are various types of triangles available. Here a few are considered ro explain inference methodology:

l = isosceles triangle (approximate)

g= equilateral triangle (approximate)
f1 = righr-angle uiangle (approximate)
R = isosceles and righr-angle triangle (approximate)
I= other triangles
By the meiliod of inference, we can obtain rhe membership values for all rhe above-mentioned triangles. sine~
we possess knowledge about geometry rhar helps us"(0make membership assignments.
The membership values of approxunate isosceles triangle is obtained using rhe following definition, whert:
X 2: Y 2: Z2: 0 and X+ Y+ Z = 180:

~;:-Y,Z)
jilL{.- = I - __I__
r.no min(X- Y,Y- i'l

_or Y = Z: rhe me -ximate isosceles rrian

hand, if X= 120', Y= 60' andZ

I
I'>(X, Y,Z) = I - - min(l20'- 60',60'- 0'1
.(. 60
I
=I-~ min(60,60"')
60
I o
=l--x60
60
=1-1=0

The membership value of approximate righr-angle triangle is given b~

I'~(X, Y, Z) =I - --'-IX- 'JIIo I

----- ____ 3~-----j
If X= 90, rhe membership value of a righr-angle triangle i~ 1, ;wd if X= \80'. rhc- membership \alt1c liN
becomes 0:

X= 90 ~ lly I =
X= 180 => flk=O

The membership value of approximate isosceles righr-<~.ngle triangle is obtained by raking rhe logical
inrersecrion of rhe approximate isosceles and approximate righr-angle rriangle membership function.~. i.e.,

t~
lR=Lnfl)
and ir is given by
(
I'IR(X, Y. Z) = min[I'I(X, Y. Z), 1'8(X, Y. Z) I
= I - mox [--'-- min(X-
60
~' Y- Z) ' 90
__I__ IX- 90'1]
 ' '
9.4 Methods of Membership Value Assignments 301

The membership function for a fuzzy equilateral uiangte is given by

\~<(X' lC' Z) = ' I -.. -180'

~.
1
-.IX- Zl --7
f:,
The membership function of orher r~fangles, deno~ed by[. is the complement of the. logical union of L E
and f., i.e ..

Lr-~ d
By using De Morgan's law, we get ' '\ ,)f-
r= n[ing ./>
/\'
The membership value can be obtained using the equation

11:r{X. Y,Z) =G'in[l -l'l(X, Y,Z), I -I'Ji(X, Y,Z), 1-l'e(X, y;zJ

I
= -min[3(X- Y),3(Y-Z ) ,21X-90o I,X-Z)
180
The inference method as discussed for triangular shape can be extended for trapezoidal shape and so on,
on the basis of knowledge of geometry. '
~ ..... 'y

y\
I 9.4.3 Rank Ordering

The formation of government is based on the polling concept; m identify a best studem, ranking may be "' "/'"t ...

performed; to buy a car, one can ask for several opinions and so on. All the above mentioned activities are carried
out on the basis of the preferences made bv an individual, a committee, a ooll and other

---.., .)
\ ,... . r-
Sets

the major difference between Ute angular fuz:zy....sets and standar<l fuuy Sets. Angu-
sets are detmed on (universe of angle$) thus rep~~ing the shapes o/0' ?zr cycles. The truth
values of the linguistic variable arc represented by angular fU:uy sets. The logical prepositions are equated
to the membership value '\ruth," as they are associated with the degree of uuth. The certain preposition
with membership value "1" is said to be true and that Ute preposition with membership value "0" is said
to be false. The intermediate values between 0 and 1 correspond to a preposition being partially true or
paniall~ ~----
The angular fuzzy sets are explained as follows: Consider the pH value of wastewater from a dyeing
industry. These pH readings are assigned linguistic labels, such as high base, medium acid, etc., to understand
the quality of the polluted water. The pH value should be taken care of because the waste from the dyeing
industry should not be hazardous to the environment. As is known, the neutral solution has a pH value of7.
The linguistic variables are build in such a way that a "neutral (N)" solution corresponds to 8 = 0 rad, and
"exact base (EB)" and "exact acid (EA)" corresponds to 8 =:rrl2 rad and 8;:::: -rrl2 rad, respecrively. The
levels of pH between 7 and 14 can be termed as "very base" (VB), "medium base" (MB) and so on and are ,.
represented between 0 torr /2. Levels of pH between 0 and 7 can be termed as "very acid (VA)," "mediu~ /_
J(-. ,'
''' . ,9 . :,,. r " \ ~y\1'>-
\'r- ' {P ~ , ';"':'-"}- .. i

l .c .
.< '
' -
I
, 71 \
.','
' r~d;( \f '-
302 Membership Functions

\~=~\t2
)Jj_O)
E.B 8=3Tr/8
8=-rr/2 ' VB
fJ=n/4
B

O=Tr/8
MB

D=Trl2c--t-----1~~---i---------t!
pf ~=0
~ 0= --rr/8
MA

A
fJ=-Tr/4
VA
0=-311'/8

'
o~-Trt2.

Figure 9-~ Model Of angular fuzzy ser.

11,(8) = t tan(B)
~nthl piO!ecnon of radial yec@~ Angular fuzzy sets are best in cases with polar coordinates or
,..~~ .... ,., ....... ~ ~ll!e of rhe variable is cyclic.

I 9.4.5 Neural Networks

The basic conceprs of neural nernrorks and various types of neural networks were discussed in derail in
Chapters 2--6. The neural nerwork can be used to obtain fuzzy membership values. Consider a case where
fuzzy membership functions are to be created foHuuy classes of an Jnpurdata.Sh:. The input data sec is collected
and divided into trainin dara set and testing dataset. The training dataset trains ffieneural network. Consider
an input training data set ass own in Figure 9-/(A). The clara ser is found to contain several clara points. The
clara points are first divided into different classes by conventional clysrering g;chnjques In Figure 9-7(A),
it can be noticed that the data points are divided into rhree-'classes, RA, Rs and Rc. Consider clara point
1 having input coordinate values XJ =
0.6 and X2 = 0.8. This data point lies in the region Rs; hence we
assign complete membership~ 1 t6 claSS R.jprn:"d-of~o=to--classes RAand. . In a similar manner, the other
dara points are given rrieffibers 1p values of I or e asses they initially belong. A neural nernrork is created
 303
9.4 Methods of Membership Value Assignments

--------- X
x, ' R,
/\----().- x xx xx1

\)\
.?~!ii.)~j,
/; X : X X xR,X X
x,
I I I I \ \ - \ 't
R,
_.....j A \ '"'( fl -,
x---
R, A"x h XRX
X X XX X X ~~ X ;x
X \ ix
;xI ~~~Rc
X,
X, (C)
(A)
(B)

~-------

' '
~ ~~..I ~: "-"
'
Data points 14
2

.J
X .6 .8

(D)
II~ _.,_
X'
Neurai~Aa
net
'-R

~Rc
A

(F)
---------
(E)

R,

\.1 ,}' ~ ? . ,t;~ /I. X,

,:.,..}) (' \ A signal data point

'- -. "
;:_-:;" ~/~H \' ~- ~
," X, ffi]' ' R, R,~.1
;> - ,-,,~ X, 0.4
-Ra 0.8
r }~ Ac 0.1
,_', "\ . !- (G)
R,
(I)
\' ,________ _'
(H) ) '

Figure 97 Fuzzy membership function evaluated from neural ner:works.

[Figures 9-7(8), (E), (H)) which uses the data point marked 1 and the cpr-responding membership values
in different classes for training itself for simularing the relationship berweeirwotd:mate loggpnS and die
~- The output of neural network is shown in Figure 9-?(C), which classifies data poinrs
imo one of the three regions. The neural ner then uses the nexr ser of data values and membership valu'
further training process as can be seen i The process is continued until the neural nerwork
simulates e enme set ofinput-ourput values. The nerwork performance is tested .usin~he testing data set.
.
- I. .\
2-"'

.-l.
: ''
\}l. "'\
\'-...;\1'\t. "'
\\.

.J' . -,'
 304
Membership Func1ions

When the neural nerwork is ready in irs final version, ir can be used to determine the membership values
of'any _input data [Figure 9-?(G)J in rhe differenr regions (classes) [Figure 9-7(1)]. A complete mapping of
the ~f various data poinrs in various fuzzy classes can be derived to determine the overlap of the
different dasses. The overlap of the three fuzzy classes is show~archel psrtion of Figure,9-7(C). In this
manner, neural network is used to determine rhe filzzy membership fj..mctions.

I 9.4.6 Genetic Algorithms

Generic algorithm is based on rhe Darwin's the01y of evolution; rhe basic rule is "survival of the finest." The
generic algorithm is used here to determine the hlzzy membership functions. This can be done using rh~fl>
following steps: \, , ,._..\"- D,t, )
~is\.. }'l:vV'-' ~-~)-1 \
1. For a panicular functional mapping system, the same membership functions a~d shapes are assumed fo~,~/
various _fuzzy variables to be ddmed. d-_j'it)}: J
2. These chosen membership functions are then coded into bit strings.
3. Then these bit strings are concatenated together.
4. The fitness function to be used here is noted. In genetic algorithm, fimess function plays a major role
similar to that played by activation function in neura.;l~n~e~tw~o~r~k~---~----~
5.
6.

The process of gcneming and evaluating strings is carried our mil we get a convergence to rhe solution
within a ~o~:~~e:0~amme membership functions with best fitness va ue. bership
functions can be obrained from generic algorithm. ' - . ~-

9.4. 7 Induction Reasoning

Induction is used w deduce causes by means ofbackward inferenc e charac~ics of inductive reasoning
_ ip functions. n ucno oys enr~("minimizarion principle, which
dusters the parameters corresponding to the out ut c es. Ti erform mduc~~~~oning merhod a well- .
defined database for the 'ripur-output relationship should exist. he inductive reasoning can be applied for
comp ex sy w ere e ata are abundant and static. For dynamic data sets, this merhod is nor best
suited, because the membership functions continually changes with rime. There exist three laws of induction
(Christeuseu;-1980): -----

1. Given a set orfrr~d~ an experiment, the induced probabilities are those probabilities
consistem widi~au_~vatlabit infurmltf:lon that maximize the entropy of the seL
2. The induced probability of~ofindePend~~tions is proportional to the probability density of
the induced probability of a single observation.
3. The induced rule is that rule consistent with all available information of that minimizes the
entropy. -- :/~.
--- c.
The third law scared above is wi , mem of membership functions. The membersh~e;-! :;-- \
functions using inductive reasoning are generated as follows: ~--- ~-
1. A fuzzy threshold is to be established berween classes of data. "''
~~~--~==::-=::-.:~.,~~==--
,. )

/
9.6 Solved Problems ((A~ 305
''
2. Using entropy minimization screening method, first determine the threshold.}in~( ~ ,9~
, ~.-. tv. ~
3. Then start the segmentation process. - yJ.'"'lt- _ L, '

4. The segmentation process resulrs into two dasse:s~

y&C - ' ,
_ ,,_;;.,..... ~~,ry. ';. J
5. Again parridoning the first two classes one mQ~e time, we obtain three different classes.
6. The partitioning is repeated with threshold value calculations, which lead Us to partition the data set into
a number of classes or fuzzy setS. ~ . . ; . r J 'yw
_ .- -
7. Then'ontheba.<iisOf\the~~On~s eterm St--Lf1 ll
~-
Thus the members 1p cno'n is generated on the basis of the partitioning or analog screening cor.cept.
This draws a threshold line between two 'classes of sample d.aCa.. The ~i~ drawing the threshold line
is ro classi~ the samples whe~~inimizmg the en~~niJ!:

1 9.5
- Summary
Membership functions and their features are discussed in this chapter. Also, the different methods of obtaining
the membership functions are dealt with. The formation of ilie membership function is the core for rhe entire
fuzzy system operation. The,capabiliry of human reasoning is very imponam for membership functions. The
inference method is based on the geometrical shapes and geometry, whereas the angular fuzzy set is based on
the angular features. Using neural networks and reasoning metho9s the memberships are tuned in a cyclic
fashion and are based on rule structure. The improvements are carried out to achieve an optimum sOlution
using genetic algorithms. Thus, the membership function can be formed using any one of the methods
discussed.

I 9.6 Solved Problems

1. Using your own intuition and definitions of the

PI~
universe of discourse, plot fuzzy membership
functions for "weight of people." T AV s vs
Solution: The universe of discourse is weigh[ of peo~
pie. Lcr the weights be in kg, i.e., kilogram. Let the
linguistic variables be the following: b (~
('(.-~-

Ve'Y <hin (Vf) ' W :0 25 .- ,,-I

r ~_ -.::' .
Thin (T), 25 < W::: 45 . :r
Avemge (AV)' 45 < W :0 60 ,'---~----JL-cc-JL----c~c---,cc---ccc-'
Swm(S),60< W::: 75 25 so 75 100 1 25
Figure 1 Membership function for weight
Very srour (VS): W> 75 of people.
Now plotting the defmed linguistic variables Solution: The universe of discourse is age of people.
using triangular membership funCtions, we obtain Let "A" denote age of people in years. The linguistic
Figure 1. variables are defined as follows:

2. Using your own inruirion, plot the fuzzy mem- Very young (VY) :A < 12
bership function for the age of people. Young (Y): 10 _5 A.:::: 22
 306 Membership Functions

Middle age (M)' 20 ,; A ,; 42 PI~

Old (0)' 40 ,; A ,; 72 MW SW
Very old (VO), 70 <A

These variables are represemed using triangular mem-

bership function in Figure 2. 0.5+---

PI~
1iVY_ _ '!. ___ ~ _____ _? ____ VO ,;---~1~o----"-"~1~~--------~,~~c----x

Figure 3 Membership function for frequency

range of receivers.
Solution: Let the universe of discourse be

u~ x~ v~ 55'::: z~ 45'
:,-f10~~2fOL-~3~0--~40~-50'-C-c~~~7~o--'ao~----x {(X, Y.Z): 80':::
and X+ Y+Z~' +55' +45' ~ 180'}
Figure 2 Membership function for age of people.
Membership value of isosceles triangle, [;
3. Compare "medium wave (MW)" and "shan .! ~ --........__1_-- ---
wave (SW)" receivers according ro their frequency !,_,~I-- min(X- Y Y- Z)l1
/ ~"".!. 60 , __/
range. Plot the membership functions using intu- r--------
ition. The linguistic variabl<!S are defined based on ~I - - min(80' - 55', 55' - 45')
60'
the following:
~I- - - min(25, 10)
1
60
Medium wave receivers: frequency lesser rhan
I o
:::::::106 Hz =}--X 10
60
Shorr wave receivers: frequency greater than ~I- 0,1667 ~ 0.833
!=:::::106 Hz
Membership value of right~angle triangle, fS:
\
,-IX. o I o o
JLo~l-- -90 1~1--180 -90 I
-::- universe of discourse. The linguistic var~the .c : 90 90
following: . I - ---
= 1- - X 10 = 0.889
90
Medium wave receivers (MW): frequency lesser
Membership value of equilateral triangle, .g:
than ::::::106 Hz
Short wave receivers (SW): frequency greater than fLo~~-

---(X- Z) ~ I- ----(80- 45)
1
180'
1
180
: : : : 106 Hz
I o
~ I- ~ X 35 ~ 0.8056
180
This is represented using Gaussian membership func~
tion in Figure 3. Membership value of isosceles and right-angle
rriangle,R:
4. Using the inference approach, find the member~
ship values for the triangular shapes l [i, .g, R, - min[0.833, 0.889}
and [for a triangle with ailgles 45,55 and 80.
1
I
I
9.6 Solved Problems

Membership value of other triangles, [: ~

JYf
--_-;,

I
\
/
,_
Q""'Tf/2
(H)
307

/~
I'I ~ min[ 1 - l'f! 1 - ~'ii I - JLsl O=T</3
~ min[0.167, 0.1944, 0.111} ~ 0.111 (SH)

Thus the membership function is calculated for the

triangular shapes. (Z) 0

5. Using the inference approach, obtain the mem-

bership values for the triangular shapes"([i,JJ
for a triangle with angles 40, 60 and 80. (SL)
6=-T</3
Solution: let the universe of discourse be
{L)
u~ {(X,Y,Z) 'X~ 80::: Y= 60 :::Z=40o and 0=-Tf/2
X+ Y + Z = 80' + 60 + 40 = 180)
Figure 4 Angular fuzzy set.
Membership value of isosceles uiangle,!;
with respect to the direcrion of the magnetic
I
!'=I-- min(X- Y Y-Z) field.
60 ' Assume rhe magnetic field B and magnetic
~ I - - - min(80 - 60, 60o - 40)
1 moment JL to be constant, and the linguistic terms
60 for rhe complement angle of magnetic moment be
I given as
= I - - min(20, 20)
60
High moment (H):B =Jr/2
= I- _!__ x 20 = 0.667
Slightly high moment (SH), 0 = rr /3
60
Membership value of right-angle triangle, fi: No momem (Z): e = 0
Slighdy low moment (SL): e = -rr/3
1
!lo =I- -
.c 90
IX- 901 = 1- --'--180- 90ol
90 Low moment (L):B = -rr/2
I Find the membership values using the angular
= I - ~ X 10 = 0,889
fuzzy set approach for these linguistic labels and
Membership value of other triangles, [: plot rhese values versus 8.
Solution: The angular fuzz.y ser is shown in Figure 4.
'".r=min[l-1' 1-JLsl Now calculate the angular fuzz.y membership values
=min[l-0.667, 1-0.889] as shown in the rable below.
=min[0.333,0.111] =0.111
8 tan8 z =cosO JL= l(z)tan61
Thus the membership values for isosceles, right-
angle triangle and other triangles are calculated. rr/2 00 u I

rr/3 1.732 0.5 0.866

6. The energy E of a particle spinning in a magner.ic 0 0 I 0
field B is given by the equation - l f /3 -1.732 0.5 -0.866
rr/2 00 0
E = JLBsinB
where J.(.is magnetic momem of spinning panicle
and B is complement angle of magnetic momem
The plot for th I e5
rn{rle_i 3. \~
nemembershipfu._n\Jonshow nmt
. hIS
' :0',) .
l \
~-
308

Table 1
Membership Functions
1~
~
Number who preferred
.,..~
Maruti BOO Scorpio Matiz Samra Octavia Total Percentage Rank order ~'
~
Maruri 800 - 192 246 592 621 1651 16.5 5 '~fc'

Scorpio 403 - 621 540 391 1955 19.6 2 '

-~t
Maciz. 235 336 - 797 492 1860 18.6 4
Samra 523 364 417 - 608 1912 19.1 3

Octavia 616 534 746 726 - 2622 26.2 1
Total 10000

PI~ Octavia}. Define a fuzzy set 4 on ilie universe of

cars "best car."

Solution: Table 1 shows rhe rank ordering for per~

0.5 formance of cars is a summary of the opinion
survey.
In Table 1, for example, om of 1000 people,
(Z)
192 preferred Manni 800 ro the Scorpio, etc. The
-1!
2
_:rr
3 3 2 toral number of responses is 10,000 (10 compar-
Figure 5 Plot of membership function. isons). On the basis of the preferences, the percentage
is calculated. The ordering is then performed. lr
7. Suppose 1000 people respond to a question- is found that Octavia is selected as the best car.
naire about their pairwise preferences among five Figure 6 shows the membership function for this
cars, X = {Maruti 800, Scorpio, Matiz, Samra, example.

;';
~- ..
Iii
i~i.
~ ;\"!
~
~~ '

~
II"t
j-:.
'
Maru!l Matlz Sanlro Scorpio Octavia
BOO

Figure 6 Membership function for best car.

 309
9.8 Exercise Problems

I 9.7 Review Questions

1. Define membership function and state its impor~ 9. Defme fuzzy number.
1
ranee in fuzzy logic. 10. Explain in detail the infere~ce method adopted
2. Explain the features of membership functions. for assigning membership values.

3. Differenciate the following: 11. How is rank ordering used to define membership
functions based on polling concept?
Convex and nonconvex fuzzy set.
12. Discuss in detail the membership value assign~
Normal and subnormal fuzzy set.
' menrs using angular fuzzy sers.
4. What is meant by crossover point in a fuzzy set? 13. Describe how neural network is used to obtain
5. Define height of a fu:zzy set. fuzzy membership functions.
6. Write short note on fuuification. 14. With suitable example, explain the method by
7. List the various methods employed for the mem~ which membership value assignments are per~

bership value assignment. formed using genetic algorithm.

8. With suitable examples, explain how member~ 15. Give details on membership value assi.p.;nmenrs
ship assignment is performed using intuition. using inductive reasoning.

I 9.8 Exercise Problems

1. Using intuition, assign the m~mbership func- (ii) Race of people
tions for (a) population of cars and (b) library
(a) White
usage.
(b) Moderate
2. Using your own intuition, develop fuzzy mem-
bership functions on the real line for rhe fuzzy (c) Black
number 5, using the following shapes:
\iii) Height of people
(a) Quadrilateral
(a) Very rail
(b) Tcapezoid
(b) Tall
(c) Gaussian function
(c) Normal
(d) Isosceles triangle
(d) Shor<
(e) Symmetric uiangle
(e) Very short
3. Using intuition and your own definition of the
4. Using inference approach outlined in this
universe of discourse, plot fuzzy membership
chapter, find the memb~ship values for each of
functions ro the following variables:
rhe triangular shapes ( 3, g, !fl, [) for each of
(i) Liquid level in the tank the following (all in degrees):

(a) Very small (a) 20, 40, 120

(b) Small (b) 90,45,45
(c) Empry (c) 35,75,70
(d) Full (d) WO, 60,1W
(e) Very full (e) 50, 75, SSO
310
Membership Funclions

5. Using inference method, find the membership

values of the triangular shapes for each of the
xl x, x, RA RB
following triangles: 1.5 0.5 2.5 1.0 0.0
(a) 30', 60', 90'
(a) Use 3 X 3 X I neural ne[Work.
(b) 45', 65', 70' (h) Use 3 x 3 x 2 neural network.
(c) 85', 55', 40'
9. For data shown in the following table (Table A),
6. The following clara was determined by the pair- show the first two iterations using a genetic algo-
wise comparison of work preferences of 100 rithm ro find the optimum membership ftmc-
people: When it was compared wiili software tion (right triangular function 5) for the input
(S), 72 persons polled pceferced hardw.ue (H), variable X and output variable Yin rhe rule table.
65 of them preferred teaching (T), 55 of them
preferred business (B) and 25 preferred textile Table A: Data
(TX). On comparison with hardware (H), rhe X 0 0.2 0.7 1.0
preferences were 60 for S, 42 for T, 66 for B
y I 0.64 0.55
and 35 for TX. When compared with reaching, 0.35
the preferences were 62 for S, 48 for H, 38 for
Table B, Rul.,
B and 25 for TX. On comparison with busi-
ness, rhe preferences were 52 for S, 47 for H, X
35 forT, 20 for TX. When compared with rex-
L s
y z
rile, the preferences were 70 for S, 65 for H, s
44 forT and 40 for B. Using rank ordering plot Note: L -large; 5- small; Z- zero.
rhe membership function for rhe "mosr preferred
work."
10. The energy E of a particular spi1ming m a
7. The following raw clara determines a pair~wise magnetic field B is given by the equation
comparison of a new scooter in a poll of 100
E::::: 11BsinB
people. On comparison with Victor (V), 79 pre~
ferrcd 5plender (5), 59 preferred Honda Acriva wheref1 is magnetic momentofspinningparricle
(HA), 85 preferred Bajaj (B) and 62 preferred and B complement angle of magnetic moment
Infinity (1). When 5 was compared, the prefer- with respect to rhe direction of the magnetic
ences were 21 for V, 22 for HA, 37 forB and field.
45 for I. When HA was compared, the pref- Assuming rhe magnetic field and magnetic
erences were 20 for V, 77 for 5, 35 for B and moment to be constant, we propose linguis-
48 for I. Finally when infinity was compared, tic terms for the complement angle of magnetic
rhe preferences were 32 for V, 54 for 5, 52 for moment as follows:
HA and 50 for B. Using rank ordering, obtain High moment (H):B =::Jr/2
the membership function of "most preferred
bike." SI;ghdy b;gh moment (SH), e ~ rr /8
No moment (N): B::::: 0
B. Develop membership function for trapezoid
similar to algorithm developed for triangle and Slighdy low moment (SL): (}::::: -Jr /8
the function should have two independent vari- Low moment (L): (} = -Jr 12
ables so that ir can be passed. For che cable shown,
show rhe first iteration to compute the member- Find the membership values using the angu-
ship values for input variables X1 ,X2 and X3 in lar fuzzy set approach for these linguistic labels
the output regions RA and Rs. for the complement angles and plot these value
versus"()."
 !")ri

>
~

10
1.,

~ Defuzzification
$
~

'i

Learning Objectives - - - - - - - - - - - - - - - - - - - ,
Need for defuzzif1cation process. crisp tolerance and crisp equivalence relation
How lambda-curs for fuzzy sets and fuzzy respectively.
relations can be carried our. An example provided m depict how the
Various types of defuzzification methods. various defuzzification methods are used to
obtain crisp outputs.
To know how A-cur relation of a fuzzy roler-
ance and fuzzy equivalence relation results in

110.1 Introduction
In fuzzificarion process, we have made the conversion from crisp quantities ro fuzzy quantities; however, in
several applications and engineering area, it is necessary to "defuzzify" the fuzzy results we have generated
through the fuzzy set analys1s, 1.e., It IS necessary to convert fuzz.y resuhs into crisp results. Defuzzificarion is
a mapping process from a space of fuzzy control actions defined over an output universe of discourse into a
space of crisp (nonfuzzy) control actions. This is required because in many ppctica:I applications cnsp conrroi
actions are ne e to actuate the con defuzzification process produces a nonfuzzy control action that
\)est represents t possi 1 It}' 1stri uno JLio.ferred fuzzy control acrion. The defuuificarion process
Jia5-[he capability to reduce a fuzzy set into a crispSingle-valued quantity or into a crisp set; to convert a
fuzzy matrix into a crisp mauix; or ro convert a fuzzy number into a crisp number. Mathematically, the
defuzzification process may also be termed as "rounding it off." Fuzzy set with a collection of membership
values or a vector of values on the unit interval may be reduced to a single scalar quantity using defuzzificmion
process. Enormous defuzz1fication methods have been suggested in the literature; although no method has
proved to be always more advantageous than the others. The selection of the method to be used depends
on the experience of the designer. It may be done on the basis of rhe computational complexity involved,
applicability to the situations considered and plausibility of the outputs obtained based on engineering point
of view. In this chapter we will discuss the various defuzzif1cation methods employed for convening fuzzy
vari:ibles into crisp variables.

110.2 Lambda-Cuts for Fuzzy Sets (AiphaCuts)

Consider a fuzzy set .d The set A,..(O<).. < 1), called the lambda (A)-cur (or alpha [a]-cut) set, is a crisp set
of the fuzzy set and is defined as follows:

A,~ {xil'-t(x)?_l.}; AE [0, I]

 1
312 Dafuzzification

The setA>. is called a weak lambda-cut set if it consists of all rhe elements of a fuzzy set whose membership
-;r:"'
~

-""
l
fi.mccio~ havevaJ~;:uer than or eq~ to a srQfied value. On~~ oilier hiiid, ~e setA), is called a Strong
lambda-cut set if it constitS"6rari die elements o a fuzzy set whose membership funaions have values suiccly i'
greater than ~specified value. A strong A-cur set is given by ~ ~
"~
All the A-cur sets form {family of cris~ets.\Ir is important to note the A-cut set A,. (or Aa, if a-cut ser)
does not have a tilde score, ecauie It 1S a crisp set derived from parent fuzzy set ,d. Any parcicular fuzzy set 4 ...l
can be transformed imo an iefinire number of A-cut sets, because there are lllfinire number of values A can
cake in the imervaU~ -
- The propemes ~sets are as follows:
I. (clU~); =AA UBA
2. (cln~); =AAnBA
3. C,j); f. (}iA) excepr whef>J. = 0.5
4. For any A :Sfi, where 0 ~is true th~ whereAo =X
The fou.nh pro perry is essenrially used in glr~re ~a 10- conrinuou.s-valued fu~
with rwo A-cut values. In Figure 10-1, notice rtiat1'oYA. == 0.2 and {:J = 0.5,Ao.2 lias a greater domain:
Ao.s. I.e., for AS./3 (0.2 :::: O.S),Ao.s ~ Ao.2- Figure 10-2 shows the features of the membership functions.
The core of 4 is the A= l-eur set A 1 The supporc of 4 is the A-cur set Ao+, where A = o+, and it can be
defined as

Ao+ = ...{x IJ.L1 (x) >,_

0}

The imervf [Ao+,AJ] rms the boundaries of the fuzzy ser4, i.e., the regions wirh rhe membership values
between 0 and ., or A= 0 ro I.

I'

1... - - - - - - - - -

Mi--------1---~~

0.2-J----- ../-- L- ___ I __

' X
0 I I-"-,-, '
I , no. , I
~-Ao.2______,

Figure 101 Two different A-cur sers for a cominuous-vaJued fuzzy set.
10.4 Defuzzification Methods 313

p(J/)

--------nr-'-'u'"'.......,tl. ------

oc_~~======~,ss~u~oo~o~rt~,========:;----~
f.-Boundary~ ~Boundary:

Figure 102 Features of the membership functions.

-
110.3 LambdaCuts for Fuzzy Relations
The A-em for fuzzy relations is similar to that for fuuy sets. Let fS be a fuzzy relation where each row of
the relational matrix is considered a fuzzy set. The jth row in a fiJZzy relation matrix B denotes a discrete
memb"frship function for a fuzzy set fS. A fuzzy relation can be convened imo a cr~Sp:fnrion in diC followinf
manner:
]--
iRA= {(x,y){J.LR(x,y) :O:J.}
' - ~-
'

where R;.. is a A-cur relation of rhe fuzzy ~.lliOll ;fS~ce here -B is defined as a rwo-dimensional array, defmed
on the universes X and Y, therefore any pair (x, y) E R>.. belongs to fS with a relation greater than or equal to A.
Similar ro the properries of )vcut fuzzy set, the A-ems on fuzzy relations also obey cenain properties. They
are listed as follows. For nvo fuzzy relations Eand~ the following properries should hold:

I. (i!U~h =RAUSA~
2. (i!n~)A =RAn S>.
3. Qlh f. (iiA) mepr whe~~ -:::--::-\
4. For any A~{3, where 0 ~is true thafJ<~~

I 10.4 Defuzzification Methods

Defuzzification is the process of conversion of a fuz:z.y quantity into a precise quantity. The output of a fuz:z.y
process may be union of two or more fuzzy membership flirlCr!OiiS-Gefiriea:on the universe of discourse of the
o~ur vari . ------------
Constaer a fuzzy omput comprising rwo parrs: rhe first part, [;1, a triangular membership shape [as
shown in Figure 10-3(A)], fie second pan, (;2. a trapezoidal shape [as shOwn in Figure 10-3(B)]. The union
of rhese [WQ membership functions, i.e., ( = Cr U (;2 involves the m~ which is going to be
the outer envelope of the rwo shapes shown in Figures 10-3(A) anl (B) ;em; hape of(; is shown in
Figure 10-3(C).
314 Deluzzification

p p

1-----

q, 0. ---------- -----

q,

0
2 4 6 8 10 z 0 L~2--~4<--~s--a---'11--0-- z

(A) (B)

1.. - - - - -

-- - _,_ - ~- - - - - -

) 2 4 6 8 10 z
(C)

Figure 103 (A) Fim parr of fully ourpur, (B) second part of fuzzy output, (C) union of parrs (A) and (B).

A fuzzy output process may involve many ourpm parts, and the membership...&metion reprbcmtng_each
part~f the output can have any ~hape. The membership function of thefuzzy output need not alwaYs.
be normal. In general, we have

(,=UC=C . ...,I ....

,,\
<..~ S!'
'\
1=1 ' \' \:, --, '
Oefuzzification merhods inch.ii:l.e the following: '. {; ;:- \~
> l ;.J\.
l. Maxmembership principle. r{ "\C. J ')' j

'r:'or
-

Jrt.f'-
~j f,
-
2. Centroid method. ! r 1\ r' 1 "\ ,rofl' I , , I,
3. Weighted average merhod.
4. Mean-max membership. rr- "...-! ,1'

.J
,..,\ ~-> t
'
""')
<{~
.
r
-I
10.4 Defuzzification Methods

p
<

315

0~-------~~-----------------l----+z
Figure 104 Maxmembership defuzzification method.
/}':~.- "~
:-~1~-,\~ r- u. rr
5. Center of sums. ,\/
-:,,
6. Center oflargest area.
7. First of maxima, last of maxima.

Now we discuss the methods listed above.

11 0.4.1 Max-Membership Principle

This method is also known as height method and is limited to peak ourpm functions. This method is given
by the algebraic expression -
1____
------* - ----
I
~~rallxEX

The method is illustrated in Figure 10-4.

11 0.4.2 Centroid Method

This method is also known as cen~.,2~s, center of area or center of gravl!J_ method. It is the most
commonly used defuzzification method. The defuzzified output x* is defined as

f iJ-((X) xtfx
x = '-/7-1'-"-'('-'-(x,...:)dx=

where the symbol J denoteS an algebraic integration. This meffiod is illustrated in Figure 10-5.
316

p
Defuzzificalion

]
I'
i

0~------~~------------~L-__.x
Figure 105 Centroid defuz.zification method.

I 10.4.3 Weighted Average Method

This method is valid fo~trical o~~membership- funcrion.So!J!

-----
weighted by its maximum memh .. ...,h .... .,~1 .... I h.. " .......... '" .t;.. ,..,. ... ;. ~;.,

LP((Xj)Xj
X =
LP,(x,)
r--------------- - - - - - - ---......,__
where L denotes algebraic sum aQil X;juJJ.e,_maximum of the irh membershi~The method
is illustrated in Figure 10-6, where~y sets are consJderejt. From Figure 10-6, we notice that the
defuzzified output is given by

, 0.5a + O.Bb
X =
0.5 + 0.8 ,. :' ~

p
1

0.8 ----------

I a b X

Figure 106 Weighted average defuzzification method (rwo symmetrical membership functions).
10.4 Defuzzification Methods 317

-----------,.--,--,

''
I

0 B x" b X

Figure 107 Mean-max membership defuzzification method.

As this method is limited to symmetrical membership funcrions, rhe values of a and b are the means of
their respective shapes.

I 1 0.4.4 Mean-Max Membership

This method is also known as the middle of the maxima. This is closely related to max~membership
method, except that the locations of the maximum membership can be nonunique. The output here is
given by ~ - \
.-.J ~ s.
x = Li= 1 X;
Li .~
0
J) .-',.
n '
This is illustrated in Figure 10-7. From Figure 10-7, we norice that the defuzzified output is given by

X=--
a+ b
2
where a and b are as shown in the figure.

I 1 0.4.5 Center of Sums

This method employs th~lgebraic sum of the individual fuzzy subsets mstead"'f'tlieu uru_Qffi The calculations
here are very fast, bur the main drawback is that imersectLn~ areas are added twice. The defuzzified value x*
is given by

.fo/ X
, J,x L7-l I''; (x)dx
fx L7~, I''; (x)dx
Figure 10-8 illustrates the center of sums method. In center of sums method, rhe weights are the areas of
the res ective membership functions, whereas in the weighted average method the weights are ~~d,iy_idual
membership v ues. - - - - - --- - ----- -. .-

I 10.4.6 Center of Largest Area

318 Defuzzificatioh

I' I'

0.54----- ---------- 0.5

0
2 4 6 8 10
X oL~-~---J:----:--;1;;-0-
, 4 6 8 x

(A) (B)

I'

"'c--+--~x
2 4
;(
(C)

Figure 108 (A) Firsr and (B) second membership functions, (C) defuz.zificarion.

set has at least two convex regions, then th~~of gravitY oftlie copyex fnzry...o;uhregon haymg the~
area is used to obtain ilie defu.zzified value rfhis value is given by

x'=Jf.L.r;(x)xdx
J f.L.u (x)dx

where j is the convex subregion that has rhe largest area making up .fi Figure 10-9 illumares the cemer of
largest area.

I -----------
10.4. 7 First of Maxima (Last of Maxima)

This method uses the overall- OU:tpi.It or union of all individual output fu...B sers y- for determining the
smallest valUe of the dom3.In wtifi maximized membership m -fi The steps used for obtaining -? are
 \
i!)
. p' }
10.4 Detuzzification Methods " .o' 319
ucr I (f'
v. '\

0.5

X
10 12 14
it.
Boundary

Figure 109 Cenrer oflargest area method.

as follows:
1. Initially, the maximum height in the union is found:

hgt(g) = sup,u,g (x)

.<EX

where sup is supremum, i.e., the :fe-ast

._,- upper bouml:
2. Then the first of maxima is found:

x = inf[x E X[i<.o (x) = hgr(i)}

...-EX
r --- ---- - ---
where inf is the infimum, i.e., the J_reatesr lower D?~_l!.'fl
{
3. After this the last maxima is found: \.!

x =sup [x E X[i<.o (x) = hgr(s)}

.<EX

I'

0.5

(
.. -,-" \
V 1///</4////f/\
I < ~
X
--( '" s-1' ''
v . /1.-"

0 2 4 12 ''-
'l r:---' rl'
~ {I-~-
c
)(' ---"\.

Figure 1010 First of maxima (last of maximaf method.

320

where sup= supremum, i.e., ilie least uppeibound; inf =infimum, i.e., the greatest lower bound. This
Defuzzification
J
l
is illusuared in Figure 1"010. From Figure 10-10, the first maxima is also the last maxima, and since it is
a distinct max, it is also rhe mean-max.

110.5 Summary
I
In this chapter we have discussed the methods of converting fuzzy variables into crisp variables by a process I
called as defuzzification. Defuu.ifi.cacion process is essential because some engineering applications need exact
values for performing the operation. For example, if speed of a motor has to be varied, we cannot instruct to
raise it "slighcly," "high," ere., using linguistic variables; rather, it should be specified as raise it by 200 rpm
or so, a specific amount of raise should be mentioned. Defunificarion is a natural and essential technique.
Lambda-cur for fuzzy sets and fuzzy relations were discussed. Apart &om the lambda-cur method, seven
defuzzificarion methods were presented. There are analyses going on to justify which of the defuzzification
method is the best? The method of defuzzification should be assessed on the basis of the output in the context
of data available.

110.6 Solved Problems

1. Consider two fuzzy sers 4 and fl, both defined (a) 8) 0j = 1-1'-tlx)
on X, given as follows:

JL(XjX) XI X}_

0.2 0.3 0.4"' 0.1

"'0.7 Xj
=
l
0.8 0.7 0.6 0.3 0.91
-+-+-+-+-
XI
(Alo.? = [x,x,,xsl
X2 XJ X<! X5

<1
0.4 0.5 0.6 0.8 0.9

Express rhc following A-cut sets using Zadeh's

notation:
(b) 8=
- l
0.4 0.5 0.6 0.8 0.91
-+-+-+-+-
X]X2X)X4X5
(B)o.2 = {XIJX2.,X3,X4,XS}

(c) (4 U ) = max[l'4_ (x), I'~ (x)]

(a) (:;:j),,; (b) lmo.z: (c) (4 U IDo.o;
(d) (4 n IDo.s:
lg) (<1 n ),,; (h)
(c) (4U:;:j) 07 ;
G1 U IDo.s
10 (n:IDo.3: =
l
(AU B)o.6 = [x,,x,,xsl
0.4 0.5 0.6 0.8 0.91
-+-+-+-+-
XJ X2 X3 X<j XS

Solution: The rwo fuzzy sets given are (d) (<1 n ) = min[l'4, (x), I'~ (x)]

~ l
0.2 0.3_ 0.4 0.7 0.11
A= - + - + - + - + -
XjXzX)X4X5
(An Blo.s
=

=
l 0.2 0.3 0.4 0.7 0.11
-+-+-+-+-

!x41
XJ X2. X3 X~ XS
,_
B=
~ l
0.4 0.5 0.6 0.8 0.91
-+-+-+-+-
XJX2.X3X4X) (e) (4U:;:j) = max[l'-t(x),l':j_(x)]

We now find the A-cut set:

1
6._=- [xll'-t (x) > J.l .
=
I0.8 0.7 0.6 0.7 0.91
-+-+-+-+-
X!

(AUA) 0.7 = [x 1,x,,X4,xsl

X2 X3 X4 X)

- I

j
 10.6 Solved Problems 321

(f) ( n Ill= min(!'~ (x), IIQ (x)] = f~ +

0.5 + 0.(,5 + 0.85

l 'I I
\o
20 4o 60
=
I
(BnB)o..l = lx,,xz,XJI
0.4+ 0.5
-
X]
- +0.4
- +0.2
X2
- +0.1
-
X.~ X<j X5
1.0
+-+-
80
1.0
100
(S, USzlo.s= [20.40,60,80, IOO}c'"
_,

o"'!u'
(g) (4n) = 1-l'(.iniD (b) (1 n ,l_,) =min[ I'>, (x), I'" (x)]

= f 0.8 + 0.7 + 0.6 + 0.3 +0.91

f 0 0.45 0.6 0.8
= \ 0 + 2o + 40 + 60
lx1
(A n Blo.6 = [x, ,xz, "'' xs I
X2 X) x4 xs
.95 . 1.0
+80 + \00
I
(S, n Szlo.s = [40, 60, 80, 1001
(h) (:;:jUI.i) = maxll':tlx),I'Q(x)]

--~ -
0.8+ 0.7
- +0.6
- +0.3
- +0.91
- (c) [I= 1-l',,(x)
X] Xz X3 X<j X)
(AU B)o.s = [x, X> I
2. Using Zadeh's notation, determine the A-em sets
!
1 0.5 0.35 0.15
= -+-+-+-
020 40
0
60
0 I
for the given fuzzy sets: + 80 + 100
IS.los=[0,201
s = f ~ + 0.5 + 0.65 + 0.85 + ~ + ~I
"' \0 20 40 60 80 100 (d)
:\2 = 1-l'"(x)
s = f~ + 0.45 + 0.6 + 0.8 .95 + ~I
<2 \ 0 20 40 60 + 80 100 = f ~ + 0.55 + 0.4 + 0.2

Express the following for A= 0.5:

\0 20
0.05
+so+
40
0
100
I60

(a) (~, U S,); (b) (~ 1 n S,); (c) [I; (d) 3}:; (5,),, = [0,201
(e) (,ll u &) ; (f) (~ 1 n &)
(e) (~, us.z) = 1-l',,us.,(,)
Solution: The two fuzzy sets given are
-E o.5 + o.35 ~
- \ 0 + 20 40 + 60
,ll -
- \0
f~ + 0.5 0.65 0.85 ~ ~I
20 + 40 + 60 + 80 + \00 0 0
+ 80 + 100
I
s = f~ + 0.45 0.6 + 0.8 + .95 + ~I (51 u Sz) 05 = [O, 201
<2 \0 20 + 40 60 80 100

The A-cut set is obtained using (f) (.ll n &) = 1-l',,n "!>

A,= [xll'4_(x) ;::J.I "\fl- '(\ - f ~ 0.55 0.4 + 0.2

Here A= 0.5.
~
(
\\J-

(( \~.)"
,/.s
i' ,,~'
-\o+ 20 +
0.05 .
+so+
4o
0
100
I 6o

(a) (1 US,)= max[l'>1 (x),l'.!>(x)i (s, n Sz) 0_5 = [O. 20}

322 Deluzzificalion

3. Consider the cwo fuz.zy sets (f] an li= min[ I'{ (x), I'~ (y)]
A=
- I 0
0.2
0.8
-+-+-
0.4
1
0.6
I -
0.1
1
0.2
0.2
- -+-+-
0.4
0 I
0.6
(An B)o.< =[I
~nd B= 10.9 + 0.7 + 0.3)
- 0.2 0.4 0.6 Case (ii)o A= 0.7
Using Zadeh's notations, express the fll7Z}' sets
imo Acur sets for A= 0.4 and A= 0.7 for the
( )
a
-A=1-!L-t(x)= ( -+-+-
-
1
0.2
0.2
0.4
0
0.6
I
following operations:
(Alo.7 = {0.21
(a) 4: (b) j!; (c) !1 U i!:
(b) - 10.1 0.3 0.71
(d) <1 n Jl; (e) 4U l!: (f14nl! = 1-I'Q(rl = 0.2 + 0.4 + 0.6
Solution: The two fuzzy sers given are (Blo, = {0.61

A=
- l-+-+- I
0
0.2
0.9 0.7 0.31
0.8
0.4
1
0.6
(c) iJU/!=max[l'{(x),I'Q(rll

0.9 0.8
-- -+-+- 1 I
and B= ( -+-+- 1 0.2 0.4 0.6
- 0.2 0.4 0.6
(AU Blo.7 = {0.2, 0.4, 0.61
Case (i): A= 0.4
(a) -
A=1-I'A(x)=
- -
11
0.2
0.2
-+-+-
0.4
0
0.6
I (d) !1 n i!= min{JL-1 (x), I'Q (y)]

(A)o.- = (0.21

- 10.1 0.3 0.71

I0

(An Blo.7 = (0.41

0.7
-- -+-+-
0.2 0.4
0.31
0.6

(b) B=H<Q(y)= -+-+-

- 0.2 0.4 0.6
(il)o.< = (0.61 (e) 4U Ji= max[!'{ (x), I'~ (y)]
1 0.3 0.71
(c) !1 U i!= max[l'oi (x), I'Q (y)] - -+-+-
I
- 1 0.2 0.4 0.6
(Au Ii) 0 _7 = (0.2, o.61
I
0.9
0.2
0.8
-- -+-+-
0.4
. r~u_ll)~az.~.4. 0.6]
1
0.6
(f] 4nJi=min[I';J:(x)./l~(y)]
0.1 0.2 0 l
!1 n i! = -- -+-+-
(d) minll'-t (x), I'Q (y)] 0.210.4 0.6

--
(An Blo.< = (0.4)
I 0 0.7
-+-+-
0.2 0.4
0.31
0.6
(An Blo.7 =[I

4. Consider the discrete fuzzy $-.defined on the

universe X- {a,'b, c, d, e} as
(e) 4U)i=maxll';t(x),l'~(y)]

1 0.3 0.7) -
1 0.9
a b
0.6
A= ( -+-+-+-+-
c
0.3
d
0
t
l
-- -+-+-
1 0.2 0.4 0.6
Using Zadeh's notation, find rhe Acut sets for
(AU Blo.< = {0.2, 0.6) A= 1, 0.9, 0.6, 0.3, o+ and 0.
l 10.6 Solved Problems

Solution: The fuzzy set given on the universe of

discourse is

1
!1= -+-+-+-+-
a b c d e
0.9 0.6 0.3 0 l
(a) ).::;:; 0.1,

~I=[: :]
323

The_)....cm set is given as (b)A=O+,

A, = {x \1'-t (x) 20A I

[0 1 1]
It should be noted that the sets presem in A-cut ~ = 1 1 1
set will have unity membership and the sets not 1 1 1
in A-cut set have zero membership. Hence A-cut
(c) I.= 0.3,
sets for different values of A can be expressed as
follows. ~
iJ"'
+- +- +- +- I ',,
[0 0 1]
~.,=110
(a) A= 1, A1 =
!1
-
a
0
b
0
c
0
d
0
e .\'
~.
1 1 1

1 1 o o ol (d) A= 0.9,
(b) A= 0.9, Ao.o= -+-+-+-+-
1 a b c d e

0I
[0 0 OJ
1 1 I 0 ~-' = 0 0 0
(c) A= 0.6, Ao.6= -+-+-+-+-
1 a b c d e 0 1 1

1 1 1 1 01 6. For the fuzzy relation B..

-+-+-+-d+-
(d) A=0.3, Ao.3=
1 a b c e
1 0.1 0

(e)A=O+, Ao+=' ! 1 I 1 I 0
-+-+-+-+-
a b c d e
l R=
-
[
0.02 0.1 0.55
0.2 1 0.6
0.5 0.3]
0.6
0
0.03 0.5 1 03 0
(f] A= 0, Ao= ~~+~+~+~+~~
a b c d e find the A-cur relation for A= o+,o.l,0.4
and 0.8.
5. Determine the crisp A-cut relation when A= Solution: For the given fuz.zy relation, the A-cut
0.1, o+' 0.3 and 0.9 for me following relation H: relation can be obtained by rhe following relation:

E=
[
0 0.2 0.4]
0.3 0.7 0.1
R}. = !
1.
O,
f.ll!jx,y)
[email protected])~ A
~A
0.8 0.9 1.0

Solution: For ilie given fuzzy relation, the A-cur

(a) = o+,

relation is given by 11 11 01 11 1]1

R, = \(x,y) \IL~.y) 20A I ~=11110
[1 1 1 1 0
= {1\1'~-> 20A; 0 \l'!ll<y) <A I
324 Oefuzzification

(b) A=O.l, (b) A= 0.4,

1
Ro.i = [' '" "]
0 1 1 1 1
1 1 1 1 0
0 1 1 1 0
,,0 l: !]
(c) A= 0.4, (c) A= 0.7,

[' "" '"]

0 0

iloA=Ol~lO
0 0 1 1 1

0 1 1 0 0
Ro.7 = ~
[
0 0
0
j
(d) A= 0.8, (d) A= 0.9,

1 0 0 0 0] 0 0 0 l 1]
0 0 0 1 0 0 0 0 1 0
Jlo, = 0 1 0 1 0 Ro.o=OOOJO
[ [
0 0 1 0 0 1 1 0 0 0

7. For rhe fuzzy relation [$, 8. Show that any A-cut relation of a fuzz.y mlerance
relation results in a crisp tolerance relation.
0.2 0.5 0.7 Solution: Consider the fuzzy relation
0.3 0.5 0.7 1 0.9]
0.8
R= 0.8 0 0.1 0.2
- 0.4 0.6 0.8 0.9 0.4
[
0.9 1 0.8 0.6 0.4 0.8 1 0.4 0 0.9

find rhe A-cur relation for A = 0.2, 0.4, 0.7

!!= Io 0.4 1 0 0
0.1 0 0 1 0.5
and 0.9
0.2 0.9 0 0.5
Solution: For the given fuzzy relation, rhe A-cur
relation is given by It is a fuzzy tolerance relation because it does not
satisfy transitive properry, i.e.,
1, Jl!Jfx.y) ~A
!Ls_(x~oxtl = 0.8, !Ls_(xt,xs) = 0.9
R; =
1 0,
Jl.@>:,y) <).,
From the relation H. we have
(a) >. = 0.2,
!Ls_(x,,xs) =0.2 (l)

1 1 1 1 1] But on calculating we obtain

l l l 1 1
Roz = 1 1 1 l l
[
l 1 1 1 1 (2)

l.
10.6 Solved Problems 325

As (l) :f.: (2), therefore rransirive properry is nor Now (xt,X2l E R, (x:hx5) E Rand (xt.xs) E R.
~tisfiedr~o:' assume ~hen the crisp r~la Hence, A-cut relation of a fuzzy equivalence relation
non for d 1s 'il . ~ results in a crisp equivalence relation.
uQ 1 o o (oi /
,fl'O. For the giv~hip function as shown in
1 0 0 // Figure I below, determine the defuzzified output
value by seven methods.
Ro.a = 0 0 1 0 0
0 0 0 1 0
/
p

t~ 0; 1 0 0 1

0.7
Now (xt,XJ.l E R, (x~.x;;) E R, bur(xt,X~) ($. 0.7
R(xt ,x,) ~ R. Hence Ru. 11 is a crisp tolerance relation.
Thus /,cut relation for a fuzzy tolerance relawn is a
crisp ro~ano11.
0.5 :f)
o
~.

9. Show that Acur relation of a fuzzy equivalence

relation results in a crisp equivalence relation.
X
2 3 4 5 6
Solution: Consider the following fuzzy equivalence
relation: p

0.8 004 0.5 0.8

0.8 1 0.4 0.5 0.9
!! = 1 o.4 o.4 1 0.4 0.4
0.5 0.5
0.8 0.9
0.4
0.4
1
0.5
0.5
0.5 ,~
~' ".,
<
\~

The rdarion fi satisfies transitive property, i.e.,

,'---~--~,~-- c---o----,----,---x
03 4 5 6
Jl/((x],.\':!) = 0.8, Jll!_(X]..X)) = 0.9
From the rdarion !J.. wr kwc Figure 1 M.:mbc:rship funcrion.s.

/~f/(Xt,X~) = O.H ill Solution: The JdU:a.ifl~J 1l1.HPU1 v:t!ue em he

obtained b~' rhc followin~ rncdwJ~.
l )n calculating we obtain
Cc:nrroiJ method
Jl /,(_(X), xs) = min {JL~ {xi'-'']), IL (X]., X))] The rwo poinrs arc (0. 0) ;tnt.l U. 0.7). '!'he maight
= m;n[0.8, 0.9[ = O.S (2) line isg.ivcn hy (y- Jt) =
m(x-x,). Hence.
As (I)= (2), therefore transitive propcn,v satisfied;
hence it forms an equivalence relation. Now assume y- 0 = (_\"- 0)
i. = 0.8. Then the cnsp relation formed is A11 ~ y = 0.35x
0 0 A~:~ ==> y = 0.7
0 0 A 13 ==> not necessary
Ro.a= io o 1 0 0 A21 => the rWO poims are (2, 0), (3. I)
0 0 0 1 0 y=x-2
0 0 A22 => y = I
326 Defuzzificalion

A23 ::;;} rhe two points are (4, 1}, (6, 0)

wegety =-0.5x+3 [J [1 X 0.7x(3+2) X 2+1 X I

(A) From Atz we obrain y = 0.7. O X (2+4) X 4jdx]

(B) FromAz 1 weobra.iny = x-2. On substituting
rhe value y = 0.7 in (B), we obtain
[J [1 X 0.7x(3 + 2) + 1 X I
x- 2 = 0.7 => x= 2.7
y=0.7 O X (2 + 4) X 4jdx]
6
The centroid merhod defuz.zified output is J (3.5 + 12)dx
x' =I ILC(x)xdx
tLc(x)dx
0
6
J (1.75 + 3)dx
2 .84

0
2.7 3
Center ofla.rgm area:
[l035x'dx + J 0.7xdx+ J (x'- 2)dx
2 2.7 I
Area of I =- 0.7 (2.7 + 0.7) = 1.19
4 6
+ J xdx+ J (-0.5x' + 3x)dx
J
' 2
I
X X

1
3 4 Area of II =- X I X (2 + 3) X - X 0.7
2 2
2 2.7 3
J0.35x'dx + J0.7xdx+J(x'-2)dx = 2.255
[
0 2 2.7
Area of II is found to be larger; therefore rhe

+ J4 dx + J6 (-0.5x' + 3x)dx J defuzzified output value is given by

3 4 , J ILG (x)xdx
X =
- 10.78 J ILQ (x)dx
- 3.445 = 3.187

Weighud average method: The defuz.zified value [J l

2.7,
X 0.3 X 0.3 X 2.85dx
6 ]
here is given by
+j I X I X 3.5dx + j l X 2 X ldx
3 4
2(0.7) + 4(1) = 3.176
x* = 0.7 +1
Mean-mnx method: The crisp ourpur value here
is given by

, a+ b 2.5 + 3.5
X =--=---=3
2 2
= 4.49
Center ofsums method: The defuzzified value x* ' 4 6
is given by J 0.045dx+ J dx+ J dx
2.7 3 4

f Xt ILQ (x)dx First ofmaxima: The defi..ru.ified output value is

:r: i=l
x' x* =3
'
J LILG (x)dx
X i=1 Last ofmf1Xima: The defuzzified output value is
x* =4
f
.

l
10.8 Exercise Problems

I 10.7 Review Questions

8. Explain in derail rhe methods employed for
327

1. Define defuzzificarion.
2. Stare the ne.cessicy of defuzzificarion process. convening fuzzy form into crisp form.

3. Wrire short nme on lambda-cur for fuzz.y ser~. -- 9. Compare first of maxima and last of maxima
method.
4. List rhe propenies oflambda-cur for fuu.y sets.
10. What is the difference between cemroid method
5. How is a fuu.y relation convened into a crisp
and center oflargest area method?
relation using lambda-cut process?
11. Differentiate between center of sums and
6. Mention the properties of lambda-cur for fuzzy
weighted average method.
relations.
12. Which of the seven methods of the defuzzifica-
7. Whar are the different methods of defuzzifica-
cion technique is the best?
tion process?

11 0.8 Exercise Problems

1. Two fuzzy sets defined on X, 4 and fi, are as 3. Consider the two fuzzy sets
follows:

IL (xi)
4.
I
= 0.35 + 0.625 + 0.256
0.7 0.725 0.75 1

I I
Xl X'1. X3 X4 X5 X6 X]

4 0 0.1 0.2 0.3 0.4 0.5 0.6 B= 0.95 + 0.815 + 0.6~

!.! 0.9 0.8 0.7 0.6 0.5 0.4 - 0.7 0.725 0.75
Using Zadeh's nmation, express the fuzzy sets
Express rhe following /,.-cur sets using Zadeh's
notation: as A-cur sets for ),. = 0.2 and ),. = 0.8 for rhe
following operations:
(a) (d)o.2; (b) (J!Io.6; (c) (4 U lllo:
(a) ij; (b) l!: (c) il U 1.!: (d) 4 n 1!:
(d) (dn!l)o.s: (e) (dUd)o.7; (n (4 n dlo.s: (e)<!U!.l: (f)<!n!.l
(g) (I.! u l!Jo o; (h) (!.! n l!Jo;
4. Consider the fuzzy sets
2. Using Zadeh's notation, determine the /,.-cur sets
for the given fuzzy sets: 1 0.8 0.5 0.21

0.1
I
0.4 0.35 0.7 1.0 I ,v = 1100 + 200 + 300 + 400
I
Jjf, = 10+ 20 +30+40+50

M -I~ 0.9 0.84 0.27 + 0.331

S2 = I 0 0.7 0.4 0.1
100 + 200 + 300 + 400

_,_ 10+ 20+ 30 + 40 50 Using Zadeh's notation, express the fuzzy sets
as A-cur sets for A= 0.2i, i = 1 to 5, for the
Express the following for A= 0.2, 0.3 and 0.7: following operations:

(a) (l,j, Uljf,); (b) (14, nljf,); (a);"; (b);"; (c);"n ,V;
(c) (l,j, Uljf,); (d) (14, nljf,); (d);"U,V; (e);& n S,; (f) ;&u ,1,;
(e) (l,j, nljf,); (f) (14, Uljf2); (g)Jjf~: (g) (,V uS,); (h) (,V n ,1,); (i) (3J u ;&J;
(h)Jjf,; (i) (14, uljf,); (j) w, nljf!l (j) (3j n;&J
328 Defuuilication

5. Consider rht: discrete funy ser defined on rhe membership functions:

universe X.:;:: {n, b, C. d, r,JJ a.s
I
Jl,j (x) 1 + 2(x 2}''
o o.; o,2 oA 1 o..l\
B=
l
-+-+-+-+-+-
-~,bcdrf
JL~(x) = r-'.Jl<;(x) = x+4
2x

Using Zadeh's nomion, find rhe i..-cUl sers for

rhevaluesi..= 1.0.7,0.2.0.4,0+ anJO.
{a) Sketch rhc membership functions.
6. Determine rhr cri~p A-cur relation for i.. =
(b) Define rhe imervals along the x-axis corre-
0.1, o~. 0..1. 0.6. 0.7. 1.0 fOr rhe funy relation
sponding to rhe A-cur sers for each of rhe
,uin:n bv. Fuzzy sets cl. fl and (for A= 0.2, 0.4, 0.6,
0.9. 1.0.
I II
0.! 0.1 11.4]
I 0. For rhe logical union of the membership func-
1
0.6 0.7 (U 0.1 0 "&
H= tions shown bdow. find rhe defuzzified value x
-
[
0.8 O.".l 0.6 0.3 0.2
using each of the defuzzificarion methods. ~I
~I
0.1 0 1 0.9 0.7
;~
,;
7. Consider rhe fuzzy rei arion "
0.9 \.II II
I ---------
0..15 0.01 IU

H= I
().()2 0.4-:-
oA

"t(
- 0.6 11.8 11.4
0.1 0 0.2.)
0.68 ()_.., 2 ()_{))

= 0 . 0. I, 0.5, 0.7. --+----+------+' 6

hnJ rht 1.-n1t n:Lnion t(n i. 0
' 3

8. For rhr t"ua: n:btion

0.2i 1\..l) 11.~5 0.6!]

() 1 0.:-l 0.9 "t
H=
- 0.1 0 ..1 0.6 o.-
[
0.4 0 I O.'J
'I
o.s-1-
ti11J rhr i.-lll\ n:lati(llb fi1r i. = O.. t ll."i. 0.
O.'l.0.7.
1
9. The fuzzy se[S .Jj, ~and (are all Jefined on L_._ ---+--+--+----+----+--+----r--
rhe universe X = [0, 5] with rhe tOIIowin~ 0

'

1
 Fuzzy Arithmetic and
Fuzzy Measures 11
Learning Objectives _ _ __:___ _ _ _ _ _ _ _ _ _ _ ___,
Basic concepts of fuzzy arirhme6c. Discusses on extension principle for general-
How interval analysis is performed for uncer- izing crisp sers imo fuzzy sets.
tain values. A description on belief, plausibility, probabil-
A note on fuzzy numbers, fuzzy ordering and ity, possibility and necessity measures.
fuzzy vecrors. Gives a view on fuzzy integrals.

111.1 Introduction
In this chapter, we will discuss the basic concepts involved in fuzzy arithmetic and fuzzy measures. Fuzzy
arirhmeric is based on the operations and computations of fuzzy numbers. Fuzzy numbers help in expressing
fuzzy cardinalities and fuzzy quantifiers. Fuzzy arithmetic is applied in various engineering applicacions when
only imprecise or uncenain sensory clara are available for computation. In this chapter we wilt discuss various
forms of fuzzy measures such as belief, plausibiliry, probabilicy and possibilicy. A representation of uncmainry
can be done using fuzzy measure. All rhe measures to be discussed are functions applied to crisp subsets,
instead of elements, of a universal ser.

111.2 Fuzzy Arithmetic

In the present scenario, we experience many applications which perform compucation using ambiguous
(imprecise) data. In all such casts, the imprecise data from the measuring instruments are generally expressed
in the form of intervals, and suitable mathemacical operations are performed over these intervals ro obtain
a reliable data of rhe measurements (which are also in the form of intervals). This type of computation
is ca!led interval arithmetic or interval analysis. Fuzzy arithmeric is a major concept in possibility theory.
Fuzzy arithmetic is also a mol for dealing with fuzz.y quantifiers in approximate reasoning (Chapter 12). Fuzzy
numbers are an extension of the concept of intervals. Intervals are considered at only one unique level. Fuzzy
numbers consider them at severallevds varying from 0 ro !.

11.2.1 Interval Analysis of Uncertain Values

Consider a data set to be uncertain. We can locate this uncertain value to be lying on a real line, R, inside a
closed interval, i.e., x E ta1, az] where a1 ~ a2. The value of xis greater than or equal to a1 and smaller than
or equal to az. ln interval analysis, the uncertainty of d1e data is limited bcrween the intervals specified by the

1
330 Fuzzy Arithmetic and Fuzzy Measures

Table 111 Set operations on imervals

ConQirions Union, U Intersection, n
dJ>~ [bl> b,] U [a1>a2J
b1>a2 [at, a,] U [b!> b,]
a 1 > b,,az< ~ [bJ, b,] [a,,az]
br> ah bz< az [a,,az] [bJ, b,]
a,<b,<al<h-z [a,' b,] [bl> a,]
b, < dJ < bz< Ill [bl> a,] [a!> b,]

lower bound and upper bound. This can be represented as

~ = [al>az] = {xla1 :;Sx:S azj

where 4 represents an inrerval [a,, a2]. Generally, the values ltJ and ttz are finite. In few cases, a1 = -oo
and/or az = +oo. If value of xis singleton in R then the interval form is x = [x,x]. In general, there are four
cypes of intervals which are as follows:
1. [aJ,az] = {xlal ::S x ::S az} is a closed intervaL
2. [a\, az) = {x Ia! ::S x < 112} is an interval closed atthe'lefr end and open at right end.
3. (a 1 , az) = {x!a 1 < x ::S azj is an interval open adefr end and dosed at right end.
4. (a,, az) = {x \tZJ < x < azJ is an open interval, open at both lefr end and right end.
The set operations performed on the intervals are shown in Table Il-l. Here [llJ, <72] and [bl> bz] are the
upper bounds and lower bounds defined on the two intervals4 and!!., respectively, i.e.,

4= [a!, ttz), where a1 ~ az

!l = [b1, bz], where bt ~ bz

The mathematical operacions performed on inrcrvals are as follows:

1. Addition (+), L" 1J [a!, az] and fi = [b1, bz] be the two intervals defined. If x E [a1, az} and
y E [bJ, bz], rhen
(x+ y] E [a,+ b,,a, + b,]
This can be wriuen as

IJ + ~ = [a~oa,] + [b,,b,] =[a,+ b,,a, + b,]

2. Subtraction (-): The subtraction for the two intervals of confidence is given by

IJ-/l = [a1oa2]- [b,, b,] =[at - b,.a,- biJ

That is, we subtract the larger value out of b1 and bz from a1 and the smaller value our of bt and bz
from az.
3. Multiplication 0: Let the two intervals of confidence be 4 = [aJ, az] and fi = [b1, bz] defined on
non-negarive real line. The multiplication of these two intervals is given by

IJIl= [a,,a,] [b,,b,] =[a, b1oa2 b,]

 11.2 Fuzzy Arithmetic 331

If we multiply an interval with a non-negative real number a, then we get

aIJ= [a,a] [a~oa 2 ] = [aa~oa a2]

a !l =[a ,a] [b,,b,] =[a b~oa b,]
4. Division(-;...): The division of two intervals of confidence defined on a non-negative real line is given by

a1
IJ+Il= [a~oa2] + [b~ob,J = [ b;'b,
a,]
il If b1 = 0 then the upper bound increases tO +oo. If b1 = b2 = 0, then interval of confidence is extended

~I
'!:!
ro +oo.
5. Image (/i): If x E [a!, az] then its image -x E [ -az, -ad. Also if 4 = [a!> az] then irs image
A= [-az; -ad. Note that
~
"}I 4 +4 = [a,,az] + [-az,-ad =[a,- az,az -ad f:. 0
:\I That is, with image concept, the subtraction becomes addition of an image.

l'l 6. Inverse (A- 1}: If x E [a 1, az] is a subset of a positive real line, then its inverse is given by
c;

mE [~~J
Similarly, the inverse of 4 is given by

d-I =[llJ,tZ2] -I = [ l ,-
-
Ill
1
fl]
J
That is, with inverse concept, division becomes multiplication of an inverse. For division by a non-negative
number a> 0, i.e. (I fa) d. we obtain

IJ+a=IJ [!;.!;] = [~.;]

7. Max and min operations: Let two intervals of confidence be 4 = [a1, 112] and /l = [b1, b1]. Their ma..x
and min operations are defined by

Max: 4 v [i = [al>az1 v [b1, b11 = [a1 v b1>a2 v bz]

Min:4A/l= [aJ>az] 1\ [b!>bz] = [n1/\ b1>a2 /1. bz1

The algebraic properties of the intervals are shown in Table 11-2.

Table 112 AJgebraic properties of intervals

Property Addition (+) Multiplication ()

Commutativity il+ll=ll+il llJl=llil

Associativity (IJ+!l) + ~=il + (il+ 0 (IJ !l) . ~ = 11 (Jl ~)
Neutral number IJ+O=O+IJ=il IJl = lIJ=il
Image and inverse IJ+J=J+IJ;I'O llil-l =[ 1 il" 1
332 Fuzzy Arithmetic and Fuzzy Measures

I 11.2.2 Fuzzy Numbers

A fuzzy number is a normal, convex membership function on the real line R. Its membership function is
piecewise concinuous. That is, every A~cut set AJ., A E (0, 1], of a fuz:zy number 4 is a dosed interval of R
and the highest value of membership of 4 is unity. Po~ two given fuzzy numbers 4 and !lin R, for a specific
A1 E [0, 1], we obrain rwo dosed intervals:

(AI)
.dA 1 = [ a 1
(A2)]
, a2
fr om fu uy number 4

#;. 1 = [ b~)q), b;J.z)J from fuzzy number lJ

The interval arithmetic discussed can be applied to both these closed intervals. Fuzzy number is an extension
!I
~
f;
of lhe concept ofimervals.lnstead of accouming intervals at only one unique level, fuzzy numbers consider ~
them at several levels with each of these levels corresponding to each A-cut of the fuzzy numbers. The notation ~:
4;. = [a~}.), ~A)] can be used m represent a closed interval of a fuzzy number .cl at a A~level. '
Let us discuss the interval arithmetic for dosed intervals of fuzzy numbers. let () denote an arithmeric
1:
~;

operation, such as addition, subtraction, muhiplication or division, on fuzzy numbers. The result.cl"' !}., where
.cl and!}. are WO fuzzy numbers is given by :,I
..
. ,
f.LdQ (z) = V [f.Ld (x), f.LQ (y)] .!
z=)."*J

Using extension principle (see Section 11.3), wherex,y E R, for min (A) and max (v) operation, we have

f.LdQ (z) = sup [f.Ld (x) * f.LQ (y)]

z=X*y

Using A~cur, the above nvo equations become

(d * /ih ="'*/b. fm ,11 AE (0,1]

where d;. = [ai'l, a~l J and lb. = [hi.\), b~\ NOe that for IZJ, Ill E [0, l], if llJ > az, then .{!111 C 4112 .
On extending rhc addition and mbmtction operations on intervals to rwo fuzzy numbers 4 and !;! in R,
we get

6>+ lb.=[!, +bj.;, +b}]

!J, - i!> = [;, - b~.;, - b\]
Similarly, on extending rhe mu!tiplicntionand division operations on rwo fuzzy numbers .cl and !}. in f?+
(non-negative real line)= [0, oo), we get

"' lb. = [!,. bj,;, bll

<1>-" lb. = ['1
bl' ~ -1]. bl> 0 i
I
I

L
 11.2 Fuzzy Arithmetic 333

Table 113 Algebraic properties of addition and mulciplication on fuzzy numbers

Property Addition Multiplication

Fuzzy numberS A,B,CcR. A,B,CcR+
Commurarivity A+B=B+.A AB=BA
Associativity (A+B)+C=A+ (B+ C) (AB) C=A (B C)
Neutral number A+O=O+A=A AI=IA=A
Image and inverse A+A=A+A;"O AA- 1 =A- 1 A;"I

The mr1ltipiication of a fuzzy number .cl C R by an ordinary number {3 E R+ can be defined as

(~6h = [~;,.~;,]

The mpport for a fuzzy number, say 4, is given by

supp!J = {x)f.Ld (x)> 0)

which is an imerval on the real line, denoted symbolically as A. The support of the fuzzy number resulting
from the arithmetic operation.d *!}.,i.e.,

supp(z) =A* B
~~
is the arithmetic operation on the nvo individual supports, A and B, for fuzzy numbers 4 and !J., respectively.
In general, arithmetic operations on fuz2y numbers based on A-cut are given by (as mentioned earlier)

VJ~h =A,.B,

The algebraic properties of fu"Z.'Z.Y numbers are listed in Table ll-3. The operations on fu2zy numbers
possess the following properties as well.
1. If A and Bare fu2zy numbers in R, then (A+ B) and (A- B) are also fuzzy numbers. Similarly if A and B
are fuzzy numbers in R+, rhcn (A B) and (A-:- B) are also fuzzy numbers.
2. There exist no image and inverse fuzzy numbers, A and A-l, respectively.
3. The inequalities given below srand true:

(A-B)+B;"A ,nd (A+B)B;"A

I 11.2.3 Fuzzy Ordering

There exist several methods m compare WO fuzzy numbers. The technique for fuzzy ordering is based on the
concept of possibiliry measure.
For a fuzzy number .cl, rwo fuzzy sets ,d 1 andd2 are defined. For this number, the set of numbers that are
possibly greater than or equal tO 4 is denoted as .cit and is defined as

'"~ (w) = n ..:1

(-00, w) =SUP I-'d (u)
u:::w

L
334 Fuzzy Arithmetic arid Fuzzy Measures

"
~_,
------------ -----------
"~

L-~-------L~----~-L----------~R

Figure 111 Fuu.y number 4 and its associated fuzzy sets.

_In a similar manner, the set of numbers rhar are necessarily greater than {! is denoted as {:h and is
defined as

/Ld, (w) = N4(-oo, w) = inf[I-JLd (u)]

II?:W

where nA and NA are possibility and necessity measures (see Section 11.4.3). Figure II-I shows rhc fuzzy
number and irs associated fuzzy sers-t,!1 and 42
When we try to compare two fuzzy numbers 4 and!}, w check whether -cl is greater than fl, we split both
the numbers into their associated fuzz.y sets. We can compare-d with fl1 and fb by index of comparison such
as the possibility or necessity measure of a fuzzy set. That is, we can calculate the possibility and necessity
measures, in the set Ill} of fuzzy sers !b and lh On the basis of this, we obtain four fundamental indices of
comparison which are given below.

1. fl 6 (i!Il =sup min (JLd (u), supJL~ (v)) =sup min(JLd (u), JL~ (v))
u v,::11 u~v

This shows the possibility that the largest value X can take is at least equal ro smallest value that Y can take.
2. fl,(i!z) =sup min (JLJ (u), inf[I-JL~(v)J) =sup inf min (JLd (u), [1-JL~(v)])
u V~/1 II V~U

This shows rhe possibility that the largest value X can rake is greater than the largest value that Y can take.
3. Nd(iil) = inf m"' (1-JLd (v), supJL~ (v)) = infsup m"' (1-JLd (u), JL~ (v))
u v,::u u v.:=rs
This shows the possibility that the smallest value X can take is at least equal to smallest value that Y can rake.
4. Nd(!iz) = inf m"' (1-JLd (u), inf(I-JL~ (v)]) = I - mp min IJL4 (u), JL~ (v)]
II _ i/~U I_ I u.::11

This shows the possibilicy that rhe smallest value X can rake is greater rhan the largest value that Ycan take.
 335
11.2 Fuzzy Arithmetic

I
i
- 11.2.4 Fuzzy Vectors

A vector = (P,, P2, ... , P,) is called a fuzzy vector if for any element we have 0 :5 P; .::;: I fori = I to n.
Similarly, the transpose of the fuzzy vector denoted by e eT'
is a column vector if f. is a row vector, i.e.,

I,.
~
p,
p,
"f e'=
~-~-
,,-.
P,

Let us define!!, and Q as fuzzy vecmrs oflengrh nand !!, QT = V(P; A Qj) as the fuzzy inner product
- - i=l

I
I of Eand g.. Then the fuzzy outer product of eand g. is defined by
fEIJQT =.A' (P;v Q,)
- t==l

The component of rhe fuv.y vector is defined as

e= (I - P,, I - Pz, ... , I - P,) = (7';, Pz, P,,. .. , P,)

The fuzzy complement vector f has rhe constraint 0 .::;: ?; .::;: 1, fori= l ton, and it is also a fuzzy vector.
~
The largest component Pin rhe fuzzy vector Eis defined as irs upper bound, i.e.,
~
P::::::.m;~.x(P;)

The smallest component P of the fuzzy vector !!, is defined by its lower bound, i.e.,
,,
p::;:. m)n{P;)
A '

The propenies that the rwo fuzzy vectors f. and g.. both of length 11, are given as follows:
----T - -T
I.J:g=I:Eilg
2. e Ell gT = gT e.
3. I: gT :o (P" {i)
4. f Ell QT = (p V Q)
- ~ ~

s. r.. eT = P
G. P EIJPT > P
- - - f\

7. If e <; Qrhenf. QT = Pnd ifQ<; fthen e Ell QT = p

- ..... - ..... f\

s. r. . E.::: ~
9. e e E::: ~
336 Fuzzy Arithmelic and Fuzzy Measures

lr should be noted chat when two separate fuzzy vectors are identical, i.e., f.= g,
the inner product f.f},T
reaches a maximum valae while ilie outer product f EEl g_T reaches a minimum value.

'111.3 Extension Principle

Extension principle. was introduced by Zadeh in 1978 and is a very imporrant roo I of fuzzy set theory. This
extension principle allows the generalization of crisp sers into the fuzzy set framework and extends point~
to-poim mappings to m3.ppings for fuzzy sets. This principle allows any function f- that maps an n-ruple
(x,, _x:z, .. , xn) in the crisp set U to a point in the crisp set V - to be generalized as a set that maps n fuzzy
subsets in U to a fuzzy set in V. Thus, any mailiemarical relationship between nonfuzzy crisp elemenrs can be
extended to deal with fuzzy entities. The extension principle is also useful to deal wirh set-theoretic oper3.tions
for higher order fuzzy sets,
Given a function J: M-+ Nand a fuzzy set in M, where
Jl.i J.L2 J.Ln
A=-+-+
- X]X2_
.. +- Xn

the extension principle states that

JLI
fr.&)=f ( -+-+ .. +-
J.L2 J.Ln) =Jil
-+- J.L2
+ " +J.Ln-
- x1 -'2 "" {(xi) {(-'2) f(x")
If[maps several elemems ofM ro the same element yin N (i.e., many~ro-one mapping), then rhe maximum
among their membership grades is taken. That is,

''NI (y) = max [1'6 (x,)]

x;e M
fl~;) "'!

where x/s are the elements mapped to same element y. The function Jmaps n-ruples in M w a poim inN.
Let M be the Cartesian proauct of universes M = M 1 x M2 x x M, and t.h ,.42, ... ,4 71 ben fuzzy
sers in M1, M2, ... , M,, respectively. The function Jmaps an n~ruplc (x!, x2, ... , x11) in the crisp set M to a
pointy in the crisp set V, i.e., y = J(xJ, .'Q, .. ,, x11 ). The function J(xl, "2 ... , x,) m be extended co act on
then fuzzy subsets of M,4J,42 ... ,4, is permitted by the extension principle such that

L= fW
where lis the fuzzy image of41 ,42, ... ,.;1, through/(). The fuzzy set !J is defined by

!! = {(y, I'~ (y)){y = f(x" -'2 ... , x"), (x,, -'2, . ,x,) E MJ

where

I'~ (y) = sup m;nll'd, (xi), 1'~, (-':!),. "1'6. (x")]

(.<1"2 -"n)EM
'"''(~1"2----"")

with a condition that J.Lf!. (y) = 0 if there exists no (xJ, xz, ... , x11) E M such that y = j(xt, X2_, , x,).
The extension principle helps in propagating fuzziness through generalized relations that are discrete
mappings of ordered pairs of elements from input universes to ordered pairs of elements from other universe.
The extension principle is also useful for mapping fuzzy inputs through cominuous-valued functions. The
process employed is same as for a discrete-valued function, but it involves more computation.

I
l
11.4 Fuzzy Measures 337

111.4 Fuzzy Measures

A fuzzy measure explains the imprecision or ambi~ic}r.in ~e assignment of an element a to rwo or more crisp
sets. For representing uncenainty condition, knoWn a$.ambiguity, we assign a value in the unit imerval [0, 1)
ro each possible crisp set to which the element in th~ problem might belong. The value assigned represents
the degree of evidence or ce_nainty or belief of the element's ffiembership in the set. The representation of
uncertainty of this manner is called fuzzy measure. In sum, a fuzzy measure assigns a value. in the unit interval
[0, 1) to each classical set of the universal set signifying the degree of belief that a particular elementx belongs
to the crisp set. In this section several diffefent fuzzy measures such as belief measures, plausibility measure,
probability measure, necessity measure and possibility measure are covered. All these measures are functions
applied ro crisp subsets, instead of elements of a universal set.
The difference berween a fuzzy measure and a fuzzy set on a universe of elements is that, in fuZzy measure,
the imprecision is in the assignment of an element to one of two or more crisp sets, and in fuzzy sets, the
imprecision is in the prescription of the boundaries of a set.
A fuzzy measure is defined by a function

g: P(X) -> [0, 1]

which assigns to each crisp subset of a universe of discourse X a number in the unit imerval [0, 1], where
P(X) is power set of X A fu:u.y measure is obviously a set function. To qualify a fuzzy measure, the function
g should possess cCnain properties. A fuzzy measure is also described as follows:

g: B-> [0, 1]

where B C P(X) is a family of crisp subsets of X Here B is a Borel field or a a field. Also, g satisfies rhe
following three axioms of fuzzy measures:
Axiom 1: Boundary Conditions (gl)

g() = 0; g(X) = 1

Axiom 2: Monotoniciry (g2) - For every classical set A, BE P{X), if A ~ B, then g(A) S g(IJ).
Axiom3: Continuity (g3)- For each sequence (A; E P(X)Ii E N) of subsets ofX, if either At ~ A2 ~
or A1 2 A2 2 ... , then

Hm g(A,) =g(,l;m A,)

1-tOO 1-tOO

where N is the set of all positive integers.

A a field or Borel field satisfies the following propercies:
L XEBandEB.
2. If A E B1 then :A E B.
3. B is dosed under set union operation, i.e., if A E Band BE B (a field), then AU B E B (a fteld)

The fuzzy measure excludes the additive property of standard measures, h. The additive property states
that when rwo sees A and Bare disjoint, then

h(A U B) = h(A) + h(lf)

 338 Fuzzy Arithmetic and Fuzzy Measures

The ~robabiliry measure possesses this additive property: Fuzzy measures are also defined by another weaker
axiom: subadditivicy. The other basic properties of fuzzy measures are the following:

1. Since 4 4 U f!. and /1 .d U f!., and because fuzzy measure g possesses monotonic pro percy, we have
g(d U !lJ 2: max[g(d),g(flj]
2. Since 4 r\) f!. 4 and 4 n f!. fi, and becawe fuzzy measure g possesses monotonic property, we have
g(d n !lJ :5 min[g(d),g(flj]

I 11.4.1 B~liel and Plausibility Measures

The belief measUfe is a fuzzy measure rhar saisfies three axioms gl, g2 and g3 and an additional axiom of
subaddiriviry. A belief measure is a function

bel 'B-+ [0,1]

satisfying axioms gl, g2 and g3 of fuzzy measures and subadditivity axiom. It is defined as follows:
'
bel (AI uA, u .. UA,.) 2:Lbel (A;)- Lbel (A;nAj)
i<j

+ ... +(-!)"-'bel (A 1 nA2 n ... nA,)

for every n E Nand every collection of subse[S of X Nis set of all positive imcger. This is called axiom 4 (g4).
For n = 2, g4 is of the form

bel (A, UA,) 2: bel (A,)+ bel (A,)- bel (A, nA,)

For 11 = 2, if A1 =A and A2 =A, axiom g4 indicates

bel (A 1 UA 2) =bel (AUA)

bel (A u A) 2: bel (A) + bel (A) - bel (A n A)

Since A u.A = XandAnA =,we have

bel (X) 2: bel (A) + bel (A)

bel (A) + bel (A) S l

On the basis of rhe belief measure, one can define a plausibility measure PI as

PI (A) = l - bel (A)

for all A E B(CP(X)). On the other hand, based on plausibility measure, belief measure can be defined as

bel (A) = l - PI (A)

Plausibilicy measure can also be defined independent of belief measure. A plausibility measure is a function

PLB-+ [0,1]
 11.4 Fuzzy Measures 339

satisfying axioms gl, g2, g3 of fuzzy measures and ~e following additional subadditivity axiom {axiom gS):

~~,n~n -n~sr:~w-r:~~u~
. ~

+ + (-1)"- 1 PI (A, UA, U UA,)

for every n E Nand all collection of subsets of X For n = 2, consider A1 =A and A2 =A, then we have
<
PI (A nil) S PI~)+ PI (A)- PI(A UA)
=> PI(A)+PI(A)~ l

The belief measure and the plausibitlcy measure are mutually dual, so ir will be beneficial to express both
of them in terms of a set funaion m, called a basic probability assignment. The basic probability assignment
m is a set function,

m: B-+ [0,1]

such that m( = 0) ;md LA E m(A) = 1. The basic probabilicy assignments are not fuzzy measures. The
0
quamiry m(A) E [0, J],A E B(CP(X)), is called A's basic probability number. Given a basic assignment m,
a belief measure and a plausibility measure can be uniquely determined by

bel (A) = L m(B)

Bt;A

PI (A) = L m(B)
BnA;!O

foc ,]I A E B(CP(X)).

The relations among m(A), bei(A) and PI(A) are as follows:
l. m(A) measures the belief that the element (x E X) belongs to set A alone, not the roral belief that the
element commits in A.
2. bel {A) indicates total evidence that rhc element (x e X) belongs ro set A and to any other special subsets of A
3. PI (A) includes d~e tmal evidence that the element (x eX) belongs w set A or to other special subsets of
f A plus the additiona1 evidence or belief associated with sets that overlap with A.

Based on these relations, we have

PI(A) 2: bel (A) 2: m(A) VA E B (u field)

Belief and plausibility measure are dual ro each other. The corresponding basic assignment m can be
obtained from a given plausibilicy measure PI:

m~) = L (-J)IA-BI[J -PI (B}] VA E B (u field)

B<;;A

Every setA E B(CP(X)) for which m{A) > 0 is called a focal element of m. Focal elemeQts are subsets of X
II on which ilie available evidence focuses.

'
340

I 11.4.2 ProbabilitY Measures

Fuzzy Arithmetic and Fuzzy Measures

On replacing the axiom of subadditivity (axiom g4) with a monger axiom of additivity (axiom g6),
l
I

bel (AU B) = bei(A) + bel (B) whenever An B = </J; A, B E B(a field)

we get dte crisp probability measures (or Bayesian belief measures). In other words, che belief measure becomes
the crisp probability measure under the additive axiom.
A probability measure is a function

P:B--> [0,1]

satisfying ilie three axioms gl, g2 and g3 of fuzzy measures and the additivity axiom (axiom g6) as follows:

P(A U B) = P(A) + P(B) whenever A n B = </J .A. B E B

With axiom g6, the theorem given below relates the belief measure and dte basic assignment to the probability
measure.
"A beliifmeasure bel on afinite u-field B, which is a mbSI!tofP(X), is a probability measure ifand only ifits
bttSic probability arsignment m is given by m({x}) = bel ({x}) and m(A) = 0 for ali subsets ofX that are not
singletons."
The theorem mentioned is very significant. The theorem indicates fiat a probability measure on finite
sers can be represented uniquely by a hmcrion defined on ilie elements of dte universal set X rather than irs
subsets. The probability measures on finite sets can be fully represented by a function,

P: X--> [0, 1] such tha< P(x) = m([x})

This function P(X) is called probability distribution function. Within probability measure, ilie total
ignorance is expressed by the uniform probability distribution function
1
P(x) = m([x}) = IX] fm all x EX

The plausibility and belief measures can be viewed as upper and lower probabilities that characterize a set
of probability measures.

11.4.3 Possibility and Necessity Measures

In this section, let us discuss two subclasses of belief and plausibility measures, which focus on nested focal
elements. A group of subsets of a universal set is nested if these subsets can be ordered in a way that each is
contained in the ne:n; i.e.,A 1 C Az C A3 C C A 11 ,A; E P(X) are nested sets. When the focal elements of
a body of evidence(, m) are nested, the linked belief and plausibility measures are called consonants, because
here the degrees of evidence allocated to them do not conflict with each other. The belief and plausibility
measures are characterized by ilie following theorem:
Theorem: Consider a comorumr body o/rvidence (E, m), rhe a.ssociated consonant beliefand pkzusibj/ity mea.sum
posses the fo/Wwing propertitJ;
bel (An B)= min [bel (A), bel (B))
PI (AU B) = max [PI (A), PI(B)] '
forai/A 1B E B(CP(X)).
I

L
 11.4 Fuzzy Measures 341

Consonant belief and plausibility measures are referred to as necessity and possibility measures and are
denoted by Nand TI. respectively. The possibility and necessity measures are defined independently as follows:
TI
The possibility measure and necessity measure N are funetions

TI :B ~ [0,1]
N' B--> [0, 1]

such fiat both TI and N satisfy the axioms gl. g2 and g3 of fuzzy measures and the following additional
axiom (g7):

f1<A U B)= max !f1(A), f1(B)) VA,B E B

N(A n B) = min(N(A), N(B)) VA, B E B

~ necessity and possibility measures are special subclasses ofbelief and plausibility measures, respectively,
they are related to each other by

f1<A) = 1- N(A)
N(A) = 1- f1<A) VA ea field

The properties given below are based on the axiom g7 and above set of equations.

1. min[N(A), N(A)] = N(A n A) = 0. This implies that A or A is nor necessary at all.

2. max0(A), n<A)] = n<A UA) = n<Xl = 1. This implies thateithet A 0[ A is completely possible.
3. n (A) ~ N(A) VA <;a field.
4. If N(A) > 0 then n(A) = 1 and if n (A)< 1 then N{A) = 0.
The two equations indicate that if an event is necessary then it is completely possible. If it is not completely
possible then it is nat necessary. Every possibility measure TI on B c P(x) can be uniquely determined by a
possibility distribution function

f1, ' __. [o,11

using the formula

TI (A) = maxf1!x)
.>:EA
Vx E a field

The necessity and possibility measure are mutually dual with each other. & a result we can obtain the
necessity measure from the possibility distribution funccion. This is given as

N(A) = 1- f1!AJ = 1- max f1!x)

x4A

The total ignorance can be expressed in terms of the possibility distribution by n(x11) = 1 and TI(x,) = 0
fori= I ton- 1, COtresponding to n<A.) = n(X) = 1 and n<A) = 0.

L
 342 Fuzzy Arithmetic and Fuzzy Measures

111.5 Measures of Fuzziness

The concept of fuzzy sets is a base frame for dealing with vagueness. In particular, the fuzzy measures concept
provides a general mathematical framework to deal with ambiguous variables. Thus, fuzzy sets and fuzzy
measures are tools for representing these ambiguous situations. Measures of uncercainry related to vagueness
are referred ro as measures of fuzziness.
Generally, a measure of fuzziness is a function

f: fJ.X) --> R

where R is the real line and F{x) is the set of all fuzzy subsets of X. T}{e function f satisfies the following
axioms:

I. Axiom 1 (fi): f(A) = 0 if and only if A is a crisp set.

2. Axiom 2 (f2): If A (shp) B, then f(A) :S j(B), where A (shp) B denotes thar A is sharper than B.
3. Ariom 3 (f:l): /(A) rakes rhe mrucimum value if and only if A is mrucimally fuzzy.
Axiom fl shows that a crisp lser has zero degree of fuzziness in it. Axioms f2 and f3 are based on concepr
of"sharper" and "maximal fuz:zr," respectively.

I. The first fu7..z.y measure can be de'fined by the function:

~-
/(A)=- L (!LA (x) log2 [!LA (x)] [l-ILA (x)]log,[l-ILA (x)Jj
EA
Ir can be normalized as

/'(A) =/(A)
lxl
where lxl is cardinality of universal set X. This measure of fuzziness can be con:sidered as the entropy of a
fuzzy set
2. A (shp) B, A is sharper rhan B, is defined as

!LA (x) :S ILB (x) for ILB 5 0.5

!LA (x) 2o ILB (x) fur 1LB (x)2o 0.5 Vx EX
3. A is maximally fuzzy if

!LA (x) = 0.5 for all x EX

111.6 Fuzzy Integrals

Sugeno in the year 1977 defined fuzzy integral using fuzzy measures based on a Lebesgue integral, which is
defined using "measures."
Let Kbe a mapping from X to[0,1]. The fuzzy integral, in the sense of fuzzy measure g, of Kover a subset
A of X is defined a's

1 A
K(x) g = sup
etE(O,l]
min[~ ,g(A n Hp)]
 11.6 Solved Problems 343

where Hf3 ;:::; {x E xiK(x) =::_p}. Here, A is called the domain of integration. If k ==a E [0, 1] is a constant,
then its fuzzy integral over Xis "a" itself, becauseg(XnH,a) ;:::; 1 for p ~ a andg(XnHf3) :::::: 0 for {3 >a, i.e.,

[ag;,a, aE [0,1]

Consider X to be a finite set such that X::::: {XJ,X'2, ... ,xn}. Without loss of generality, assuming the
funccion to be integrated, k can be obtained such that k(x1) =::_ k(xt) ~ ~ k(xn) This is obtained after
proper ordering. The basic fuzzy integral then becomes

1
X
k(x): g = max mio[k(x,),g(H,)]
i=l[on

where H; = {xt, X'l, .. , x;}. The calcuJadcin of the fuzzy measure "i' is a fundamental point in performing
a fuzzy integmion.

111.7 Summary
In iliis chapter we discussed fozzy arithmetic, which is considered as an exiension of interval arithmecic. The
chapter provides a general methodology for extending crisp concepts to address fuzzy quantities, such as
real algebraic operations on fuzzy numbers. One of the imponam tools of fuzzy set theory introduced by
Zadeh is the extension principle, which allows any mathematical relationship berween non fuzzy dements to
be extended ro fuzzy enticies. This principle can be applied to algebraic operations to define ser.rheoretic
operations for higher order fuzzy sets. The operations and properties of fuzzy vectors were discussed in this
chapter for their use in similarity mercies. Also, we have discussed the concept of fuzzy measures and the
axioms that must be satisfied by a set function in order for it to be a fuzzy measure. We also discuss belief
and plausibility measures which are based on the dual axioms of subadditiviry. The be(ief and plausibility
measures can be expressed by the basic probability assignment m, which assigns degree of evidence or belief
indicating that a particular dement of X belongs only to set A and not to any subset of A. Focal elements
are the subsets that are assigned with nonzero degrees of evidence. The main characteristic of probability
measures is tim each of them can be distinctly represented by a probability distribution function defined on
the elements of a universal set apart from its subsets. Also the necessity and possibility meMures, which are
consonam belief measures and consonant plausibility measures, respectively, are characterized distinctly by
functions defined on the elements of the universal ser rath.er than on its subsets. The fuzzy integrals defined
by Sugeno (1977) are also discussed. Fuzzy integrals are used to perform integration of fuzzy functions. The
measures of fuzziness were also discussed. The defmirions of measures of fuzziness dealt in this chapter can
be extended to noninfinite supports by replacing the summation by integration appropriately.

I 11.8 Solved Problems

1. Perform the following operations on intervals: (a) [3, 2] + [4,3] = + [b1, b,]
[a1,a2J
(a) [3, 2] + [4, 3] (b) [2, 1) X [1, 3)
= [a1 + b~oa2 + b,]
= [3+4,2+3] = [7,5]
(c) [4,6]-i- [1,2] (d)[3, 5] - [4, 5]
(b) [2, 1] x [1,3]= [a1,a2l [b~ob,J
Solution: The operations were performed on ilie basis = [a1 b1,a2 b,]
i of the interval analySis. = [2 . 1' l . 3] = [2, 3]
I
'
I
344 Fuzzy Arithmetic and Fuzzy Measures

(c) [4, 6] + [1, 2] = [aJ, az] + [b,. bz] Solution:

=[~.] 1
--
+
1 = (
0.5
012
1
-+-+- + (0.5
0.5) 1
-+-+-
012
0.5)

= [i ~] = [2,6]
l,= (
min(0.5, 0.5)
0
(d) [3, 5] - [4, 5] = [aJ, az] - [b 1, b,] max[min(0.5, 1), min(!, 0.5)]
= [a 1 - b,, a2 - bJ] + 1
=[3-5,5-4]=[-2,1] max [min(0.5, 0.5), min(!, 1),
2. For the imerval4 = [5, 3], find irs image and min(0.5, 0.5)]
inverse. + 2
max[min(1, 0.5), min(0.5, 1)]
Solution: The given interval is
+ 3
4 = [5,3] = [aJ,az]
+
min(0.5, 0.5)
4
l
(a) ImogeA = [-az, -aJ] = [-3, -5]
0.5 max[0.5, 0.5] max[0.5, 1, 0.5]
(b) Inverse A-I = [_1_, _1_] = [~)] = ( o-+ 1 + 2
az a,
= [0.333. 0.2]
3 5

+
max[0.5, 0.5]
3 +
min(0.5, 0.5)
4
l
3. Given the two imeiVals . == [2, 4], E= [ -4, 5],
perform the max and min operations over these 0.5 0.5 1 0.5 0.51
2 = ( -+-+-+-+-
intervals. - 0 0 2 3 4

Solution: The given intervals are . = [a,, az] = S. The two fuzzy vecwrs of length 4 are defined as
[2, 4] and E= [bJ, bz] = [ -4, 5].
~ = (0.5, 0.2. 1.0, 0.8)
(a) Max operarion
and lz.. = (0.8, 0.1, 0.9, 0.3)
f. v E= [a,. a,] v [bJ,bz] = [a 1 v b1 ,a, v bz]
Find the inner product and curer product for these
= [2 v -4,4v 5] = [2,5]
two fuzzy vectors.
(b) Min operation Solution:

f. 1\ E= [a,. az]l\ [b,. b,] = [2, 4]/\ [-4, 5] (a) Inner product:
= [2 1\ -4. 4 1\ 5] = [ -4, 4]

4. Consider a fuzzy number l, the normal convex 0.8)

T 0.1
membership function defined on integers ~. lz.. = (0.5, 0.2, 1.0, 0.8) 0.9
(
0.5 1 0.5) 0.3
l = ( -+-+-
0 1 2 = (0.5 1\ 0.8) v (0.2/\ 0.1) v (1.0 1\ 0.9)
v (0.8 1\ 0.3)
Perform addition of rnro fuzzy numbers, i.e., add
! to l using extension principle. = 0.5 v 0.1 v 0.9 v 0.3 = 0.9

""--
 11.9 Review Questions 345

(b) Outn product. Solution: The belief measures are obtained as follows:

bei(P) = m(P) = 0.04

0.8)
aEB 1z..T = (0.5, o.2. 1.0. o.8J 0.1
o.
bei(B) = m(B) = 0.04
9
( bel() = m(E) = 0.04
0.3

= (0.5 v 0.8) 1\ (0.2 v 0.1) bei(P U B) = m(P U B) + m(P) + m(B)

1\ (1.0 v 0.9) 1\ (0.8 v 0<3) = 0.12 + 0.04 + 0.04 = 0.2

= (0.8) 1\ (0.2) 1\ (1.0) 1\ (0.8) = 0.2 bel(P U ) = m(P U ) + m(P) + m(E)

6. Let X be the universal set and let A, B, and C be = 0.08 + 0.04 + 0.04 = 0.16
the subsets of X The basic assignments for the
bel(B U ) = m(B U ) + m(B) + m(E)
corresponding focal elemenrs are mentioned in
the following table. Determine rhe corresponding = 0.04 + 0.04 + 0.04 = 0.12
belief measure.
bei(PUBU) = m(PU BUE) + m(PU B)
Focal elements m()
+ m(PUE) + m(BUE)
p 0.04
B 0.04 + m(P) + m(B) + m(E)
E 0.04
= 0.64 + 0.12 + 0.08 + 0.04
PUB 0.12
PUE 0.08 + 0.04 + 0.04 + 0.04
BUE 0.04
PUBUE 0.64 = 1.0

I 11.9 Review Questions

1. Stare the importance of fuzzy arithmetic. 9. Explain in derail the concept of fuzzy vectors.
2. How is an interval analysis obtained in fuzzy 10. State the extension principle in fuzzy set theory.
arithmetic?
11. What are fuzzy measures?
3. List the set operations performed on intervals.
12. Explain in detail the belief and plausibility
4. Discuss the mathematical operations performed measures.
on intervals.
13. How are necessity and possibility measures
5. What are the properties of performing addition obtained from belief and plausibility measures?
and multiplication on intervals?
14. Discuss in detail:
6. Define fuzzy numbers.
Probability measure;
7. Mention the properties of addition and multi
FU7.Zy imegrals.
plication on fuzzy numbers.
8. Write short note on fuzzy ordering. 15. Mention the measures of fuuiness in derail.

--
346 Fuzzy Arithmetic and Fuzzy Measures

I 11.10 Exercise Problems

1. Perform the following operations on intervals 6. Consider the three fuzzy sets 4, ~ and Cand
their membership functions:
(a) [5, 3] + [4, 2] (b) [6, 9] - [2, 4]
(c) [I, 2]
(e) [5, 3]
X [5, 3] . (d) [7, 3]+ [3, 6]
(f) [6, 5]-l ILJ (x) = 1+\Ox' C
!L~ (x) = ~.)
2. Perform the max and min operations over the
intervals f =
[5, 6] and G [9, 2].
3. Given the following
=
fuzzy numb~rs and using
J'((X) = c~2x )
0.5

Zadeh's extension. principle, calculate K = 4 /i Order .the fuzzy sets. Take x 2; 0.

and show why l 0 is nonconvex. 7. The two fuzzy vectors of length 6 are defmed as
0.2 I 0.1
4 =~=2+3+4 ~ = (0.5, 0.7, 0.2, 0.3, I, 0.8)

B=, = ~ + ~ + 0.2
it..= (0, 0.2, 0.1, 0.4, 0.6, 1.0)
- I 2 3
Find the inner product and curer product of two
4. Given vectors.
0.4 I 0.4
A=-+-+- 8. Determine the corresponding belief and plausi-
- 0.2 0.4 0.6 bility measures from the table below:
I 0.4 0.5
B=-+-+-
- 0.2 0.4 0.6 Focal elements m

calculate the following: 4 + ,, 4 - !},, p 0.05

4!l.4+!l. B 0.05
S. For the two triangular fuzzy numbers and Ji, ,a E 0.05
whose membership functions are respectively PUB 0.50
PUE 0.15

xt
2-x if-lsx:::o BUE 0.05
PUBUE 0.15
/LJ(x)= if 0::: x::: 0

1 otherwise 9. Consider the possibility disrribution induced by

the proposition "xis an even integer" is
x+ I

I
if-l~xso

!L~(x) = 3ix if o::::x::::o n = ((1. 1).(2.3).(3.0.5).(4.0.4).

X
otherwise (5, 0.6), (6, 0.3)}
compute the following:
If A = {l, 2, 3) is a crisp ser, then find the
(a) 4+!l, 4-!l possibility and necessity measures of A.
(b) 4 A !l, 4 V !l 10. With suitable example, show that the maximum
(c) 4c-!l, 4c-il measure of fuzziness is lXI.
!

Fuzzy Rule Base and Approximate

Reasoning 12
Learning Objectives
Discusses on various fuzzy propositions. Different modes of fuzzy approximate
This chapter gives an idea of how to form reasoning.
the fuzzy rules, decompose and aggregate A note on fuzzy inference system and irs types.
them. An overview of fuzzy expert system.

112.1 Introduction
This chapter focuses on formation of fuzzy rules and reasoning. The degree of an element in a fuzzy ser
corresponds ro the truth value of a proposition in fuzzy logic systems. The chapter continues with using
natural language in the expression of various knowledge forms; such systems are known as rule-based systems.
Thereaft:er we address concepts such as formation, decomposition and aggregation of fuzzy rules. We explore
and discuss nor only the different modes of fuzzy reasoning but also introduce the basic concepts of fuu.y
inference system, along with irs rwo different types. The chapter closes with a basic overview of fuzzy expert
system.

112.2 Truth Values and Tables in Fuzzy Logic

Fuu.y logic uses linguistic variables. The values of a linguistic variable are words or sentences in a natural or
artificial language. For example, height is a linguistic variable if it rakes values such as tall, medium, short
and so on. The linguistic variable provides approximate characterization of a complex problem. The name of
the variable, the universe of discourse and a fuu.y subset of universe of discourse characterize a fuzzy variable.
A linguistic variable is a variable of a higher order than a finzy variable and its values are taken ro be fuzzy
variables. A linguistic variable is characterized by
1. name of the variable (x);
2. term set of the variable t(x);
3. syntactic rule for generating the values of x;
4. semantic rule for associating each value of x with its meaning.
Apart from the linguistic variables, there exists what are called as linguistic hedges (linguistic modifiers). For
example, in the fuzzy set "very tall", the word "very" is a linguistic hedge. A few popular linguistic hedges
include: very, highly, slightly, moderately, plus, minus, fairly, rather.
348 Fuzzy Rule Base and Approximate Reasoning

&asoning has logic as irs basis, whereas propositions are text sentences expressed in any language and are
generally expressed in an caponical form as

zisP

where z is the symbol of the subject and P is the predicate designing the characteristics of the subject. For
example, "London is in United IGngdom" is a proposition in which "London" is the subject and "in United
Kingdom" is the predicate, which specifies a property of''London," i.e., its geographicallocacion in United
Kingdom. Every proposition has its opposite, called negation. For assuming opposite rruclt values, a proJX>sicion
and its negation are required.
Truth cables define logic functions of two propositions. Let X and Ybe two propositions, either of which
can be true or false. The basic logic operations performed over the propositions are the following:
I. Conjunction (A) : X AND Y.
2. Disjunction (v) : XOR Y.
3. Implication or conditional(=>): IF X THEN Y.
4. Bidim:tional or equivaknce (> ): X IF AND ONLY IF Y.
On the basis of these operations on propositions, inference rules can be formulated. Few inference rules
are as follows:

[Xi\ (X:; Y)J:; Y

[Y" (X:; Y)J:; X
[(X:; Y) i\ (Y:; Z)]:; (X:; Z)

The above rules produce certain propositions that are alwar.; true irrespective of the truth values of
propositions X and Y. Such propositions are called tautologies. An extension of set-theoretic bivalence logic is
the fuzzy logic where the trmh values are terms of the linguistic variable "rruth."
The truth values of propositions in fuzzy lOgic are allowed to range over the unit inrerval [0, I]. A trmh
value in fuzzy logic "very true" may be interpreted as a fuzzy set in [0, I]. The truth value of the proposition
'' Z is A," or simply the truth value ofA, denoted by rv(A) is defined by a poinr in [0, 1] (called the numerical
truili value} or a fuzzy set in [0, 1) (called the linguistic truth value).
The rruth value of a proposition can be obtained from the logic operations of other propositions whose
truth values are known. If rv(X) and rv(Y) are numerical rruth values of propositions X and Y, respecrively,
men

rv(XAND Y) = rv(X) 1\ rv(Y) =min (rv(X), rv(Y)] (!nte<Sec<ion)

rv(XOR Y) = tv(X) v rv(Y) =max (tv(X), rv(Y)J (Union)
rv(NOT X) = I - rv(X) (Complemenc)
rv(X:; Y) = rv(X) :; tv(B) = max (I - tv(X), min [rv(X), rv(Y)]}

I 12.3 Fuzzy Propositions

For extending the reasoning capability, fuzzy logic uses fuzzy predicates, fuzzy-predicare modifiers, fuzzy
quantifiers and fuzzy qualifiers in the fuzzy propositions. The fuzzy propositions make the fuzzy logic differ

L
 12.4 Formation of Rules 349

from classical logic. The fuzzy propositions are as follows:

1. Fuzzy predicates: In fuzzy logic the predicares'can be fuzzy, for example, rail, shon,quick. Hence, we have
proposition like "Peter is tall." It is obvious that moSt of the predicates in narurallanguage are fuzzy rather
than crisp.
2. Fuzzy-predicate modifiers: In fuzzy logic, there exlsrs a wide range of predicate modifiers that act as hedges,
for example, very, fairly, moderately, rather, slightly. These predicate modifiers are necessary for generating
the values of a linguistic variable. An example can be the proposition "Climate is moderately cool," where
"moderately" is the fuzzy predicate modifier.
3. Fuzzy quantifiers: The fuzzy quantifiers such as most, several, many, &equencly are used in fuzzy logic.
Employing iliese, we can have proposition like "Many people are educated." A fuzzy quantifier can be
interpreted as a fuzzy number or a fuzzy proposition, which provides an imprecise characterization of the
cardinality of one or more futty or nonfuzz.y sers. Fll1Z}' quantifiers can be used to represent the meaning
of propositions containing prqbabilities; as a result, they can be used ro manipulate probabilities within
fuzzy logic.
4. Fu:zzy qualifiers: There are four modes of qualification in fuu:y logic, which are as follows:
Fuzzy troth qualification: It is expressed as "xis r ," in which r is a fuzzy truth value. A fuzzy truth
value daims the degree of truth of a fuzzy proposition. Consider the example,
(Paul is Young) is NOT VERY True.
Here the qualified proposition is (Paul is Young) and the qualifying fuzzy truth value is "NOT Very True."
Fuzzy probability qualification: It is denoted as "xis).," where). is fuzzy probability. In conventional
logic, probability is either numerical or an interval. In fuzzy logic, fuzzy probability is expressed by
terms such as likely, very likely, unlikely, around and so on. Consider the example,
(Paul is Young) is Likely.
Here th~ qualifYing fuzzy probability is "Likely." These probabilities may be interpreted as fuzzy
numbers, which may be manipulated using fuzzy arithmetic.
Fuzzy possibility qunlification: It is expressed as "xis 7r ,"where 7r is a fuzzy possibility and can be of the
following forms: possible, quire possible, almost impossible. These values can be imerprered as labels
of fuzzy subsets of the real line. Consider the example
(Paul is Young) is Almost Impossible.
Here the qualifYing fuzzy possibility is "Almost Impossible."
Fuzzy usua/ity qruz!ification: It is expressed as "usually (X) =usually (Xis F)," in which the subject X
is a variable raking values in a universe of discourse U and ilie predicate F is a fuzzy subset of U and
interpreted as a usual value of X denoted by U(X) = F. The propositions that are usually true or rhe
events that have high probability of occurrence are related by the concept of usuality qualification.

112.4 Formation of Rules

The general way of representing human knowledge is by forming natural language expressions given by
IF antecedant THEN consequent.
The above expression is referred ro as rhe IFTHEN rulebased form. There are three generaJ forms that exist
for any linguistic variable. They are: (a} assignment statements; (b) conditional statements; (c) unconditional
statements.

L
350 Fuzzy Rule Base and Appro:o:.imale Reasoning

Ta~le 121 The canonical form of fuzzy rulebased system

Rule 1: H conamon !=I 1HEN restri~ion 81
Rul< z, If condition !;2. THEN restriccion &

Rule n: If condition{;,, THEN restriction &

1. ksignment statements: They are of the form

y =small
Orange color = orange
a=s
Paul is not tall and not very short
Climate = autumn
Ourside temperature = normal

These statements milize "=" for assignment.

2. ConditionaL statements: The following are some examples.

IF y is very cool THEN stop.

IF A is high THEN B is low ELSE B is nm low.
IF temperature is high THEN climate is hor.

The conditional staremems use the "JF.THEN" rule-based form.

3. Unc01rditional statements: They can be of the form

Goro sum.
Swp.
Divide by a.
Turn the pre5sure low.

The assignment statements limit the value of a variable to a specific quantity. The canonical rule formation
for a fuzzy rule-based system is given in Table 12-1. Generally, bolh unconditional as well as conditional
statements place some restrictions on the consequent of the rule-based process. Fuzzy sets and relations
generally. model the restric6ons. The restriction statements, irrespective of conditional or unconditional
statements, are usually connected by linguistic connectives such as "and," "or" or "else." The restrictions
denoted by R1, R1, ... , R11 apply co the consequent of the rules.

112.5 Decomposition of Rules (Compound Rules)

A compound rule is a collection of many simple mles combined together. Any compound rule structure
may be decomposed and redua;d to a number of simple canonical rule forms. The rules are generally based
on natural language representations. The following are the methods used for decomposition of c~mpound
linguistic rules into simple canonical ~es.
I

I 12.5 Decomposition of Rules (Compound Rules) 351

I 1. Multiple conjunctive antecedents

IFxis{!J,.dl. ~- .dn THENyisB111
Assume a new fuzzy subset.dm defined as

.dm =.dl n.,12 n ... n.d/1

and expressed by means of membership function

1-'d.(x) = min [1-'d,(x), l-'d 2 (x), ... 1-'d.(x)].

In view of the fuzzy intersection operation, the compound rule may be rewritten as
IF <!m THEN J!,.
2. Multiple disjunctive antecedents
IFxis.d-1 ORxis.dz, ... ORxis-cln THENyisn.
This can be wrinen as
IF X is dn THENy is nm
where the fuzzy set -clm is defmed as

dm=.d-1Ud2Ud3UUdn

The membership function is given by

1-'d,(x) = mru<[!-'d, (x), I-'d, (x), 'I'd. (x)]

which is based on fuzzy union operation.
3. Conditional statements (with ELSE and UNLESS):
Statements of the kind
IF <!1 THEN (ill ELSE ilz)
can be decomposed into rwo simple canonical rule forms, connected by "OR":

IF <!1 THEN il1

OR
IF NOT <! 1 THEN ilz
IF<! 1 (THEN ,1) UNLESS<!z
can be decomposed as

IF <!1 THEN il1

OR
IF <!z THEN NOT ,1
IF<! 1 THEN (,1) ELSElF<!z THEN (/!z)
can be decomposed into the form

IF <!1 THEN 1!1

OR
IF NOT <! 1AND IF <!z THEN ilz
352 Fuzzy Rule Base and Approximate Reasoning

4. Nesr<d-IF-THEN rules:
The rule "IF !11 THEN ~F !12 THEN (il1ll" can be of the form
IF !11 AND !12 THEN !J1
Thus, based on all the above-menrioned methods compound rules can be decomposed imo series of
canonical simple rules.

112.6 Aggregation of Fuzzy Rules

The rule-based system involves more than one rule. Aggngation ofruks is the process of obtaining the overall
consequents fi:om rhe individual consequents provided by each rule. The following rwo methods are used for
aggregation of fuzzy rules:

l. Conjunctive system ofruks: For a sysrem of rules to be joindy satisfied, the rules are connected by "and"
connectives. Here, the aggregated ourpur,y, is determined by rhe fuzzy imersecrion of all individual rule
consequems,y;, where i = I ron, as

y =)I and )'2 and ... and Yn

0< y=y1 nnn 13 n ... ny,
This aggregated ourput can be defined by the membership function

tL1 (y)=m;n[tL1 ,(y), tLy,(y), ... ,tL1.(y)J fm yE Y

2. Disjunctive syJtem of mler. In this case, the satisfaction of at least one rule is required. The rules are
connected by "or" connectives. Here, the fuzzy union of all individual rule comriburions determines the
aggregated ourput, as

y =}I or yz or ... or J 11
or y= Jl Uyz Uy3U Uyn

Again it can be defined by the membership function

!1 1 (y) = mm<[tL1,(y), !Ly,(y), ... tL1.(y)J foryE Y

112.1 Fuzzy Reasoning (Approximate Reasoning)

Fuzzy reasoning is the collection of topics discussed in Sections 12.4-12.6. In fuzzy logic borh the antecedents
and consequents are allowed to be fu'l.Zy propositions. There exist four modes of fuzzy approximate reasoning,
which include:

I. categorical reasoning;
2. qualitative reasoning;
3. syllogistic reasoning;
4. dispositional reasoning.

.L
 12.7 Fuzzy Reasoning (Approximate Reasoning) 353

I 12. 7.1 Categorical Reasoning

In rhis cype of reasoning, the antecedents comain' no li,lzzy quantifiers and fuzzy probabilities. The anrecedenrs
are assumed ro be in canonical form. For underSrancfing the inference rules of categorical reasoning in fuzzy
logic, one should take nore of the following notarimls:
bM, fY, ... = fu"Z.Zy variables raking in the universes U, V, W;
4, Q, {; = fuzzy predicates.
I. The projection rule of inference is defined by

f:;, l)f, is fi
~;;sf!!.!]

where [ !1 t kJ de~otes the projection of fuzzy relation B. on !::,.

2. The conjunction rule of inference is given by

/:;is4./:.is !i::} k is4 nn

(J;, MJ ;, 4, L ;, I! => (J;, MJ ;, 4 n (!J x .!:1
([_, MJ ;, .:1, {y, !':J) ;, !J =} ([_, M. !':J) = (4 X \D n (k[ X I!)
3. The disjunction mle of inference is given by

k is4 OR: is /i::::} k is4 X fl

k is 4, .1:1 is !i ::} (/:;, /}f) is 4 x fl
4. The negative rule of inference is given by

NOT( ;,4) =>!; ;,J

5. The compoJitional ruieofinference is given by

k is4, (l:_, M is fl::::} /yf is4 fl

where 4 fi denotes the max-min composition of a fuzzy set A and a fuzzy relation R given by
ILLJft(v) = m ~ min [!Ld(u), !Lft(tt, v)J
1
6. The extension principle is defined as

~;;s4 => f(!J ;sf(!l)

where ''f" is a mapping from u to v so that Lis mapped into f(!J; and based on the exrens~on principle,
the merr.oership fimction off(.~) is defined as

ILfrdl (v) = sup lld(u), u E U, v E V

v:lj(u)

L.
 354 Fuzzy Rule Base and Approximate Reasoning

112.7.2 Qualitative f<easoning

In qualitative reasoning the inpm-ourput relationship of a system is expressed as a collection of fuzzy IF

THEN rules where the antecedents and consequents involve.fuzzy linguistic variables. Qua1iracive reasoning
is widely used in control system analysis. Let .d and !l be the fuzzy input variables and {;be the fuzzy output
variable. The relation among.(,!, fl. and (;may be expressed as
lf-4 is XI AND lJ isy1, THEN {;is ZJ
If 4 is X2 AND ll is y., THEN [is z2

If-dis x, AND !lis Ym THEN [;is z,

where x;,y; and z;, i = 1 ron, are fuzzy subsets of their respective universe of discourse. This is similar to the
canonical rule formation shown in Table 12~1.

I 12. 7.3 Syllogistic Reasoning

' In syllogistic reasoning, antecedents with fuzzy quantifiers are related to inference rules. A fu:u.y syllogism can
be expressed as follows:

x == k1 A's are B's

y=kzC'sareD's
z = k3 E's are F's
In the above A, B, C, D, E and Fare fuzzy predicates; ,(q and kz are the given fuzzy quamifters and k3 is
rhe fuzzy quantifier which has to be decided. All the fu1.1.y predicates provide a collecrion of fuzzy syllogisms.
These syllogisms creare a set of inference rules, which combines evidence through conjunction and disjunction.
Given below are some important fuuy syllogisms.
1. Prodrtce syllngism: C A A B, F = C A D
2. Chai11i11g sy/Wgism: C= B, F = D, E =A
3. Comequent conjunction sy/Wgism: F = B A D,A = C=E
4. Conscqutllt disjunction sy/lngism: F = B V D,A = C = E
5. Precondition conjunction syllogism: E =A A C, B = D = F
6. Preconditio11 diJjuuction sy!lngism: E = A V C. B = D = F

-
112.7.4 Dispositional Reasoning

In this kind of reasoning, the antecedents are dispositions thar may contain, implicitly or explicitly, the fuzzy
quantifier "usually." Usuality plays a major role in dispositional reasoning and it links together the dispositional
and syllogistic modes of reasoning. The important inference rules ofdispositional reasoning are the following:
1. Dispositional projection rule of inference:
usually ((L,MJ is R) => usully (Lis [R ~ L])

where [R t LJ is the projection of fuu.y relation Ron L.

 '
12.8 Fuzzy Inference Systems {FIS) 355

2. Dispositional chaining hypersyllogism: ktA's are Us, kzl!s are C's, usually (B C A)

usually(-> (Q,(-)0) Ns a<e C's)

The fuzzy quantifier "usually" is applied to the conplnment relation B C A

3. Dispositional consequence conjunction syllogism:

usually (Ns '" B's), usually (Ns '" C's) => 2 usually (-) t (Ns "' (B nd C)'s)
is a specific case of dispositional reasonin&:
4. Dispositional enrailmenr rule of inference:
usually (xis A), A C B => usually (xis B)
xis A, usually (A C B) :::::} usually (xis B)
usully (xis A), usually(A""C B) => usuallr(x is B)
is the dispositional entailment rule of inference. Here "usuallyl" is less specific than "usually."

112.8 Fuzzy Inference Systems (FIS)

Fuzzy rulehased systems, fuzzy models, and fuzzy expert systems are generally known as ~erence
systems. The key unit of a fuzzy logic system is FIS. The primary work of this system is decision making.
FIS uses "JF ... THEN" rules along with connec[Qrs "OR" or "AND" for making necessary decision rules.
The input to FlS rnay be fuzzy or crisp, but the ourput from FIS is always a fuzzy set. When FlS is used as
a c~oUer, it is 11ecessary to have cnsp output. Hence, rhere shoui'CI1ieaoefU:UlfiC3.tiOi1"unit fur convening
fuzzy variables into crisp variables along FIS. The entire FIS is discussed in detail in following subsections.

12.8.1 Construction and Working Principle of FIS

A FIS is conmucted of five functional blocks (Figure 121). They are: ' ..
r') \\\96 I)"
ln
~
1. A rule base rhat conrains numerous fuzzy IF-THEN rules. Lf'c,.-
2. A database th;u defmes rhe membership functions of fu:zzy sers used in fuzzy rules. "
.. _, . QC.J'
3. Decision making unit that performs operati.Q.D....Q!L!:be rules.
4. Fuzzification interface unit that converts the crisp quantities into fuzzy quantities.
5. Defuzzification imerfoce ~tnit that convens the fuzzy quantities into crisp quantities.
The working methodology ofFIS is as follows. Initially, in the fuzzif1cation unit, the crisp input is convened
into a fuzzy inpU[. Various fuzzification methods are emplOyed for this . .After this process, rule base is formed.
Database and rule base are collecrively called the knowledge bast. Finally, defuzzification process is carried our
to produce crisp output. Mainly, the fuzzy rules are formed in the rule base and suitable decisions are made
in the decision-making unit.

I 12.8.2 Methods of FIS

There are two important types of FIS. They are:

I. M=dani F!S (1975);
2. Sugeno F!S (1985).
 356 Fuzzy Rule Base and Approximate Reasoning

---------------------------------------------
'
'
'
'
'
r-------___,~-~--:.~:~~~nb:;:--1--r;
i bas~ i
_____,
'' base

' '---------------------1
:r---'---,
r---~-.1~
'
Fuzzificalion Defuzzificalion
~ intertace interlace
(~ unit unit 1 (Crisp)
>. : L - - r - - _ j ---,,- :'
tl '( :
1
,/\\ ).'JL ; 1
IJ ! ;_; ('I
_r I I
l
Decision-making
.
I

I ',
r( ~~ J .} 1 Fuzzy UM Fuzzy I
, (j (fJ, ,_ I I

.,. . ; ...,k.~ :------------------------------ -------------- _:

-'!' u;~v; C
Figure 12~1 Block diagram ofFIS.
\ \' ;J ',\"'{;.\;"'JVf2"
,\"
,, u '
The difference between the two methods lies in the cnnseqnenr of uzzr
rules. Fuzzy sets are used as rule
. .'( consequents in Mamdani FIS and liQear functions of input variables are useCI as rule cqruequenfs mSU~eno's
meJlo~amdanrs rllie fmds a greater acceptance in all universal approx1mators than Sugeno's model.

12.8.2, 1 Mamdani FIS

Ebsahim Mamdani proposed this system in the year 1975 to comrol a steam engine and boiler combination
by synthesizing a set of fuzzy rules obtained from people working on the system. In this case, the...Q.!!ijllJ.t
membership functions are expected to be fuzz sets. After aggregation process, each our ur variable s
a fuzzy set, hence e UZZJ 1cauon is important at the ourput smge. e o owmg steps have to be followed to
Compute r:he output from this FIS:

I Step 1: Determine a set of fuzzy rules. I

Step 2: Make the inputs fuzzy using input membership functions.
Step 3: Combine the fu~ified inputs according to the fuzzy rules for est~blishing a rule strength.
Step 4, Determine the f~-~~9~t~rtnej'Yle by combining the rule strength and the output membership
function. - _ ! ;{v~-
Step 5: Combine all the consequents to get an output distribution.
Step 6: Finally, a defuzzified output disrrib'G;:ion is obt;uned. ---
~rT''
The fuzzy rules are formed using "IF-THEN" statements and "AND/OR'' connectives. The consequence
of the rule can be obtained in two steps:

1. by computing the rule strength complerdy using the fuzzified inputs from the fuzzy combination;
:V 2. by clipping the output membership function at the rule strength.

The outputs of all the fuzzy rules are combined ro obtain one fuzzy output distribucion. From FIS, it is
desired to get only one crisp output. This crisp output may be obtained from defu1..zification process. The
common techniques of defuzzification used are center o[1_1111Js and "!ean ofrruv:imum.
I

L
 '
12.8 Fuzzy lnlerence Systems (FIS) 357

If Rule Then
strength

___ ,_~-l--lA
'---'--"----'--~ X 1 r:.- . . . . ,uo.. ' x X

""

r
"
"
- - - !1
""
~ ~-- --~----":l:Jiii'J __ ~ c --- .>c-

"
"
L--L~----L-~x
1 ,.. .... :.-.-.. v //A \ x X

Input
Il&.
I_k Figure 12-2 A two-input, two-rule Mamdani FIS wid1 a fuzzy inpur.
Oulpul
d;str;bulions x

Consider a LWo-input Mamdani FIS with rwo rules. The model fuzzifies the two inpurs by finding the
intersecrion of two crisp input values with rhe input membership function. The minimum operation is
used to compute the fuzzy input "and" for combining the two fuzzified inputs to obrain a rule strength.
The output membership function is clipped at the rule strength. Finally, the maximum operawr is used
to compure the fuzzy output "or" for combining rhe ourpur of the rwo rules. This process is illusrrated in
Figure 12-2.

12.8,2.2 Takagi-Sugeno Fuzzy Model (TS Method)

Sugeno fuzzy method was proposed by Takagi, Sugeno and Kang in the year 1985. The formar of the fuzzy
rule of a Sugeno fuzzy model is given by

IF xis A andy is BTHEN z: j(x,y)

where AB are fuzzy sers in the antecedents and z f (x,y) is a crisp function in rhe consequent. Gen~
=
eraJly, f (x,y) is a polynomial in the input variables x andy. Iff (x,y) is a f~-order polynomial, we ger
fi!~~:?.!fk.L.~_!.Igeno f1,1~~- .~odsl Iff is a consranr, we ger zero~order Sugeno f~~y moO.et A 7ro~order
Sugeno. fully model is functionally equivalent ~ a radial basis func~~- ~e~C?~l<- under cerrain minor
constraints. -~---- -

L
358 Fuzzy Rule Base and Approximate Reasoning

Input
membership function
Input 1
I
[
~~
X

I
Rule strength

lnpu\2
AND ~
_j
y
11 - I

f Input
J
. Output
membership lunc\Lon
.
membershipfunc\Lon

~
~
(Mput
L level

Z"'ax+by+c

Figure 123 Sugeno rule.

The main sreps of the fuzzy inference process namely,

l. fuzzifying rhe inputs;
2. applying the fuzzy operator

arc exactly the same. The main difference between Mamdani's and Sugeno's methods is rhar Sugeno outpUt
membership functions are either linear or constant. ~
The rule format of Sugeno form is given by

"If3 = xand 5 = yrhen output isz =ax+ by+ c."

For a Sugeno ~of zero order, rhe oumm level z is a constanr. The operation of a Sugcno rule is as shown
in Figure 12-3.
Sugeno's merhod can act as an i.Q!erpolaring supervisor for multiple linear comrollers, which are to be
applied, because of the linear dependence of each rule on Ute input variables of a system. A Sugeno model
is suited for smooth interpolation of linear gains that would be applied across the input space and for
modeling nonlinear systems by interpolating between multiple linear models. The Sugeno system uses adaptive
techniques for constructing fuzzy models. The adaptive techniques are used to customize the membership
functions.

.
12.8.2.3 Companson b etween Mamdani and Sugeno Method ~
' "'-. \
The main difference between Mamdani and Sugeno methods lies in the outpur~ership functions. The
Sugeno output membership functions are either linear or constant. The difference also lies tn the consequents
 '
359

of their fuz~ :?fd as a result their regation and defi.Jzz" . A large

number of hizzy rUles must be employed n ugeno method for approximating periodic or highly oscillatory
functions. The configuration of Sugeno fuzzy systeffis qm be reduced and it becomes smaller than that of
Mamdani fuzzy systems if nontriangular or nontrapezoid_al fuzzy input sets are used. Sugeno controllers have
more adjustable parameters in the rule consequent and the number of parameters grows exponentially with the
increase of the number ofinputvariables. There exist several mathematical results for Su eno fUzzy controllers
than for Mamdani controllers. Formation of Mam ani 1S more easier than Sugeno FIS.
1'lleffi';[~~d~~~~g;; Of Mamdani method are:
1. it has widespread acceptance;
2. it is well-suitable for human input;
3. it is intuitive.
On the other hand, the advantages of Sugeno method include:
1. It is computationally efficient.

- -
2. It is compact and works well with linear technique, optimization technique and adaptive technique.
3. It is best suited for ~arhematical analysis.
4. It has a guaranteed cominuiry of the omput surface
The most important modeling tool based on fuzzy set theory is FIS, and is widely used in various
applications.

112.9 Overview of Fuzzy Expert System

An expert fuzzy syste~ is a concept that is much like an expen for a parricular problem in humans. There are
t'NO major functions of expert systems:

I. h is expecred ro deal with uncertain and incomplere information.

2. lr possess user-inreracrion function, which contains an explanation of systems intentions and desires as
well as decisions during and after the application has been solved.
The basic block diagram of an expert system is shown in Figure 12-4. From Figure 12-4, it can be noticed
that an expert system conrains three major blocks:
I. Knowledg~ bttse rhar contains the knowledge specific ro the domain of application.

Knowledge
base

User J User
inlerface

Inference
engine

Figure 124 Block diagram of an expert system.

360 Fuzzy Rule Base and Approximate Reasoning
i
2. lnfuenu tngine rhar uses rhe knowledge in clle knowledge base for performing suitable reasoning for user's
queries.
3. Uur.inteiface rhar provides a smooth communication berween the user and the system.
I
This also helps the user for undemanding emire problemsolving method carried our by the inference engine.
An example of an expert system is MYCIN, which introduces the concept of certainty factors for dealing
uncertainty. MYCIN rules have astrengrh, called as cenainty factor. This factor lies in the unit imerval {0, 1].
When a rule is fired, its presrare condition is evaluated and-a firing strength, a value between -1 and+ 1, is
associated with the prestate condition. For the firing strength higher than the previously mentioned threshold
interval, the consequent of the rule is determined and the conclusion is made with a certainty. The obrained
conclusion and its cenainty are the evidence provided by this fired rule for ilie hypoilieses given by user. The
hypotheses evidence from -different rules is combined into belief measures and disbelief measures which are
values lying in the interval [0, 1] and [-1, 0}, respectively. If belief measure lies above a ilireshOld value, a
hypoiliesis is believed, and if disbelief measure is below a threshold value, a hypothesis is disbelieved. The
use of fuzzy logic in traditional expert systems leads ro fUzz.y expert systems. Fuzzy expert systems are those
systems iliar incorporate fuzzy sets and/or fuzzy logic for their reasoning process and knowledge representation
scheme. The fUzzy sets and possibility rheory applications ro rule-based expert system are mainly developed
along the following line.
I. Generalization of certainty facwr in MYCIN: enlarging the operations ro be used for combining the
uncerraimy coefficients or by allowing the use of linguistic certainty values along with conventional
i
numerical certainty values.
2. Method of handling of vague predicates in the expression of expert rules or available information.

Fuzzy expert systems effectively handle both uncertainry and vagueness (imprecision). Examples of fuzzy
expert system include Z-II, MILORD, etc. Researchers are in the process of developing a wide variety of fuzzy
expert systems. One such system is SPERIL, which is a special fuzz.y expert system for analyzing earthquake
damages.

112.10 Summary

In fuzzy logic, the linguistic variable "uuth" plays an imporram role. The various forms of fuzzy propositions
and fuzzy IF-THEN rules rhar are a useful paradigm for rhe implementation of human knowledge are
discussed. This provides a means for sharing, communicating and transferring the human knowledge w
systems and processes. Fuzzy rules are presented in canonical form. The decomposition of fuzzy compound
rules and aggregation of fuzzy rules were also discussed, as also four meiliods of approximate reasoning thereby
creating fuzzy inference rub. The Mamdani and Sugeno FIS give a base for building fuzzy rule base system.
The comparisons between the rwo methods are also included. Finally, we provide an overview offuzzy expert
system, which deals wirh cerraimy factor.

112.11 Review Questions

l. Define linguisdc variable. 3. What is meant by linguistic hedges?

2. State the importance of trufh values and truth 4. What are the characteristics of a linguistic
tables. variable?

I
l
 12.12 Exercise Problems

I
361

5. List the basic logic operations performed over 15. State the inference rules of dispositional
the propositions. reasoning.
6. Write short note on fuzzy propositions. 16. What is fuzzy inference system (FIS)?
7. How is a canonical rule formed based on che ~ 7. With suitable block diagram, explain ilie work-
human knowledge? ing principle of an FIS.
8. Mention the general forms that exist for a 18. List the methods of FIS.
linguistic variable..
19. Describe in detail of formation ofinference rules
9. In what ways is the decomposition of compound in a Mamdani FIS.
linguistic rules established?
20. Discuss in brief on Takagi-Sugeno FIS.
10. Discuss the methods of aggregation of fuzzy
21. Scare ilie advantages and disadvantages of
rules.
Mamdani FIS.
11. Why is approximate reasoning important m
22. List the application ofSugeno FIS.
fuzzy logic?
23. Differentiate berween Mamdani FIS and Sugeno
12. What are four modes of approximate reasoning?
FIS.
13. Explain in detail: categorical reasoning and qual-
24. Define expert system. How is a fuzzy expert
itative reasoning.
system formed? State its importance.
14. How is a fuzzy syllogism expressed and list the
25. Menrion a few fuzzy expert systems used in
important fuzzy syllogism used generally? current scenario.

I 12.12 Exercise Problems

I. The membership functions for the linguistic 4. Wirh a suirable case srudy, demonstrate the
variables "rall" and "shan" are given below. canonical rule formation, aggregation of the

.. _, 1..
[~
I0.2 0.3 I
0.7
= -+-+-+-+- 0.9 1.0
funy rules and decomposition of compound
rules formed.

I-+-+-+-+- I
5 7 9 II 12 5. Give an example for the following propositional
"shorr=
" 0.3 0 I 0.5 0 principles:
0 30 60 90 120
(a) Fuzzy rrurh qualification
Develop membership functions for the following (b) Fuzzy possibility qualification
linguistic phrases: (c) Fuzzy probability qualification
(a) Very rail; (d) Fuzzy usuality qualification
(b) Faidy r,]J; 6. Provide examples for fuzzy propositions includ-
(c) Nor very short. ing fuzzy predicates and fuzl.y quantifiers.
7. Give an example for each of the following
2. Develop an FIS editor for a liquid level con- approximate reasoning rules:
troller model (Mamdani and Sugeno fuzzy
inference models). (a) Compositional rule of inference
3. Develop an FIS (Mamdani) model for control- (b) Conjunction ruJe of inference
ling temperature in an air conditioner. (c) Disjunction rule of inference
 Fuzzy Rule Base and Approximate Reasoning
362

8. Change the following symbolic rule to canonical 10. With sui~able application case srudy, ana
form: lyze MILORD fuzzy expert sysrem. Com-
If L1 ~ R1 (THEN M1 AND M, (IF L, is R, pare its performance with conventional fuzzy
(THEN M, (IF L, is R3 THEN M,)))) system.
9. Develop a Sugeno FIS for a satellite tracking
control system.
r
I
~

Fuzzy Decision Making 13

Learning Objectives
Discusses on variable paradigms available for How evaluation of alternatives are carried out
fuzzy decision making. using che attributes of the object.
The importance of muhiobjective and multi An overview on fuzzy Bayesian decision
person decision making. making.

113.1 Introduction
Decision making is a very important social, economical and scientific endeavor. DeCISJt. making activities
are the steps taken to choose a suitable alternative from thOse thar are needed for realizing a cenain goal. The
decision~ making process involves rhree sreps:

I. determining the ser of alternatives;

2. evaluating alternatives;
3. comparison between alrernarives.
In any decision process, the information about the ourcome is considered and a suitable path has to be
chosen from rwo or more alternatives for subsequent action; when good decisions are made, good output
is expected. If a decision is made under certainry, then the outcome for each process can be determined
precisely; one should note thar whenever decision is made, it is under risk condition. The prime domain
for fuzzy decision making is the existing uncertainty. There are several situations under the decision-making
process. There may be situations when even though decisions made are good, the outpur may be adverse
or vice-versa. When good decisions are made continuously for a longer period, advantageous situations may
prevail.
\'\'hen there are several objectives to be realized in making a decision, the decision making is called
multiobjective decision making. The knowledge ofexperts becomes very essen rial when decision making is very
tedious. The information may be available for the following: the possible outcomes, change in conditions wirh
respect to time about value of new information, when the priority for each acrion is typically ambiguous, vague
and otherwise fuzzy. Obtaining an evaluation structure for selecting alternatives and establishing selection
standards are very imponanr stages. The evaluation ofalternatives may be carried out based on several anributes
of the object; such a decision making is called multiattribute decision making. In this chapter we would discuss
the various paradigms available for decision making.
l
!
I
I
j
 ~
364 Fuzzy Decision Making
I
>

- I
113.2 Individual Decision Making
A decision-making model in chis siruation is characterized by the folloWing:

1. set of possible actions;

2. set of goals G;(i E Xn);
3. set of constraints Cj(j E Xm).
The goals and constraints are expressed in terms of fuzzy selS. These fuzzy sets in individual decision
making are nor defined direcdy on the set of actions, bur by means of other sets char characterize relevant
states of narure. Consider a set A. Then the goal and constraint for this set are given by I
G;(a) = Composirion[G;(a)] = c:(G;(a)) with Gl
Cj(a) = Composition[Cj(a)] = cj(Cj(a)) with c)
for a E A. The fuzzy decision in this case is given by

FD = min [ inf G;(a), inf Cj(n)]

ieX~ jeX,

-
113.3 Multiperson Decision Making
Decision making in this case includes several persons. The experr knowledge from various persons is utilized
to make decisions. The difference berween rhe individual decision making and mulriperson decision making

'

is: The goals of individual decision makers differ, i.e., each places a different ordering arrangement. On the
other hand, in muhiperson decision making, the decision makers have access ro differem information upon
which ro base their decision.
Here, each member of a group of "n" individual decision makers has a preference ordering POk, k E Xn,
which totally or partiaJly orders a set X. A social choice (sc) function has to be found, given the individual
preference ordering. The fuzzy relation for a social choice preference function is given by

SC:XxX--+ [0, I]

which has a membership of SC(X;,Xj), which indicates the preference of alternative X; over Xj. Let
I
Number of persons preferring X; ro Xj = N(X;,Xj)
Total number of decision makers= n i
Then,

SC(x;, Xj) = N(x;, xj)

I i

The multi person decision making is also given by

n
J:i
'I
SC(x;.x;) = g if x; >~txj for some k
mherwise
ll
'

'
--~~
 13.4 Mul!iobjective Decision Making
365

113.4 Multiobjective Decision Making

In making a decision when there are several objectives to be realized, chen the decision making is called
multiobjective decision making. Many decision processes may be based on single objectives such as cosr
minimization, time consumption, profit maximization and so on. However, if all the above-mentioned objec-
tives are to be considered for a decision-making process, then it becomes multiobjecrive decision making. The
main issues in mulciobjective decision making are:

1. to acquire proper information related to the satisfaction of che objectives by various alternatives;
2. to weigh the relative imponance of each objective.
Mulriobjective decision making involves selection of one alternative a; from universe of alternatives A given a
collection of objectives {o} that are important for a decision maker. It is necessary to evaluate how best each
alternative satisfies each objective. The main aim here is to combine the weighted objectives into an overall
decision function in some way. The decision function represents a mapping of alternatives in A to an ordinal
sec of ranks. In order to make suitable decisions, the process needs to weigh che relative importance of each
objective.
Let us define a universe of n alternatives as

A= {a,,a2, ... , a;, ... , an}

and a set of"m" objectives as

0= {o,,olo,-, ... ,om}

where o; indicates the ith objective. The degree of membership of alternative a in o;, denoted j.Lo,-(a), is the
degree to which alternative a sa(isfies the criteria mentioned for this objective. A decision function is formed,
which simultaneously satisfies all rhe decision objectives. As. a result, the decision function, OF is given by
the intersection of all the set of objectives, i.e.,
DF = o1 n o2 n n o1 n nom
The grade of membership chat OF has for each alternative a is defined by
mDF(a) = min[J.LOJ, (a), f.L02(a),. . , j.Lo;(a), ... , J.L0 711 (a)]
The optimal decision, a*, will then be the alternative that satisfies the equation
moF(a*) = maxnEA[J.LoF {a)]
.Ler {P) be rhe set of preferences -linear and ordinal. The element of rhe preference set will possess linguistic
values or will have values in the interval [0, 1], or in the intervals [-1, 1],[1, 10], etc. The preferences are
attached to each of rhe objectives in order ro notifY the decision maker about the influence that each objeaive
should possess on the chosen alternative. The preference set P contains the parameters hi= 1 tom, i.e.,
{P} = {bJ. b,, ... , b;, ... , bm}
Thus for each object, we have a mea.~ure as to say how important it is to the decision maker for a given
decision. The decision function is then defined by decision measure (DM), which involves objectives and
preferences. The intersection of m-ruples of OM gives the decision function:

DM = DM(o;, b,-) -7 DM(objeccives, preferences)

DF = DM(o,, bJ) A DM (o,, b1 ) A 1\ DM (o;, b;) A A OM {om, bm)
366 Fuzzy Decision Making

The DM for a p~cular alternative, a;, is given by

DM(o;(a), b;) = b;-> o;(a) = b; V o;(a)

where bi = 1 - b; and b; --+ o; indicateS a distinct relationship between a preference and its corresponding
objective. Neverthless, several objectives can have the same preferences weighting in a cardinal sense; however,
chey will be distinct in an ordinal sense, even though the equality siruation b; = bj for i # j can exist for
certain objectives. A joint imersection of "m" decision measures will give an appropriate decision model:
m
DF = fi (b;u o;)
i=l

The optimal solution, a*, is the alternative that maximizes dte decision function. When we define

C;==-"b;Uo;
IJ.,,(a) = max [!1-bj(a),~J.,.(a)]

the optimal solution in membership form is given by

11-oF(a') = max[min(IJ.o (a),1J. 0 (a), ... ,IJ.,.(a), ... ,IJ.,.(a)]

,eA

When iili objective becomes very imporram in ilie final decision, b; increases, sob; tends to decrease. A:s a result
C;(a) decreases, thereby increasing the likelihood that C;(tt)- o;(a), where o;(a) at present will be the value of
the decision function, is DF, denoting alternative a. When this process is repeated for several alternatives a,
the largest value o;(a) for ocher alternatives will automatically result in the choice of the optimum solution,
? . The multiobjective decisionrnaking process works in this manner.

113.5 Multiattribute Decision Making

When there are several objectives to be realized in making a decision, then the decisionmaking process is
known as multiobjecrive decision making. On the other hand, the evaluation of alternatives can be carried
our based on several attributes of the object, in which case the decisionmaking process is called multiattribute
decision making. The attributes may be classified into numerical data, linguistic data and qualitative data. In
case of muhiamibute measurement and evaluation of alternatives, which have the addition of probabilistic
noise, probabilistic statistical methods are used m identify the structures. The problem of decisionrnaking
suucrure in multiattributes deals with determination of an evaluation structure for the mulrianribure decision
making from the available multiattribute data X;(i = l to n) shown in Table 131 and alternative evaluations
Y. It can also be said chat the multiattribute evaluation is carried out on the basis of the linear equation
Y=A1X1 +AzXz + +A;X; + +A,Xr
and this is the determinacion ofclte weight of each amibut_e. In Table 131, Xij is the value for attribute i of
alternative j. The term Jj is the evaluation of alternative j. For j = 1 to nand i = l to r, each value is obtained
here as a numerical value or a litiguistic.expression based on their respective problems.
It is necessary to determine the coefficient A; for linear multiattribute evaluation which best estimates the
evaluation of the alternative for the given object. A few vector expressions are given below:

Y= (ri.Y2 Yj ,y.]
r
I
13.5 Multiattribute Decision Making

Table 131 Multiattribute data evaluation

Alternative number Alternative evaluation Evaluation of alcernative attributes
367

J y XJoooXjoooXr

1 YI X"llXiJ XrJ
2 )'2 XJ2 x;z .. . Xr2

j Yj XJjXijXrj

n y. XlnXinXrm

XJJ X;J X:J]

X- .
[
Xtu Xin Xm

A= [aJ,ttz, .. ,a;, ... ,ttr]

Triangular fuzzy numbers are used to explain possibilistic regression analysis. The niangular fuzzy number
A is given by

_ la-xl
!-' (x)= 1 [ ' a-fsxsa+f
0 {
0 ; orherwise

A is a fuzzy number wirh center a and width fIn this case ir can be written as A = (a, f). The possibilistic
linear multiattribute evaluation equation is expressed by

Y= A tXt +AzXz + +A;X; + +A 11 Xn

Using extension principle, its membership function can be calculated as

1- ~-xTal
1-'r(y) = f'lxl ; xI 0
1;
0; x=O,y=O
1 x=O,y!O
Here, x = (xl ,xz, ... ,xn), a= (a 1, a2 , .. , an) and/= (/t ,fi., ... ,fn), and xT gives a transposition of vector
x. Also note that here, y and A; are fuzzy numbers. Additionally, for y such that cTixl< lY- xT a!, JL (y) = 0.
For determining this kind of possibilistic evaluation function, a measure for minimizing the possibility width

o=fo+fi ++fi++t.

,,.

'
368 Fuzzy Decision Making

is used. To determine the possibilistic evaluation, the following linear equation has robe solved:
1
mino= min(JO + +fi + +fi,)
4L( AiC

Here

(I - k) L.filxijl +La; Xij 2: y;

(1- k) Lfilxijl- L>zrx!i ~ -y;, i= l mn

where k = [0, l] indicates the congruence of the possibilistic regression model. Thus, for evaluation of
multiattribute decision making, possibiliscic regression analysis is effective.

113.6 Fuzzy Bayesian Decision Making

In classical Bayesian decision-making method, the furore stares of the nature are characterized as probability
events. Conventionally, the probabilities sum to-unity. The problem with the fuzzy Bayesian scheme is rhat
the events are ambiguous.
Consider the formation of probabilistic decision analysis. let the set of possible states of nature be given
by 8 = {sJ,sz, ... ,sn} Then the vector representation of probabilities of rhese stares is

"
P= {P(s,),P(s,), .. ,P(s")J whm :[P(s;) =I
i=l

These probabilities are called "prior probabilities." The decision maker can choose from "m" alternatives,

A= {aJ,ll2, ... ,a;, ... , am)

For a given alternative llj, a utility value J.lji is assigned, if the future state of nature becomes states;. The
decision maker determines these utility values. These values express rhe value or cost for each alternative stare
pair, i.e., for each llj- s; combination. The expected utility with the jth alternative is given by

"
EX(t~) =L t~;P(s;)
i=l

The common decision criterion is the maximum expected utility among all the alternatives, i.e.,
I
EX(u') = mex EX(uj)
1
I
This leads to the selection of alternatives ilk if u = EX (uA-), Let the informacion regarding the true states of
nature 8 be from n experiments and let it be given by a data vector X= {x1.X2 ... , x11 J. This information is I
used in Bayesian approach for updating the prior probabUities P(s;). Based on the new)nformacion, conditional
probabilities are formed, where the probability of each piece of data is determined according ro where the
true stare of natures; is known; these probabilities are the presumptions of the future.
The conditional probabilities are also known as likelihood values, given by P(x11 ls;). This conditional
i
~
'
I
probabilities are used as weights over th~ previous information, i.e., prior probabilities P(s;), ro determine I

!
.L
1 13.6. Fuzzy Bayesian Decision Making 369

updated probabilities called posterior probabilities, P(x;,x11 ). Bayes rule is used tO determine the posterior
probabilities:

P(x"ls;) P(s;)
P(s;lxn) = P(x )
11

P(x71 ) is marginal probability of data (x71 ) and is found using the total probability theorem,
m
P(x") = L P(x"ls;) P(s;)
i=l

For a given data Xm the expected utility for the alternative is found from the posterior probabilities:
m
EX (ujiXn) = L upP(s;lxn)
i=l

and the maximum expected utility for a given data x 71 is given by

EX (u*lx11 ) = m~ EX (ujlx,)
1

For determining the unconditional maximum expected utility, it is necessary to weigh each of the
11conditional compacted withies of the above equation by the respective marginal probabilities for each
Xm i.e., P(xn):

E.,(U;) = L"' EX (U'Ix,.)P(x,.)

n=1

At this stage, a norian called value of information, u(x), is introduced. There exist certain uncertainty in
the new information X= {XJ,Xz, ... ,x11 j called as imperfect information. This value of information V{X)
is found by the difference between the maximum expected utili[) without any new information and the
maximum expected utility with the new information, i.e.,

V{X) = EX (U:)- EX (U')

There exist perfect information as well. For information to be perfect, the conditional probabilities are free
of dissonance. The perfect information is represented by the posterior probabili;:ies of 0 or 1, i.e.,

P(S;Ix,.) = I~
The perfect information is denoted by Xp For this perfect information, the maximum expected utility
becomC;S

EX (l.f) = L EX (u;,lx,)P(x 11 )

=I
and the value of perfect information becomes

V(xp) = EX (u; ) - EX (u*)

'

L
370 Fuzzy Decision Making

Let the new information X= {x!,X2., ... ,Xj, .. ,x71 } be a universe of discourse in the units appropriate
for the new inforrnatiori. Then the corresponding events (fuzzy events .g on this information are defined. The
membership for the fuzzy evem may be given by /LE (x,), x = 1 to n. Let us define the idea of a "probabilicy
of a fuzzy event," i.e., the probability of .ff., as

"
P(!!) = LJL<:(x,)P(x,)
~]

If the fuzzy event, for the above equation is crisp, i.e . .g = E, then ilie probability reduces to

P(E) = L P(x,)
XrEE

f.L =
II, x,EM
0, otherwise

This equation describes the probability of a crisp evem as the sum of the marginal probabilities of those
data points, Xn which are defined tO be in the event,. The posterior probability ofSj, given fuzzy informacion
.g, is
L:" P(x,ls;)JL fi(x,)P(s;)
P(,s;)P(s;)
P(S;Ifi) = ~' P(E)
P(fi)

where

"
P(,@S;) = L P(x,IS;)JL<:(x,)
~]

Defining the collection of all the fuzz.y events describing fuzzy information as an orthogonal fuzzy information
system, we have 1/1 = {f'I>Ez, ... ,Qrn}.
The orthogonal means the sum of the membership values for each fuzzy event !J;, for every data point in
rhe universe of information, Xn equals uniry, i.e.,
m
L JL<; (x,) = l foe oll x, EX
""'
When the fuzzy events on rh.e new information universe are orthogonal, we extend the Bayesian approach
for considering fuzzy information. The fuzzy equivalents for the posterior probability, maximum expected
utility and the rriarginal probability are given by, for a fuzzy event .g,,

E(u;l fi,) =L u;;P(S;I fi,)

i=l
E= (u~l/h) = m;tXE(ujl'f.')
1

'
E(u;) = LE(ulfi,)P(fi,)

""'
r
J
13.9 Exercise Problems

The value of fuzzy information can be determined as

v(<P) = E(u;) - E(u)

371

113.7 Summary
In iliis chapter, various fuzzy decision-making methods are discussed. One of ilie decision-making method-
fuzzy Bayesian decision making - is given td accept both fuzzy and random uncertainty. Based on the
several objectives to be realized in making a decision, mulri,,bjective decision making was included. The
evaluation of alternatives based on several attributes of the object can be carried our; this process called
multiattribute decision making is discussed. Also based on the decision of persons involved, individual
decision making and mulriperson decision making are also dealt with. The main processes involved in deci-
sion making are the determination of set of alternatives, evaluating alternatives and comparison becween
alternatives. In many decision-making situations, the goals, constraints and consequences of the defined
alternatives are known imprecisely, which is due to ambiguity and vagueness. Meiliods for addressing this
form of imprecision are important for dealing with many of the uncertainties as we deal within human
systems.

I 13.8 Review Questions

1. What are the steps involved in decision-making 6. State the decision function for a muhiobjecrive
process? decision making.
2. Write shore nore on individual decision making. 7. Explain multiamibme decision making in detail.
3. Differentiate beWeen individual decision maker 8. Compare and contrast muhiobjecrive decision
and multiperson decision maker. making and multiattribute decision making.
4. Discuss multiperson decision making in derail. 9. Discuss fuzzy Bayesian decision making in detail.
5. What is meant by multiobjective decision 10. What are rhe advantages of fuzzy Bayesian
making? decision-making process?

I 13.9 Exercise Problems

1. Evaluate three different approaches for controlling fl.). = over all efficiency
conditions of a metal smelting cell. The control .eJ = error reduction
approaches are:

a1 = FT, fast tuning The control approaches are rated as

az = MT, medium tuning
a3 = ST, slow runing
,, = I0.35 + ~ + 0.2
There are several objectives to consider which are - Ff MT ST 1
'
given below:
-10.1Ff+MT+ST
0.4 0.61
II .e1 = less power consumption !!J.-

II
37.2

-I0.5 0.65 0.31

1!3- Ff+MT+ST
Fuzzy Decision Making

2. With a suitable decisionmaking algoriclun, help

a water aurhoricy to. decide whether or not to
j
build dive for preventing flooding in case of
The preferences are given by br = 0.6, bz = 0.5 excess rainfall. Assume necessary parameters and
and b3 = 0.4. What is the best choice of control? membership functions.

!
I
L
 Fuzzy Logic Control Systems 14
Learning Objectives
Need for a fuzzy logic controller. A brief note on fuzzy logic comroller model.
How rhe control system design has to be Application of fuzzy logic conuoller ro air
carried our? craft landing conuol problem.
The basic architecture and operation. involved
in a fuzzy logic controller sysrem.

114.1 Introduction
Fuzzy logic control (FLC) is rhe most active research area in che application of fuzzy sec theory, fuzzy reasoning
and fuzzy logic. The application ofFLC extends from industrial process control to biomedical inmumemation
and securities. Compared to conventional control techniques, FLC has been best utilized in complex ill-defined
problems, which can be controlled by efficient human operator without knowledge of their underlying
dynamics.
A control system is an arrangement of physical components designed to alter another physical system so
that this system exhibits certain desired characteristics. There exist WO types of control systems: open-loop
and dosed-loop control systems. In open-loop control systems, the input control action is independent of the
physical system output. On the other hand, in closed-loop control system, the input control action depends
on the physical system output. Closed-loop control systems are also known as fiedbackcontrolsysums. The first
step toward controlling any physical variable ism measure it. A sensor measures the con;rolled signal. A plant
is the physical system under control. In a dosed-loop control system, forcing signals of the system -called
inpms- are determined by the output responses of the system. The basic control problem is given as follows:
The output of the physical system under control is adjusted by the help of error signal. The difference beween
the actual response (cal_culated) of the plant and the desired response gives the error signal. For obtaining
satisfactory responses and characteristics for the closed-loop control system, an additional system, called as
compensator or control!rr, can be added ro the loop. The basic block diagram of dosed-loop control system is
shown in Figure 14-l,
The basic concept behind FLC is to urilize the expen knowledge and experience of a human operator for
designing a controller for controlling an application process whose input-output relationship is given by a
collection of fuzzy c;:onuol rules using linguiscic variables instead of a complicated dynamic model. The fuzzy
control rules are basically IF-THEN rules. The linguistic variables, fuzzy control rules and fuzzy appropriate
reasoning are best utilized for designing the controller.

L
 374 Fuzzy Logic Control Systems

- ..
Manipulated
Error Controller variable OutpL 11

~
or Plant (or)
S' compensator ,
~ m

{ Sensor 1...
Figure 14a1 Block diagram of a dosed-loop comrol system.

In this chapter we shall introduce the basic strucrure and design methodologies of an FLC model. FLC
is strongly based on the concepts of fuzzy sets, fuzzy relations, fuzzy membership functions, defuu.ification,
fuzzy rule-based systems and approximate reasoning discussed in the previous chapters.

'!' -

~ 114.2 Control System Design

Designing a controller for a complex physical system involves the following steps:

1. Decomposing the large-scale system into a collection of various subsystems.

2. Varying the plam dynamics slowly and linearizing the nonlinear plane dynamics about a set of operating
points.
3. Organi"Z.i.ng a set ofstate variables, control variables or output fearures for the system under consideration.
' - 4. Designing simple P, PD, PID controllers for the subsystems. Optimal controllers can also be designed.

Apan from the first four sreps, there may be uncertainties occurring due to external environmental condi~
" tions. The design of the controller should be made as dose as possible to the optimal controller design based
on the expert knowledge of the conuol engineer. This may be done by various numerical observations of the
input-output relationship in the form of linguistic, intuitive and Q[her kinds of related information related
to the dynamics of plant and external environment.
Finally, a supervisory control system, either manual operator or amomatic, forms an extra feedback control
loop to tune and adjust the parameters of the controller, for compensating the variational effects caused by
nonlinear and unmodeled dynamics.
In comparison with a convemional control system design, an FLC system design should have the following
assumpcions made, in case it is selected. The plant under1consideration should be observable and controllable.
A wide range of knowledge comprising a set of expert linguistic rules, basic engineering common sense, a set
of data for input/output or a controller analytic model, which can be fuz.z.ified and from which the fuzzy rule
base can be formed, should exist.
Also, for the problem under consideration, a solution should exist and it should be such that the control
engineer is working for a "good" solution and not especially looking for an optimum solution. The controller
in this case should be designed to the best of our ability and within an acceptable range of precision. It
should be noted iliat the problems of stability and optimality are ongoing problems in fuzzy controller
design.
 r
~
I'
14.3 Architecture and Operation of FLC System 375

In designing a fuzzy logic controller, the process of forming fuzzy rules plays a vital role. There are four
structures of fuzzy production rule system (Weiss and D6nnel, 1979) which are as follows:

1. A set ofruks iliat represents the policies and heUristic strategies of the expert decision maker.
2. A set ofinput data that are assessed immec/.iatelf prior to the actual decision.
3. A method for evaluating any proposed action in terms of its conformity to the expressed rules when
there is available data.
4. A method for generating promising 3;_~tions and determining when to stop searching for better ones.

All the necessary parameters used in fuzzy logic comroller are defined by membership functions. The rules
are evaluated lliing techniques such as approximate reasoning or interpolative reasoning. These four structures
of fuzzy rules help in obtaining the control surface that relates the control action to the measured state or
output variable. The control surface can then be sampled down to a finite number of points and based on
this information, a look~up table may be Constructed. The look~up table comprises the informacion about the
conuol surface which can be downloaded into a readonly memory chip. This chip would constitute a fixed
controller for the plant.

114.3 Architecture and Operation of FLC System

The basic architecture of a fuzzy logic controller is shown in Figure 142. The principal components of an
FLC system are: a fuu.ifier, a fuzzy rule base, a fuzzy knowledge base, an inference engine and a defuzz.ifier.
It also includes parameters for normalization. When the output from the defuzzifier is not a comrol action
for a plant, then the system is a fuzzy logic decision system. The fuzzifier present converts the crisp quantities
into fuzzy quantities. The fuzzy rule base stores the knowledge about the operation of the process of domain
expertise. The fuzzy knowledge base stores the knowledge about all the input-output fuzzy relationships. It
includes the membership functions defining the input variables tO the fuzzy rule base and the outpmvariables
to the plant under control. The inference engine is the kernel of an FLC system, and it possess the capability
ro simulate human decisions by performing approximate reasoning to achieve a desired control strategy. The
defuzzifier converts the fuzzy quantities into crisp quantities from an inferred fuzzy control action by rhe
inference engine.

'

Inputs Normalization
x--'"' input scaling
factors
e

l
I x,
r
t
I Normalization
output scaling
!actors
Sensors

r
Figure 142 Basic architecture of an FLC system.

1
376

The various steps involved in designing a fuzzy logic controller are as follows:
Fuzzy Logic Control Systems
1
IStep 1: Locate the input, output and slate variables of the plane under consideration. I
Step 2: Split the complete universe of discourse spanned by each variable into a number of fuzzy subsets,
assigning each with a linguistic label. The subsets include all the elements in the universe.

Step 3: Obtain the membership function for each fuzzy subset.

Step 4: Assign the fuzzy relationships between the inputs or states of fuzzy subsets on one side and rhe
ompurs of fuzzy subsets on other side, thereby forming the rule base.
Step 5: Choose appropriate scaling factors for rhe input and output variables for normalizing the variables
I
between [0, 1] and [-1, I] imerval.
Step 6: Carry out the fuzzification process.
Step 7: Identify r:he output contributed from each rule using fuzzy approximate reasoning.
Step 8: Combine the fuzzy ourputs obtained from each rule.
Step 9: Finally, apply defuzzificarion to form a crisp output.

The above steps are performed and executed for a simple FLC system. The following design elements are
adopted for designing a general FLC system:

l. Fuzzificarion srr:uegies and rhe interpretation of a fuzzifier.

2. Fuzzy knowledge base:
normalization of the parameters involved;
partitioning of input and output spaces;
selection of membership functions of a primary fuzzy set.
3. Fuzzy rule base:
selection of input and output variables;
source from which fuzzy control rules are w be derived;
types of fuzzy control rules;
completeness of fuzzy control rules.
4. Decision making logic:
proper definition of fuzzy implication; '
interpretation of connective "and";
interpretation of connective "or";
inference engine.
i
!
5. Defuzzification mategies and the interpretation of a defuzzifier. i
When all the above five design parameters are fixed, the FLC system is simple. Based on all this, the features
of a simple FLC system are as follows: I
fixed and uniform input and output scaling factors for normalization;
flXed and nonimeracrive rules;
flXed membership functions;
I
only limited number of rules, which increases exponentially with the number of input variables;

L
1 14.5 Application of FLC Systems

fixed expertise knowledge;

no hierarchical rule strucrure and low-level comrol.
377

I
- 14.4 FLC Syst!llll Models
There are rv.r.o- different forms of FLC system mod~ls:
.1. fuzzy rule-based suucrures;
2. fuzzy relational equations.
Fuzzy rule-based models have already been discussed in a previous chapter. The fuzzy relational equation
describing a commonly used FLC model can be of the following forms:
The basic fuzzy model for a first-order discrete system with input a, which is described in state-space
representation, is of the form

XJ..+I = Xk o Uk o!! fork= 1,2, .. . ,n

where o is the composition and !! is the fuzzy system transfer relation. Consider a discrete prh order system
with single in puc u represented in statespace form. The basic fuzzy model of such a system is given by (for
k = 1 ton)

Xk+p = Xk 0 XJ..+] 0 . 0 Xk+p-1 0 Uk+p-1 0 B

Yk+p = XJ..+p

where Bis rhe fuzzy system transfer relation and n-+p is the single output of the system considered.
A second-order system with complete state feedback is given by the fuzzy system equation as (fork= I to 2)

UJ.:=XJ.:OXk-lof3
Yk = Xk

where Jk is the output of the system. Consider a discrete pch order single-input-single-output system wirh
complete stare feedback. The fuzzy model of such a system has the following form:

UJ..+p =JJ.:O]x+l o OJJ..+p-1 o .f5 fork= 1 ton

The stability of a fuzzy system can be tested by Lyapunov's srabiliry theorem.

114.5 Application of FLC Systems

FLC systems find a wide range of application in various industrial and commercial products and systems.
In several applications- related to nonlinear, rime-varying, ill-defined systems and also complex systems -
FLC systems have proved to be very efficient in comparison with other conventional control systems. The
applications ofFLC systems include:
1. traffic control;
2. steam engine;
3. aircraft flight control;
4. missile control;
5. adaptive control;

L
378 Fuzzy Lo_gic Control Systems

6. liquid-level control;
7. helicopter model;
8. automobile speed controller;
9. braking system controller;
10. process conuol (includes cement kiln comrol);
11. robotic control;
1.2. elevator (aum lift) comrol;
13. automatic runing control;
14. cooling plant control;
15. water uearment;
16. boiler control;
17. nuclear reactor control;
18. power systems control;
19. air conditioner control (temperature controller);
20. biological processes;
21. knowledge based system;
22. faulr detection control unit;
23. fuzzy hardware implementation and fuzzy computers.

Amidst all these practical applications, the best performance was noticed in cement kiln control system. FLC
system has also been successfully implemented to auromacic tuning operations and container crane system. The
application of an FLC system to household purposes include: washing machines, air conditioners, microwave
ovens, cameras, television, palmtop compmers and many others. The companies that manufacture fuzzy
logic technique based appliances as commercial products are Mitsubishi, Hirachi, Sony, Toshiba, Matsushira,
Canon, Sanyo and so on. In the next part of the section, as an illustration of fuzzy logic controller we discuss
che application of fuz.zy logic in aircraft landing control problem in more detail.
Consider an aircraft landing approach (Figure 143). It is necessary tO simulate the final descent approach.
When the aircraft lands onto the ground, the downward velocity is proportional to the square of the heighL
Hence, at higher attitudes, a large downward velocity is desired. When the height starrs decreasing, the desired
downward velocity goes on decreasing. As the height becomes negligibly small, the downward velocity goes
to zero. In this manner, the flight descends from attitude promptly but touches the land very gently. The plot
for desired downward velocity vs. attitude is shown in Figure 14-4.
The variables utilized for performing this simulation are as follows:
1. height above ground, h;
2. vertical velocity of aircraft, v.
The omput to be controlled is the force "f' When this force is applied to the aircraft, it will- alter the
aircrafts height "h" and velocity "v." It is necessary m derive the differential equation for analyzing.
From Figure 14-5, the momentum "a" for a panicle of mass "m" moving with a velocity "v" is given by
rhe product of mass and velocity, i.e. a:;:::: mv. When an external force''/" is applied in a time interval dt
and the panicle of mass "in" continues in the same direction with the same velocity "v", then the change
in velocity is given by 6. v :;:::: /6. tfm. When 6 t:;:::: 1 sand m :;:::: 1.0, we get the change in velocity directly
r-
ii:

I 14.5 Application of FLC Systems

379

;j:
::r0
~c= o):;,
Vertical veloCity
(v) ~ /""
.,. --~- ..,. Landing towards
ground
1i
~I
'
ii\ Height (h)

~~ I 7 7 I I I I I I I I

Ground
I I I 7 I; I l 11

Figure 143 Aircraft landing problem.

ilI
'i
t t
Downward
velocity

0
Heigh! above ground ~
Figure 144 Plor of desired downward vclociry vs. height.

Mass (m)
(or)
Weight(!'.)

l
Velocity (v)

Figure 145 Principle of mass and velociry (a= mv).

proportional to the applied force. Based on this we obtain rhe following set of equations:

Vi+l =v;+Ji; h;+l :;::::h;+v;

where v;+l is rhe new velocicy; v; the old velocity; f, the force; h;+ 1 the new height; h; the old heighr. To
implement an FLC model for this, the following steps should be adopted.
380 Fuzzy Logic Control Systems

Table 141 Membersh~p values for height

Height (F) 0 100 200 300 400 500 600 700 800 900
Large (L) 0 0 0 0 0 0.2 0.4 0.6 0.8 l.O
Medium(M) 0 0 0.2 0.4 0.6 0.8 1 0.8 0.6 0.4
Smoll (S) 1 0.8 0.6 0.4 0.2 0 0 0 0 0

1. Define dte fuzzy membership functions for the state variables (height and velocity).
2. Define the fuzzy membership function for lhe output variable (force).
3. Form rhe fuzzy rule base system model.
4. Based on the fuzzy rules, form the fuzzy associative memory (FAlvi) table. The values in the FAM table
give rhe output (force).
5. Define the initial conditions and carry out simulation for one cycle. Several cycles of simulation can be
carried om. Let the aircraft be scaned at an altirude of900 feet with a downward velocity of-20ft s- 1.
The equations used for updarion of state variables are (for each cycle)

Vi+l = v; +fi; hi+ I = h; + v;

The membership values for height are given in Table 14-1 and its triangular membership conmuction
is shown in Figure 14-6. The membership values for velocity are given in Table 14-2 and irs triangular
membership construction is shown in Figurel4-7. The membership values for control force are given in
Table 14-3 and irs triangular membership consrrucrion is shown in Figure 14-8. The fuzzy rules are formed
as follows:

1. IF height isLAND velocity is D, then conrrol force is Z.

2. If height isLAND velocity is OS, rhen conrrol force is OS.

In a similar manner, rhe other rules are formed. There are rhree linguistic variables defined for height and
five linguistic variables defined for velocity; based on these IS fuzzy rules are formed. The rules are stored in
FAM table (Table 14-4). Here initial height, h0 =900ft; initial velocity, vo = -20 fr s- 1; control force,
[o =to be computed.

p(h) (M) (L)

Medium Large
1.0kall(S)

O.B

0.6

0.4

0.2

0 100 200 300 400 500 600 700 800 900 1000
Height(ft)

Fiaure 146 Membership function of height (h).

L
 14.5 Application of FLC Systems
381
Table 142 Membership values for velocity

Vertical velocity (ftls)

30 25 20 15 -10 -5 0 5 10 15 20 25 30
Up(U) 0 0 0 0 0 0 0 0 0 0.5 1 1 1
Up ,moll (US) 0 0 0 0 0 0 0 0.5 1 0.5 0 0 0
Zem (Z) 0 0 0 0 0 0.5 1 0.5 0 0 0 0 0
Down small (DS) 0 0 0 0.5 I 0.5 0 0 0 0 0 0 0
Down(D) 0.5 0 0 0 0 0 0 0 0 0

t
p(>J (DS) (Z) (US)
Down (D) Down small Zero Up Small Up(U)
1.0

0.8

0.6

0.4

0.2

0
-30 -25 -20 -15 -10 -5 0 5 10 15 20 25 30
v- Vertical velocity (IUs)

Figure 147 Membership function of velocicy.

Table 143 Membership values for control force

Output force (lbs)

30 25 20 15 10 -5 0 5 10 15 20 25 30
Up (U) 0 0 0 0 0 0 0 0 0 0.5 0
Up moll (US) 0 0 0 0 0 0 0 0.5 I 0.5 0 0 0
Zero (Z) 0 0 0 0 0 0.5 I 0.5 0 0 G 0 0
Down small (DS) 0 0 0 0.5 I 0.5 0 0 0 0 0 0 0
Down (D) 0.5 0 0 0 0 0 0 0 0 0

Height h (900) fires L ar 1.0 and M at 0.4; velocity t (-20) fires only 0 at 1.0.

Height Velocity Output

L (1.0) AND D (1.0) => z (1.0)
M (0.4) AND D (1.0) => us (0.4)
The defuzzification can be carried out and rhe crisp quantity can be extracted. Figure 14-9 shows the
consequems rruncared and union of fuzzy consequent for cycle 1. The output is[o = 5.2lbs (approximarely).

L
 Fuzzy logic Control Systems
382

t (OS) (Z) (US) (U)

I'( f) 1 Down (D) Down small Zero ...!psmall Up
1.0

0.8

0.6

0.4

0.2

0 L_,-~~~~~~--~~--~~-,r-+
-30 -25 -20 -15 -10 -5 0 5 10 15 20 25 30
Control output force (lbs)

Figure 148 Membership value of conrrol force.

Table 144 FAM <able

Height I Velocity D DS z us v
L z DS D D D
M us z DS D D
s u us z DS DS

1.0 1.0
I us

0.4

h~
I~
0 10 20 v~

1.0

~~~~~~~c~" 'F

fo (Defuzzifier value)

Figure 149 Union of fuzzy.consequents for cycle 1.

 14.8 Exercise Problems 383

Based on this, the new values of ilie state variables and outpUt for the next cycle are given by

h, = ho +vo = 900+(-20) =880ft

., = vo + fii = -20+ s.2 = -14.8 rr,- 1
These are used as the initial values for the next cycle. A number of cycles are carried out until we get
a decent pro'file as shown in Figure 14-3. Generally, a fuzzy lOgic controller has only a single-layer-rule
firing.

114.6 Summary
The basic architecrure and design aspects of fuzzy logic controller are introduced in this chapter. Also, an
applicacion to aircraft landing problem has been dea1t with in detail.
The main key behind the fuzzy logic controller is rhe set of fuzzy control rules, which describes the
input-output relationship of a controlled system. The two types of fuzzy control rules used in the design of
fuzzy logic controllers are state evaluation and object evaluation. This chapter mainly focuses on rhe state
evaluation rules, because they find a wide application. The object evaluation fuzzy conuol rules predict the
present and future control actions; in addition the control objectives are evaluated. If these objectives are
satisfied, then the control action is applied to the process.
The concepts of stability, observabiliry and controllability are wdl-esrablished in modern control theory.
Owing to the complexity of mathematical analysis of fuzzy logic controllers, rhe notions of stability and
concepts of automatic control theory for fuzzy logic controllers are under research.

114.7 Review Questions

1. Stare the importance of a control system. 8. Give the principle design element necessary for
2. What are the two types of control systems? the design of general fuzzy logic controller.

3. Differentiate between open-loop and closed- 9. Mention the features of a simple FLC system.
loop control systems. 10. What arc the special forms of FLC system
4. List the various control system design aspects . models?

5. Mention the four structures of fuzzy production 11. List the various applications of fuzzy logic con-
rule system. troller.

6. With a neat block diagra:n, explain the architec- 12. With a suitable application case study explain a
ture of a fuzzy logic conuoller. fuzzy logic controller.

7. What are the steps involved in designing a fuzzy

logic controller?

I I 14.8 Exercise Problems

-I 1. Write a computer program to implement a fuzzy

logic comroller for a aircraft landing problem dealt
2. Using fuzzy logic controller, simulate the camera
tracking conuol system.

~
in Section 14.5.
..

t
 Fuzzy Logic Control Systems
384

3. Design a fuzzy logic controller to simulate a

temperature conuol System for a room.
4. lmplemenr a process conrrol applicacion via a
fuzzy logic controller.
S. Design and analyze a fuzzy controller for an
inverted pendulum as shown in Figure l.

Figure 1 In pur pattern.

\
'
!'
I

~
}'
 Genetic Algorithm 15
Learning Objectives - - - - - - - - - - - - - - - - - - ,
Gives an introduction ro naruraJ evolution. GA, parallel GA and independem sampling
Lists rhe basic operators (selection, crossover, GA.
mutation) and other terminologies used in The variants of parallel GA (fine-grained par-
Generic Algorithms (GAs). allel GA and coarse-grained parallel GA) arc
Discusses the need for schemata approach. included.

Details the comparison of traditional algo- Enhances the basic concepts involved in Hol-
rithm with GA. land classifier system.

Explains the opera.tional flow of simple GA. The various features and operational proper-
ties of genetic programming are provided.
Description is given of the various classifica-
tions ofGA- Messy GA, adaptive GA, hybrid The application areas ofGAare also discussed.

Charles R. Darwin says rhat "Although the beliefrhat tin organ so perfect as the eye could have been formed
by natuml selection is enough to stagger tlnJ one; yet in the case ofnny organ, ifwe know ofn kmg series of
gradations in complexity, each goodfor its possessor, then, undercha11gingconditions ofli.Je, there is no logical
impossibility in the acquiremem ofany conceivable degree ofpeJfection through namml selection."

lts.t Introduction
Charles Darwin has formulated the fundamental principle of natural selection as the main evolutionary
tool. He put forward his ideas without the knowledge of basic hereditary principles. In 1865, Gregor Mendel
discovered these hereditary principles by the experiments he carried out on peas. After Mendel's work genetics
was developed. Morgan experimentally found that chromosomes were the carriers of hereditary informa-
tion and chat genes representing the hereditary factors were lined up on chromosomes. Darwin's natural
selection theory and natural generics remained unlinked until 1920s when it was proved that genetics and
selection were in no way contrasting each ocher. Combination of Darwin's and Mendel's ideas lead to rhe
modern evolutionary theory.
In The Origin ofSpecies, Charles Darwin stated the theory of natural evolution. Over many generations,
biological organisms evolve according ro the principles of narural selection like "survival of the fittest" to
reach some remarkable forms of accomplishment. The perfecr shape of the albatross wing, the efficiency and

I the similarity berween sharks and dolphins and so on are good examples of what random evolution wicll
absence ofinreUigence can achieve. So, ifirworks so well in nature, it should be interesting to simulate natural
evolution and ely to obtain a method which may solve concrete search and optimization problems.
For a better understanding of cllis theory, it is important first to understand the biological terminology
used in evolutionary ~amputation. It is discussed in Section 15.2.
 386 Genetic Algorithm

In 1975, Holland developed this idea in Adaptation in Natural and Artificiizl Systems. By describing how
to apply the principles Of natural evolution to optimization problems, he laid down the first GA. Holland's
theory has been further developed and now GAs stand up as powerful adaptive methods to solve search
and optimization problems. Today, GAs are used to resolve complicated optimization problems, such as,
organizing the time table, scheduling job shop, playing games.

I 15.1.1 What are Genetic Algorithms?

GAs are adaptive heuristic search algorithms based on the evolutionary ideas of natural selection and generics.
As such they represent an intelligent exploitation of a random search used to solve optimization problems.
Ahhough randomized, GAs are by no means random; instead iliey exploit historical information to direct
the search imo the region of better performance within the search space. The basic techniques of the GAs
are designed to simulate processes in natural systems necessary for evolution, especially those that follow the
principles first laid down by Charles Darwin, "survival of the fittest," because in narure, competition among
individuals for seamy resources results in the fittest individuals dominating over the weaker ones.

I 15.1.2 Why Genetic Algorithms?

"':'" They are better than conventional algorithms in rhat they are more robust. Unlike older AI systems, they do
not break easily even if the inputs are changed slightly or in the presence of reasonable noise. Also, in searching
a large state~space, rnultimodalstate~space or n~dimensional surface, a GA may offer significant benefilS over
more typical optimization techniques (linear programming, heuristic, depth~ first, breath~first and praxis.)
,..
115.2 Biological Background
The science that deals with the mechanisms responsible for similarities and differences in a species is called
' Genetics. The word "genetics" is derived from the Greek word "genesis" meaning "to grow" or "to become."
The science of generics helps us to differentiate be[Ween heredity and variations and accounts for the resem~
blances and differences during the process of evolution. The concepts of GAs are directly derived from natural
evolution and heredity. The terminologies involved in the biological background of species are discussed in
the following subsections.

I 15.2.1 The Cell

Every animal/human cell is a complex of many "small" factories that work together. The center of all this is
the cell nucleus. The genetic information is contained in the cell nucleus. Figure 15-1 shows anatomy of the
animal cell and cell nucleus.

I 15.2.2 Chromosomes

All the genetic information gets stored in the chromosomes. Each chromosome is build of deoxyribonucleic
acid (DNA). In humans, chromosomes exist in pairs (23 pairs found). The chromosomes are divided into
several parts called genes. Genes code the properties of species, i.e., the characteristics of an individual The
possibilities of combination of the genes for one property are called alleles, and a gene can take different all des.
For example, there is a gene for eye color, and all the different possible alleles are black, brown, blue and green
(since no one has red or violet eyes!). The set of all possible alleles present in a particular population forms
a gene pool. This gene pool can determine all the different possible variations for the future generations. The
size of the gene pool helps in determining the diversity of the individuals in the population. The set of all the
genes of a specific species is called genome. Each and every gene has a unique position on the genome called
II 15.2 Biological Background

Anatomy of the animal cell

387

Mitochondria
'

Micro
tuubules

endoplasmic
reticulum

The cell nucleolus

Nucleolus

pores

Chromosomes

Figure 151 Anatomy of animal cell, cell nucleus.

kcus. In faa, most living organisms store their genome on several chromosomes, but in the GAs, all the genes
are usually stored on the same chromosomes. Thus, chromosomes and genomes are synonyms with one other
in GAs. Figure 15-2 shows a model of chromosome.

.I 15.2.3 Genetics

For a particular individual, the entire combinacion of genes is called genotype. The phenotype describes the
physical aspect of decoding a genotype to produce the phenotype. One interesting point of evolution is rhat
selection is always done on the phenotype whereas the reproduction recombines genotype. Thus, morpho-
genesis plays a key role between sdection and reproduction. In higher life forms, chromosomes contain {Wo
sets of genes. These are known as diploids. In the cru;e of conflicts be{Ween {WO values of the same pair of genes,
the dominant one will determine the phenotype whereas the other one, called recessive, will still be present and
388 Genetic Algorithm

Figure 152 Model of chromosome.

Figure 153 Development of genotype ro phenotype.

can be passed omo the offspring. Diploidy allows a wider diversity of alleles. This provides a useful memory
mechanism in changing or noisy environment. However, most GAs concemrare on haploid chromosomes
because they are much simple ro construct. In haploid representation, only one set of each gene is stored, rhus
the process of determining which allele should be dominant and which one should be recessive is avoided.
Figure 15~3 shows the development of genorype to phenotype.

I 15.2.4 Reproduction

Reproduction of species via genetic information is carried out by the following;

1. Mitosis: In mitosis the same genetic informacion is copied to new offspring. There is no exchange of
informacion. This is a normal way of growing of multicell strucrures, such as organs. Figure 15-4 shows
_,i
mitosis form of reproducrion. I
2. Mtiosis: Meiosis forms the basis of sexual reproduction. When meio!ic division rakes place, two gametes
,I
appear in the process. When reproduction occurs, these two gamerescPnjugate to a zygote which becomes ;.I
the new individual. Thus in this case, the generic informacion is shared between the parents in order to
create new offspring. Figure 15-5 shows meiosis form of reproduccion. .
r_-1

i;i

L
 15.2 Biological Background 389

replication

H
(@)(@
>-Cell
division

Figure 154 Mirosis form of reproduction.

DNA
Meiotic
Replication
Division 1
and Recombination

Meiotic

Cell
Division 2

Figure 155 Meiosis form of reproduction.

L
 390 Genetic Algorithm

Table 151 Comparison of natural evolution and genetic algorithm terminology

Natural evolution Genetic algorithm
Chromosome String
Gene Feature or character
Allele Fearure value
Loous String posicion
Genocype Srrucrure or coded string
Phenocype Parameter set, a decoded structUre

I 15.2.5 Natural Selection

The origin of species is based on "Preservation of favorable variations and rejection of unfavorable variations."
The variation refers to the differences shown by the individual of a species and also by offspring's of the
same parents. There are more individuals born than can survive, so iliere is a continuous struggle for life.
Individuals wirh an advantage have a greater chance of survival, i.e., the survival of the fittest. For example,
Giraffe with long necks can have food from tall uees as well from ilie ground; on the other hand, goat and
deer having smaller neck can have food only from the ground. As a result, natural selection plays a major role
in this survival process.
Table 15.1 gives a list of different expressions, which are common in natural evolution and genetic
algorithm.

.- 115.3 Traditional Optimization and Search Techniques

The basic principle of optimization is the efficient allocation of scarce resources. Optimization can be applied
ro any scientific or engineering discipline. The aim of optimization is to find an algorithm which solves a given
class of problems. There exists no specific method which solves all optimization problems. Consider a &merion,

f(x) : [x 1,x")-> [0, 1) (15.1)

where
f(x)= \ 1 if llx-aii<E, E>O
-I elsewhere

For the above funaion,fcan be maintained by decreasing E or by making the interval of[x1 , x"] large. Thus,
a difficult task can be made easier. Therefore, one can solve optimization problems by combining human
creativity and the raw processing power of rhe computers.
The various conventional optimization and search techriiques available are discussed in the following
subsections.

15.3.1 GradientBased Local Optimization Method

When the objective function is smooth and one needs efficient local optimization, it is bener to use gradient-
based or Hessian-based optimization methods. The performance and reliability of the different gradient
meiliods vary considerably. To discuss gradient-based local optimization, let us assume a smooili objective
funa:ion (i.e., continuous f!rsr and second derivatives). The object function is denoted by

[(x) : K'-> R (15.2)

 II 15.3 Traditional Optimization and Search Techniques

The first derivatives are oomained in rhe gradient vector 'iJf(x)

391

0 j(x)/o Xi]
Vf(x) = : (15.3)
[
. &f(x)l& x,

The second derivatives of the object function are contained in the Hessian manix H(x):

i
&2 f(x)
~
&
2
[(x)l
ax1Bxn
H(x) = 'VT'Vf(x) = : (15.4)
( &'f(x) &' [(x)
ax! i.Jxn a2xn
Few methods need only the gradient vector, but in the Newton's method we need the Hessian matrix. The
general pseudocode used in gradient methods is as follows:
Select an initiaJ guess value x 1 and set n = I.
Repeat
Solve the search direction pn from Eq. (15.5) or (15.6) below.
Determine the next iteration point using Eq. (15.7) below:
xn+I = X"+An P"

Setn=n+l.
Until \\X 11 -X 11 - 1 \\< E
These gradient methods search for minimum and nor maximum. Several different methods are obtained
based on the details of the algorithm.
The search direction pn in conjugate gradient method is found as follows:
P" = -Vf(X")+fi,P"- 1 (15.5)

In secant method,

B,P" = -Vf(x") (15.6)

is used for finding search direction. The matrix B11 in Eq. (15.6) estimates the Hessian and is updated in each
I iteration. When B11 is defined as rhe identity matrix, the steepest descem method occurs. \Xfhen rhe matrix
Bn is the Hessian H(>fl), we get the Newton's method.
Ij
The length A, of the search step is computed using:

An= argminf(:/'+AP 11 ) (15.7)

l bO
The discussed is a one-dimensional optimization problem. The steepest descent method provides poor perfor-

I mance. As a result, conjugate gradient method can be used. If the second derivatives are easy to compute, rhen
Newton's method may provide best results. The secant methods are faster than conjugate gradient merhods,

~.l
bur there occurs memory problems. Thus, rhese local oprimizarion methods can be combined with other
methods to get a good link berween performance and reliability.

.~
' --
392 Genetic Algorithm -l
I 15.3.2 Random Search
I
Random u11rch is an eXtremely basic method. Ir only explores the search space by randomly selecting solu-
tions and evaluates their fitness. This is quire an uninrelligem srmegy, and is rarely used. Nevertheless, dtis
method is sometimes worth resting. It doesn't rake much effort to implement it, and an imporram number
of evaluations can be done fairly quickly. For new unresolved problems, it can be useful to compare the I
resulrs of a more advanced algorithm ro those obtained just with a random search for che same number
of evaluations. Nasp;y surprises might well appear when comparing, for example, GAs to random search.
Ir's good ro remember that the efficiency of GA is extremely dependent on consisrenr coding and relevant
I
reproduction operators. Building a GA which performs no more than a randOm search happens more often
chan we em expect. If rhe reproduction operators are jrnt producing new random solutions without any
concrete links to the ones selected from the last generation, rhe GA is jrnt doing nothing else chan a random
s=ch.
I
Randotn search does have a few interesting qualities. However good rhe obtained solution may be, if it's
not optimal one, it can be always improved by continuing the run of the random search algorithm for long
enough. A random search never gers stuck at any point such as a local optimum. Furthermore, theoretically,
if the search space is finite, random search is guaranteed w reach the optimal so!U[ion. Unfortunately, this
result is completely useless. For mosr of problems we are interested in, exploring the whole search space takes
a lor of rime.
I
115.3.3 Stochastic Hill Climbing

Efficient methods exist for problems with well-behaved continuous fitness functions. These methods use a
kind of gradient ro guide the direction of search. Stochmtic hill climbing is rhe simplest merhod of rhese kinds.
Each iteration consists in choosing randomly a solution in the neighborhood of the current solurion and
retains this new solution only if ir improves rhe fitness funcrion. Stochastic hill climbing converges towards
rhe optimal solution if rhe fitness function of rhe problem is conrinuous and has only one peak (unimodal
function).
On functions with many peaks (mulrimodal funcrions), rhe algorithm is likely to stop on the first peak
ir finds even if it is nor the highest one. Once a peak is reached, hill climbing cannot progress anymore,
and rhat is problematic when chis point is a local oprinmm. Stochastic hill climbing usually starrs from a
random select point. A simple idea to avoid gening stuck on rhe first local optimal consists in repeating
several hill climbs each rime starting from a different randomly chosen point. This method is sometimes
known as iterated hill climbing. By discovering different local optimal points, chances ro reach the globaJ
optimum increase. It works wdl if rhere are nor roo many local optima in the search space. However, jf rhe
firness function is very "noisy" wirh many small peaks, stochastic hill climbing is definitely nor a good method
to use. Neverrheless, such methods have the advantage of being easy to implement and giving fairly good
solutions very quickly.

115.3.4 Simulated Annealing

-'
Simulated annealing (SA) was originally inspired by formation ofcrystal in solids during cooling. As discovered I
a long rime ago by Iron Age blacksmirhs, the slower the cooling, rhe more perfect is the crystal formed. By "I
cooling, complex physical sysrems naturally converge rewards a stare of minimal energy. The system moves cl
randomly, bur the probability to stay in a parricularconfiguration depends directly on rhe encrgyofthesystem

:I
L
 15.3 Traditional Oplimization and Search Techniques 393

and on irs temperature. This probability is formally given by Gibbs law:

,"' = eflkT (15.8)

where stands for rhe energy, k is rhe Boltrzmann con.sram and Tis the temperature. In the mid0l970s,
Kirkpatrick by analogy of this physical phenomena; laid out the first description of SA.
As in the stochastic hill climbing, rhe iteration of rhe SA consists of randomly choosing a new solution in
rhe neighborhood of rhe acrual solution. If the firness function of rhe new solution is better than rhe firn~
function of the current one, the new solution is accepted as the new current solution. If the fitness function
is nor improved, the new solurion is retained w~rh a probability:

P = ('-lj"(y)-[(x)[lkT (15.9)

wherej(y) - J(x) is rhe difference of the fitness function belween rhe new and rhe old solution.
The SA behaves like a hill climbing merhod but with rhe possibility of going downhill to avoid being
trapped at local optima. When rhe temperature is high, rhe probability of deteriorate the solution is quire
important, and chen a lor of large moves are possible ro explore the search space. The more the temperature
decreases, rhe more difficult it is to go downhill. The algorithm thus rries ro climb up from the current
solution ro reach a maximum. When temperature is lower, rhere is an exploitation of the current solution. If
the temperature is too low, number deterioration is accepted, and rhe algorithm behaves just like a srochastic
hill climbing method. Usually, rhe SA srarrs from a high temperature which decreases exponentially. The
slower the cooling, rhe better it is for finding good solutions. lr even has been demonstrated that with an
infinitely slow cooling, rhe algorithm is almost certain to find the global optimum. The only point is char
infinitely slow cooling consists in finding the appropriate temperature decrease rare ro obtain a good behavior
of the algorithm.
SA by mixing exploration features such as rhe random search and exploitation features like hill climbing
usually gives quite good results. SA is a se"rious competitor of GAs. It is worth trying co compare the result5
obtained by each. Both are derived from analogy with natural system evolution and borh deal wich rhe same
kind of optimization problem. GAs differ from SA in two main features which makes them more efficient.
First, GAs use a population-based selection whereas SA only deals with one individual at each iteration. Hence
GAs are expecred ro cover a much larger landscape of the search space ar each iteration; however, SA iterations
are much more simple, and so, often much F.tsrer. The grcar advantage of GA is irs exceptional ability ro be
paral!elized, whereas SA does nor gain much of chis. It is mainly due to rhe popularion scheme use by GA.
Second, GAs use recombination operators, and are able to mix good characteristics from different solutions.
The exploitation made by recombination operators arc supposedly considered helpFul to find optimal solmions
of the problem. On rhe orher hand, SA is still very simple to implement and gives good resulrs. SAs have
proved their efficiency over a large spectrum of difficult problems, like the optimal layout or primed circuit
board or the famous traveling salesman problem.

15.3.5 Symbolic Artificial Intelligence

Most symbolic artificial intelligence (AI) systems are very static. Most of them can usually only solve one
given specific problem, since their architecture was designed for whatever rhar specific problem was in rhe
first place. Thus, if rhe given problem were somehow robe changed, these systems could have a hard rime
adapting co them, since rhe algorithm char would originally arrive co the solution may be either incorrect
or less efficient. GAs were created ro combat rhese problems. They are basically algorithms based on natural
biological evolution. The architecture of sysrems that implement GAs is more able to adapt ro a wide range of

L
394 Genetic Algorithm

problems. A GA functions by generating a large set of possible solutions to a given problem. It then evaluates
each of chose solutions, and decides on a "fimess level" (you may recall the phrase: "survival of the fittest")
for each solution ser. These solutions rhen breed new Solutions. T~e pacem solut.ions that were more "fir"
are more likely m reproduce, while those that were less "fir" are more unlikely to do so. In essence, solutions
are evolved over rime. This way we evolve oUr search space scope to a poinr where you can find the solution.
GAs can be incredibly efficiem if programmed correctly.

lt5.4 Genetic Algorithm and Search Space

Evolucionary computing was imroduced in the 1960s by I. Rechenberg in the work "Evolution Strategies."
This idea was then developed by other researches. GAs were invented by John Holland and developed chis
idea in his book "Adaptation in Natural and Artificial Systems" in the year 1975. Holland proposed GA
as a heuristic method based on "survival of the finest." GA was discovered as a useful tool for search and
optimization problems.

I 15.4.1 Search Space

Most often one is looking for the best solution in a specific set of solutions. The space of all feasible solutions
(rhe set of solutions among which the desired solution resides) is called search space (also state space). Each
and every point in rhe search space represems one possible solution. Therefore, each possible solution can be
"marked" by its fitness value, depending on rhe problem definition. With GA one looks for the best solution
among a number of possible solutions- represented by one poim in the search space; GAs are used to search
the search space for che best. solution, e.g., minimum. The difficulties in chis case are che local minima and
the starting point of the search. Figure 15~6 gives an example of search space.

2.5 ,-----,-,----,-,---,---,----,---,---,--,

2 ---~---~--~-~--~-~-~--L
I I I I

''
''
---~--- --~----~----+----~
1.5

.
I I I I I I
- - - - , - - - - , - - - -"T- - - -r- -- -r- ---,---
' '
. . .
0.5 ----,-----,----
I I I
~- ---,----,-----,-----,--
I I I I
-~----,

'
I I I I
' '
''
0
0 100 200 300 400 500 600 700 800 900 1000

Figure 15~6 Ail example of search space.

 15.4 Genetic Algorithm and Search Space 395

115.4.2 Genetic Algorithms World

GA raises again a couple of important features. First, iris a stochtJ.Stic algorithm; ~andomness has an essential
role in GAs. Both selection and reproduction need random procedures. A second very imponam poim is
rhat GAs always consider a population of solutions. Keeping in memory more chan a single solution at each
iteration offers a lot of advantages. The algorithm can' recombine different solutions to gee better ones and so
it can use che benef'iLS of assortmem. A population-based algorithm is also very amenable for parallelization.
The robu.stnm of the algorithm should also be menrioned as somerhing essential for the algorithm's success.
Robustness refers to che ability to perform consis.rently well on a broad range of problem rypes. There is no
particular requirement on rhe problem before u;ing GAs, so it can be applied to resolve any problem. All
these features make GA a really powerful optimization roo!.
With rhe success of GAs, ocher algorithms making use of the same principle of natural evolution haVe
also emerged. Evolution strategy, generic programming are some algorithms similar to these algorithms. The
classification is nor always clear between the different algorithms, rhus to avoid any confusion, rhey are all
gathered in what is called Evoh1tionmy Algorithms.
The analogy with nature gives these algorithms something exciting and enjoyable. Their ability ro deal
successfully with a wide range of problem area, including those which are difficult for other methods to solve
makes them quite powerful. However coday, GAs are suffering from mo much rrendiness. GA is a new fifld,
and partS of rhe theory still have to be properly established. We can find almost as many opinions on GAs as
there are researchers in this field. In this document, we will generally find the most current point of view. Bur
things evolve quickly in GAs roo, and some comments might not be very accur:ue in few years.
It is also important to mention GA limits in chis introduction. Like most srochascic methods, GAs are
not guaranteed to find the global optimum solmion ro a problem; they are satisfied with finding "acceptably
good" solutions co che problem. GAs are extremely general too, and so specific techniques forsolvingparricular
problems are likely to our-perform GAs in borh speed and accuracy of the final result. GAs are something
wonh trying when everything else fails or when we know absolutely nothing of the search space. Nevertheless,
even when such specialized techniques exist, it is often interesting ro hybridize them with a GA in order
to possibly gain some improvements. h is important always tO keep an objective point of view; do not
consider that GAs are a panacea for resolving all optimization problems. This warning is for those who
might have the remprarion ro resolve anything wirh GA. The proverb says "If we have a hammer, a.ll the
problems look like a nails.'' GAs do work and give excellent results if they are applied properly on appropriate
problems.

115.4.3 Evolution and Optimization

To depict the importance ofevolution and optimization process, consider a species Basilosaurus char originated
45 million years ago. The Basilosaurus was a prototype of a whale (Figure 15-7). It was about 15 m long and

J
I

~
' Figure 157 Basilosaurus.
396 Genetic Algorithm

="

Figure 158 Tursiops flipper.

weighed approximately S tons. It still had a quasi~independem head and posterior paws, and moved using
undulatory movementS and hunted small preys. Its anterior members were reduced to small flippers with an
elbow anicularion, Movements in such a viscous element (water) are very hard and require big efforrs. The
anterior members ofbasilosaurus were not really adapred ro swimming. To adapt them, a double phenomenon
must occur: the shortening of rhe "arm" wirh the locking of the elbow aniculation and rhe extension of rhe
fingers consrimte ilie base srrucrure of the flipper (refer Figure 15-8).
The image shows that two fingers of rhe common dolphin are hypemophied to the detriment of rhe rest
of the member. The basilosaurus was a hunter; ir had to be fast and precise. Through rime, subjects appeared
with longer fingers and short arms. They could move faster and more precisely than before, and therefore,
live longer and have many descendants.
Meanwhile, other improvemems occurred concerning rhe general aerodynamic like the integration of the
head to the body, improvemem of the profile, strengthening of the caudal fin, and so on, ~lnally producing
a subject perfectly adapted to the constraints of an aqueous environment. This process of adaptation and
this morphological optimization is so perfect rhar nowadays the similarity between a shark, a dolphin or a
submarine is striking. The first is a cartilaginous fish (Chondrichtyen) that originated in rhe Devonian period
(-400 million years), long before the apparition of rhe first mammal. Darwinian mechanism hence generated
an optimization process -hydrodynamic optimization- for fishes and others marine animals- aerodynamic
optimization for pterodactyls, birds and bars. This observation is rhe basis of GAs.

15.4.4 Evolution and Genetic Algorithms

The basic idea is as follows: the generic pool of a given population potemially con rains rhe solution, or a berter
solmion, to a given adaptive problem. This solurion is nor "acrive" because rhe generic combination on which
it relies is split among several subjects. Only rhe association of different genomes can lead ro rhe solution.
SimplisticaJly speaking, we could by example consider rhat the shortening of rhe paw and rhe extension of
the fingers of our basilosaurus are controlled by two "genes." No subject has such a genome, bur during
reproduction and crossover, new genetic combination occur and, finally, a sub jeer can inherit a "good gene"
'
from both parents his paw is now a flipper. r'
Holland method is especially effective because he nor only considered the role of mutation {mutations
improve very seldom rhe algorithms), but also utilized generic recombination (crossover): these recombination,
d~e crossover of parrial solutions, greatly improve the capability of the algorithm to approach, and evcnruzlly
Ir
find, the optimum.
Recombination or sexual reproduction is a key operator for natural evolution. Technically, it takes two
genotypes and it produces a new genotype by mixing the gene found in the originals. In biology, the most J
!
common fnrm of recombination is crossover: two chromosomes are cur at one point and the halves are spliced ;I
to create new chromosomes. The effect of recombination is very important becal!.se it allows characteristics -r
from rwo different parenrs to be assorted. If the father and the mother possess different good qualities, we ~I
would expect that all the good qualities will be passed to the child. Thus the offspring, just by combining all l

L
15.5 Genetic Algorithm vs. Traditional Algorithms 397

the good features from its parems, may surpass irs ancestors. M:iny people believe that this mixing of genetic
material via sexual reproduction is one of the most powerful fearures of GAs. As a quick parenthesis about
sexual reproduction, GA represenrarion usually does 'not differentiate male and female individuals (without
any perversity). As in many livings species (e.g., snails) <~:ny individual can be either a male or a female. In
fact, for almost all recombination operators, mother .and father are interchangeable.
Mutation is the other way to get new genomes. Mutation consists in changing the value of genes. In natural
evolution, mutation mostly engenders non-viable genomes. Actually mmation is not a very frequem operator
in natural evolution. Nevertheless, in optimization, a few random changes can be a good way of exploring
the search space quickly.
Through those low-level notions of generic, we have seen how living beings store their characteristic
information and how this information can be passed into their offspring. It very basic but it is more than
enough ro understand rhe GA theory.
Darwin was totally unaware of rhe biochemical basics of genetics. Now we know how the genetic inherita-
ble information is coded in DNA, RNA, and proteins and that the coding principles are actually digital, much
;esembling the information storage in compmers. Information processing is in many ways totally different,
however. The magnificent phenomenon called the evolution of species can also give some insight inro infor-
mation processing methods and oPtimization, in particular. According to Darwinism, inherited variation is
characterized by the following properties:

1. Variation must be copying because selection does nor create directly anything, bm presupposes a large
population to work on.
2. Variation must be small-scaled in practice. Species do not appear suddenly.
3. Variation is undirected. This is also known as the blind watch maker paradigm.

While the natural sciences approach to evolution has for over a century been to analyze and study different
aspecrs of evolution to find the underlying principles, rhe engineering sciences are happy to apply evolutionary
principles, that have been heavily tested over billions of years, to arrack the most complex technical problems,
including protein folding.

115.5 Genetic Algorithm vs. Traditional Algorithms

The principle of GAs is simple: imirare genetics and natural selection by a computer program: The param-
eters of the problem are coded most naturally as a DNA- like linear data structure, a vector or a suing.
Sometimes, when the problem is naturally r:wo or three dimensional, corresponding array structures are
used.
A ser, called populn.tion, of these problem-dependent parameter value vectors is processed by GA. To starr,
there is usually a totally random population, rhevalues of different parameters generated by a random number
generator. Typical population size is from few dozens to tlwusands. To do optimization we need a cost function
or Hmess function as it is usually called when GAs are used. By a fimess function we can select the best solution
candidates from the population and delete the not so good specimens.
The nice thing when comparing GAs to other optimization methods is that the fitness function can be
nearly anything that can be evaluated by a computer or even something that cannotl In the latter case it might
be.a human judgment that cannot be seated as a crisp program, like in the case of eye wimess, where a human
bemg selects from the alternatives generated by GA. So, there are nor any definite mathematical restrictions

L
on the properties of the fimess ftmction. Ic may be discrete, mulrimodal, ere.
398 Genetic Algorithm

The main criteria used ro classify optimization algorithms are as follows: continuous/discrete, con-
suainedlunoonsuained and sequemial/parallel. There is a clear difference berween discrete and continuous
problems. Therefore, it is instructive to notice that continuous methods are sometimes used to solve inher-
ently discrete problems and vice versa. Parallel algorithms are usually used to speed up processing. There are,
however, some cases in which it is more efficient to run several processors in parallel rather than sequentially.
These cases include among .mhers those in which there is high probabilicy of each individual search run to
get sruck into a local extreme.
Irrespective of the above classification, opdmizarion methods can be further classified imo deterministic
and non-deterministic methods. ~n-addition, optimization algorithms can be classified as local or global. In
terms of energy and entropy local search corresponds to entropy while global optimization depends essentially
on the fitness, i.e., energy landscape.
GA differs from convemionaJ optimization techniques in following ways:
I. GAs operate with coded versions of rhe problem parameters rather chan parameters themselves, i.e., GA
works with the coding of solution sec and nor with the solution irself.
2. Almost all conventional optimization techniques search from a single point, bur GAs always operate on a
whole population of points (strings), i.e., GA uses population of solutions rather chan a single solution for
searching. This plays a major role ro che robustness of GAs. It improves the chance of reaching the global
optimum and also helps in avoiding local stationary point.
3. GA uses fimess ftmcrion for evaluation rather than derivatives. As a result, they can be applied tO any kind
of continuous or discrete optimization problem. The key point to be performed here is ro identify and
specify a meaningful decoding function.
4. GAs use probabilistic transition operates while conventional methods for continuous optimization apply
deterministic transition operates, i.e., GAs do nor use deterministic rules.
These are the major differences rhat exist between GA and conventional optimization techniques.

Its.& Basic Terminologies in Genetic Algorithm

The two distinct elements in the GA are individuals and populations. An individual is a single solution while
the population is the set of individuals currently involved in the search process.

I 15.6.1 Individuals

An individual is a single solution. An individual groups rogerher two forms of solutions as given below:
I. The chromosome which is rhe raw "genetic" informacion (genotype) char the GA deals.
2. The phenotype which is the expressive of rhe chromosome in rhe terms of the model.
A chromosome is subdivided into genes. A gene is the GA's representation of a single factor for a control
factor. Each factor in che solution set corresponds to a gene in the chromosome. Figure 15-9 shows the
represemation of a genotype.
A chromosome should in some way conlain informacion about the solution that ir represenrs. The mor-
phogenesis function associates each genotype wirh irs phenotype. It simply means that each chromosome must
define one unique solution, but it does not mean that each solution is encoded by exactly one chromosome.
Indeed, the morphogenesis function is not necessarily bijective, and it is even sometimes impossible (especially
with binary representation). Nevertheless, the morphogenesis function should at lease be subjective. Indeed;
 15.6 Basic Terminologies in Genetic Algorithm 399

Solution Set Phenotype

Factor 1 Factor 2 Factor 3 FactorN

t t t t
Gene Ge~~ __ Gene 3 l GeneN

I
1 \ J

Chromosome Genotype

I Figure 159 Representation of genorype and phenotype.

11010101110101101

Figure 1510 Represemation of a chromosome.

I
r
all rhe candidate solutions ofrhe problem must correspond to at leasrone possible chromosome, to be sure that
the whole search space can be explored. When the morphogenesis function char associates each chromosome
~ ro one solution is not injective. i.e., different chromosomes can encode the same solution, the represenlation
' is said to be degenerated. A slight degeneracy is not so worrying, even if the space where the algorithm is
looking for the oprimal solution is inevitably enlarged. Bur a roo important degeneracy could be a more serious
problem.lt can badly affect rhe behavior of the GA, mostly because if several chromosomes can represent the
same phenotype, rhe meaning of each gene will obviously not correspond to a specif1c characteristic of the
solution. It may add some kind of confusion in the search. Chromosomes encoded by bit strings are given in
Figure 15-10.

115.6.2 Genes

Genes are the basic "instructions" for building a GA. A chromosome is a sequence of genes. Genes may describe
a possible solution ro a problem, without actually being the solution. A gene is a bir string of arbitrary lengths.
The bit string is a binary representation of number of intervals from a lower bound. A gene is theGNs represen-
tation of a single factor value for a control factor, where control fucror must have an upper bound and a lower
bound. This range can be divided into the number of intervals rhat can be expressed by the gene's bit string.
A bit string oflengrh "n" can represent (2'1 - 1) intervals. The size of rhe interval would be (range)/(2n- 1).
The mucrure of each gene is defined in a record of phenoryping parameters. The phenotype parameters
are instructions for mapping bet~.veen genotype and phenotype. It can also be said ~ encoding a solution
set into a chromosome and decoding a chromosome to a solution set. The mapping between genotype and
phenotype is necessary to convert solution sets from the model into a form that the GA can work with, and
i for converting new individuals from the GA into a form that rhe model can evaluate. In a chromosome, the

I genes are represented as shown in Figure 15-11.

I 115.6.3 Fitness
The firness of an individual in a GA is the value of an objeaive function for its phenotype. For calculating

1
fimess, the chromosome has ro be first decoded and the objective function has to be evaluated. The fitness
400 Genetic Algorithm

~-, o 1 of111 o TTY! 1 JO -1-01 J

t t t t
Gene 1 Gene2 Gene 3 Gene4

Figure 1511 Represemadon of a gene.

not only indicates how good the solucion is, but also corresponds ro how dose the chromosome is ro the
optimal one.
In the case of multicrirerion optimization, rhe fitness function is definitely more difficult tO determine. In
multicriterion optimization problems, there is often a dilemma as how to determine if one solution is better
than another. What should be done if a solution is better for one criterion buc worse for another? But here,

I
the trouble comes more from the definition of a "better" solmion rather than from how to implement a GA
tO resolve it. If sometimes a fitness function obtained by a simple combination of the different criteria can give
good result, it supposes iliar crirerions can be combined in a consistent way. But, for more advanced problems,
it may be useful to consider something like Pacem optimally or orher ideas from multicriteria optimization
theory.

I 15.6.4 Populations

A population is a collection of individuals. A population consists of a number of individuals being reseed,

the phenotype parameters defining the individuals and some informacion abour the search space. The two
imporcam aspects of population used in GAs are:

1. The inirial population generation.

2. The population size.

For each and every problem, the population size will depend on the complexity of che problem. It is often
a random initialization of population. In the case of a binary coded chromosome this means chat each bit
is initialized to a random 0 or I. However, there may be instances where che initialization of population is
carried out with some known good solutions.
Ideally, the first population should have a gene pool as large as possible in order to be able co explore
the whole search space. All the different possible alleles of each should be present in che population. To
achieve this, the initial population is, in most of the cases, chosen randomly. Nevertheless, sometimes a kind
of heuristic can be used ro seed the initial population. Thus, che mean fitness of the population is already
high and it may. help the GA to find good solutions faster. Bur for doing chis one should be sure that che gene
pool is scilllarge enough. Otherwise, if the population badly lacks diversity, the algorithm will just explore a
small part of the search space and never find global optimal solutions.
The size of the population raises few problems roo. The larger the population is, the easier it is m explore
the search space. However, it has been established that the time required by a GAm converge is O(n log n)
function evaluations where n is the population size. We say that the population has converged when all the
individuals are very much alike and further improvement may only be possible by mutation. Goldberg has
also shown that GA efficiency to reach global optimum instead of local ones is largely determined by the size of
the populacion. To sum up, a large population is quite useful. However, it requires much more computational
cost memory and rime. Practically, a population size of around 100 individuals is quite frequent, but anyway
this size can be changed according ro the rime and the memory disposed on the machine compared co the

i
quality of the result to be reached.
15.7 Simple GA 401

Population Chromosome 1 1 1 1 0 0 0 1 0

Chromosome 2 0 1 1 1 1 0 1 1

Chromosome a 10101010

Chromosome 4 11001100

Figure 15-12 Populacion.

Population being combination of various chromosomes is represented as in Figure 15-12. Thus the
population in Figure 15-12 consists of four chromosomes.

115.7 Simple GA
GA handles a population of possible solutions. Each solution is represented through a chromosome, which
is just an abstract representation. Coding all the possible solutions into a chromosome is the first part, bur
certainly not the most straightforward one of a GA. A set of reproduction operators has tO be determined,
coo. Reproduction operators are applied directly on the chromosomes, and are used to perform mutations
and recombinations over solutions of the problem. Appropriate representation and reproduction operators
are che determining factors, as the behavior of the GA is extremely dependent on it. Frequendy, it can be
exrremely difficult to find a representation that respects the structure of the search space and reproduction
operators that are coherent and relevant according to the properties of the problems.
The simple form of GA is given by che following.

I. Scan with a randomly generated population.

2. Calculate the fitness of each chromosome in the population. u
(
3. Repeat the following seeps umilu offsprings have been created:
i
Select a pair of parent chromosomes from the current population.
With probability pc crossover the pair at a randomly chosen poinr co form two offsprings. il
Mutate ~le two offsprings at each locus wirh probability p,. i
4. Replace the current population with the new population.
5. Go to seep 2.

Now we discuss each iteration of this process.

Generation: Seleaion: is supposed to be able to compare each individual in the population. Selection is
done by using a firness function. Each chromosome has an associated value corresponding to the firness of the
solution it represents. The fitness should correspond to an evaluation ofhow good the candidate solution is.
The optimal solution is the one which maximizes the fitness function. GAs deal with the problems that
maximize the fitness function. Bur, if che problem consists of minimizing a cost function, the adaptation is
quite easy. Either rhe cost function can be transformed inro a fitness function, for example by inverting it; or
the selection can be adapted in such way char they consider individuals with low evaluation functions as better.
Once the reproduction and rhe fitness function have been properly defined, a GA is evolved according ro the
same basic structure. It starts by generating an initial population of chromosomes. This first population must

II'!''!
t
 402 Genetic Algorithm

offer a wide diversicy of genetic materials. The gene pool should be as large as possible so that any solution of
the search space can be engendered. Generally, the initial popuJacion is generated randomly. Then, the GA
loops over an iteration process ro make the population evolve. Each iteration consists of the following steps:
1. Stkction: The first step consists in selecting individuals for reproduction. This selection is done randomly
with a probability depending on the relative fimess of the individuals so that besr ones are often chosen
for rt:production rather than the poor ones.
2. Reproduction: In the second step, offspring are bred by selected individuals. For generating new
chromosomes, the algorithm can use both recombination and mutation.
3. EvaLuation: Then the fitness of the new chromosomes is evaluated.
4. &placement: During the last step, individuals from the old population are killed and replaced by the new
ones.
The algorithm is stopped when the population converges toward the optimal solmion.
BEGIN/* genetic algorithm"/
Generate initial popuJation;
Compme fitness of each individual;
WHILE NOT finished DO LOOP
BEGIN
Select individuals from old generations
For mating;
Create offspring by applying
recombination and/or mutation
w the selected individuals;
' Compute fitness of the new individuals;
Kill old individuals w make room for
new chromosomes and insert
offspring in the new generalization;
IF Population has converged
THEN finishes: =TRUE;
END
END
Genetic algorithms are not too hard to program or understand because they are biological based. An
example of a flowchart of a GA is shown in Figure 15-13.

115.8 General Genetic Algorithm

The general GA is as follows:

Step 1: Create a random initial state: An initial population is created from a random selection of solutions J

(which are analogous to chromosomes). This is unlike the situation for symbolic AI systems,
where the initial State in a problem is already given.
Step 2: Evaluate fitness: A value for fitness is assigned to each solution (chromosome) depending on how
close it actually is w solving the problem (rhus arriving to the answer of the desired problem).
(These "solutions" are not to be confused with "answers" to the problem; think of them as possible
charaCteristics thar the system would employ in order to reach the answer.)
 15.8 General Genetic Algorithm 403

Evolution

No
Terminate
?

! Figure 1513 Flowchart for generic algorithm.

Step 3: Reproduce (and children mutate): Those chromosomes with a higher fitness value are more likely
to reproduce offspring (which can mutate after reproduction). The offspring is a product of the
father and mother, whose composition consists of a combination of genes from the rwo (this
process is known as "crossingover").
Step 4: Nat generation: If the new generation coma.ins a solution that produces an output that is dose
enough or equal to the desired an~er then the problem has been solved. If this is not the case,
then the new generation will go through the same process as their parents did. This will continue
L until a solution is reached. 1
404 Genetic Algorithm

Table 152 Fitness value for corresponding

chromosomes (Example 15.1)
Chromosome Fitness
k 00000110 2
B, 11JOI110 6
c, 00100000 I
o, 00110100 3

Table 153 Fitness value for corresponding

chromosomes
Cluomosome Fitness
k 011011JO 5
B, 00100000 I
c 10110000 3
o, 01101110 5

c Fitness-proportionate selection
8
(Roulette wheel sampling)

Figure 1514 Roulette wheel sampling for firness-proponionare selecrion.

Example 15.1: Consider 8-bir chromosomes with rhe following properties:

1. Fimess function j(x) = number of 1 bits in chromosome;
2. population size N = 4;
3. crossover probability p, = 0.7;
4. mutation probabiliry Pm = 0.001;
Average fitness of population= 12/4 = 3.0.

1. If B and C are selected, crossover is not performed.

2. If B is mutated, then

B, 11101110-> B',OIIOIIIO

3. IfB and Dare selected, crossover is performed.

o, 00110100 f, 01101110
I
4. lfE is mutated, ilicn
B, 11101110 E, 10110100....,
I
E, 10110100--. E', 10110000

-I

~
 15.9 Operators in Genetic Algorithm 405

Best-fit suing from previous population is lost, but the average fitness of population is as given below:

Average fitness of population 14/4 = 3.5

Tables 15-2 and 15-3 show ilie fitness value for the corresponding chromosomes and Figure 15-14 shows
he Roulette wheel selection for ilie fitness proponionate selection.

115.9 Operators in Genetic Algorithm

The basic operators that are to be discussed in this section include: encoding, selection, recombination and
mucation operarors. The operators with their various types are explained with necessary examples.

I 15.9.1 Encoding

Encoding is a process of representing individual genes. The process can be performed using bits, numbers,
trees, arrays, lists or any other objects. The encoding depends mainly on solving the problem. For example,
one can encode direcdy real or integer numbers.

15.9. 1. 1 Binary Encoding

The most common way of encoding is a binary string, which would be represemed as in Figure 15-15.
Each chromosome encodes a binary (bit) suing. Each bit in the suing can represent some characteristics of
the solution. Every bit string therefore is a solution but not necessarily the best solution. Anmher possibility is
that the whole string can represent a number. The way bit strings can code differs from problem to problem.
Binary encoding gives many possible chromosomes with a smaller number ofalleles. On the ocher hand, this
encoding is not narural for many problems and sometimes corrections must be made afrer generic operation
is completed. Binary coded strings with Is and Os are mostly used. The length of the string depends on the
accuracy. In such coding
1. Integers are represented exactly.
2. Finite number of real numbers can be represemed.
3. Number of real numbers represemed increases with string length.

15.9.1.2 Octal Encoding

This encoding uses string made up of octal numbers (0-7) (see Figure 15-16).

Chromosome1 11 1 0 10 0 0 11 0 1 0

Chromosome2 I 0 1 1 1 1 1 1 1 1 1 0 0

Figure 1515 Binary encoding.

Chromosome 1 03467216

Chromosome 2 15723314

Figure 1516 Octal encoding.

~
 406
Genetic Algorithm

Chromosome 1 9CE7

Chromosome 2 3DBA

Figure 1517 Hexadecimal encoding.

Chromosome A 15 3 2 6 4 7 9 8

ChromosomeS 8 56 7 2 314 9

Figure 1518 Permutarion encoding.

15.9.1.3 Hexadecimal Encoding

This encoding uses string made up of hexadecimal numbers (0-9, A-F) (see Figure 15-17).

.- 15.9.1.4 Permutation Encoding (Real Number Coding)

Every chromosome is a string of numbers, represented in a sequence. Sometimes corrections have to be done
after generic operation is complete. In permutation encoding, every chromosome is a suing of integer/real
;;. values, which represents number in a sequence.
Permutation encoding (Figure 15-18) is only useful for ordering problems. Even for this problem, some
types of crossover and mmarion corrections must be made to leave the chromosome consisrenr (i.e., have real
sequence in it).
:~

15.9.1.5 Value Encoding

Every chromosome is a string of values and the values can be anything connected w rhe problem. This
encoding produces best results for some special problems. On the other hand, it is often necessary ro develop
new genetic operator's specific ro the problem. Direct value encoding can be used in problems, where some
complicated values, such as real numbers, are used. Use of binary encoding for this type of problems would
be very difficuk
In value encoding (Figure 15-19), every chromosome is a string of some values. Values can be anything
connected to problem, form numbers, real numbers or characters to some complicated objects. Value encoding
is very. good for some special problems. On the other hand, for this encoding it is often necessary to develop
some"'riew crossover and murarion specific for the problem.

Chromosome A 1.2324 5.3243 0.4556 2.3293 2.4545

Chromosome B ABDJEIFJDHDIERJFDLDFLFEGT

Chromosome C (back), (back), (right), (forward), (left)

Figure 15~19 Value encoding.

15.9 Operators in Genetic Algorithm 407

15.9. 1. 6 Tree Encoding

This encoding is mainly used for evolving program expressions for genetic programming. Every chromosome
is a tree of some objects such as functions and commahds of a programming language.

I 15.9.2 Selection

Selection is the process of choosing two parents from the population for crossing. After deciding on an
encoding, the next step is to decide how to perform selection, i.e., how to choose individuals in the population
that will create offspring for the next generation_ and how many offspring each will create. The purpose of
selection ism emphasize fitter individuals in the-population in hopes that their offspring have higher fitness.
Chromosomes are selected from the initial population to be parents for reproduction. The problem is how
w select these chromosomes. According to Dat\Yin's theory of evolution the best ones survive to create new
offspring. Figure 15-20 shows the basic selection process.
Selection is a method that randomly picks chromosomes out of the population according to their evaluation
function. The higher the fitness function, the better chance that an individual will be selected. The selection
pressure is defined as the degree to which che better individuals are favored. The higher cheselection pressured,
the more the better individuals are favored. This selection pressure drives rhe GA to improve che population
fitness oyer successive generations.
The convergence rate ofGA is largely determined by the magnitude of the selection pressure, with higher
selection pressures resulting in higher convergence rates. GAs should be able to identify optimal or nearly
optimal solutions under a wide range of selection scheme pressure. However, if the selection pressure is too
low, the convergence rate will be slow, and the GA will take unnecessarily longer to find the optimalsolmion.
If rhe selection pressure is coo high, there is an increased change of the GA prematurely converging to an
incorrect (sub-optimal) solution. In addition to providing selection pressure, selection schemes should also
preserve population diversity, as this helps to avoid premature convergence.
Typically we can distinguish rwo types of selection scheme, proportionate-based seleccion and ordinal-
based selection. Proportionate-based selection picks out individuals based upon their fitness values relative to
rhc fitness of the other individuals in the population. Ordinal-based selection schemes select individuals nor
upon their raw fitness, bur upon their rank within the population. This requires that the selection pressure
is independent of the fitness distribution of the population, and is solely based upon the relative ordering
(ranking) of the population.

I -1
,---- I
The two best
Individuals

Mating
Pool

Figure 15~20 Selection.

408 Genetic Algorithm

It is also possible to use a scaling function to redistribute the fitness range of the population ip order to
adapt ilie selection pr~ure. For example, if all the solutions have their fitnesses in the range [999, 1000],
rhe probability of selecting a better individual than any other using a proponionare-based method will nor be
important. If the fitness every individual is bringing ro the range [0, 1] equitable, rhe probabilicy of selecting
good individual instead of bad one will be important.
Selection has to be balanced with variation from crossover and mutation. Too strong selection means
sub-optimal highly fit individuals will take over the population, reducing the diversity needed for change and
progress; too weak selection will result in roo slow evolution. The various selection methods are discussed in
the following subsections.

15.9.2. 1 RouleNe Wheel Selection

Roulette selection is one of the rraditional GAselection techniques. The commonly used reproduction operator
is the proponionate reproductive operator where a string is selected from the mating Pool with a probability
proportional to the fitness. The principle ofRoulerte selection is a linear search through a Roulette wheel with
the sims in the wheel weighted in proportion to the individual's fitness values. A target value is set, which is
a random proportion of the sum of the fit nesses i~ the population. The population is stepped through until
the target value is reached. This is only a moderately strong selection technique, since fir individuals are not
guarameed to be selected for, bur somewhat have a greater chance. A fit individual will contribute more to
rhe target value, bur if it does not exceed it, the next chromosome in line has a chance, and it may be weak.
It is essential that the population not be sorted by fitness, since this would dramatically bias the selection.
The Roulette process can also be explained as follows: The expected value of an individual is individual's
fitness divided by the actual fitness of the population. Each individual is assigned a slice of rhe Roulerre wheel,
the size of the slice being proponional to the individual's fimess. The wheel is spun N times, where N is the
number of individuals in the population. On each spin, the individual under ilie wheel's marker is selected

I
to be in rhe pool of parents for the next generation. This method is implemented as follows:
1. Sum the total expected value of the individuals in the population. Let it beT.
2. Repeat N times:
~
i. Choose a random integer "r" between 0 and T.
ii. Loop through the individuals in the population, summing the expected values, until the sum is greater I
rhan or equal to "r." The individual whose expected value puts rhe sum over this limit is rhe one
selected. !
Roulerre wheel selection is easier to implement bur is noisy. The rate of evolution depends on the variance t\
of fitness's in the population.

15.9.2.2 Random Selection

This technique randomly selects a parent from the population. In terms of disruption ofgeneric codes, random
selection is a linle more disruptive, on average, than Roulette wheel selection.

15.9.2.3 Rank Selection

The Roulette wheel wiU have a problem when the fitness values differ very much. If the best chromosome
fitness is 90%, its circumference occupies 90% of Roulette wheel, and then other chromosomes have too few
chances to be selected. Rank Selection ranks the population and every chromosome receives fitness from the
ranking. The worst has fitness 1 and the best has fitness N. It results in slow con~ergence bur prevenrs roo

!
l
 15.9 Operators in Genetic Algorithm 409

quick convergence. It also keeps up selection pressure when the fitness variance is loW. It preserves diversity
and hence leads to a successful search. In effect, potential parems are selected and a tournament is held
to decide which of the individuals will be the paienr. :There are many ways this can be achieved and two
suggestions are:
1. Select a pair of individuals at random. Gener:ite a random number R between 0 and 1. If R <ruse the
first individual as a parent. If the R 2: r then use the second individual as the parent. This is repeated to
sdect the second parent. The value of r is a parameter to this method.
2. Select two individuals at random. The individual with the highest evaluation becomes the parent. Repeat
to find a second parent.

15.9.2.4 Tournament Selection

An ideal selection strategy should be such that it is able to adj!Jst its selective pressure and population diversity
so as ro fine-rune GA search performance. Unlike, the Roulette wheel selection, the tournament selection
strategy provides selective pressure by holding a tournament competition among Nu individuals.
The best individual from rhe tournament is the one with the highest fitness, who is the winner of Nu.
Tournament competitions and the winner are then inserted into the mating pool. The tournament competition
is repeated until the mating pool for generating new offspring is filled. The mating pool comprising the
tournament winner has higher average population fitness. The fitness difference provides the selection pressure,
which drives GA to improve the fitness of the succeeding genes. This method is more efficient and leads to
an optimal solution.

15.9.2.5 Boltzmann Selection

SA is a method of function minimization or maximization. This method simulates the process of slow

I cooling of molten metal to achieve the minimum function value in a minimization problem. Controlling a
temperature-like parameter imroduced with the concept of Boltzmann probability distribution simulates rhe
cooling phenomenon.
In Boltzmann selection, a continuously varying temperature controls the rate of selection according to
a preset schedule. The rem perature scans our high, which means that the selection pressure is low. The
temperature is gradually lowered, which gradually incre:J.Ses the selection pressure, thereby allowing the GA
to narrow in more closely to the besr parr of the search space while maintaining the appropriate degree of
diversity.
A logarirhmically decreasing temperature is found useful for convergence without getting stuck co a local
minima state. However, it takes rime to cool down the system to the equilibrium scare.
let fm~ be the fitness of rhe currently available best suing. If rhe next string has fitness J(X:) such that
/(X;)> /max then the new string is selected. Otherwise it is selected with Bole/Mann probability

P= exp[-lfm.,- j(X,))IT] (15.10)

where T = T0 (1-CI:' )It and k = (I + 100 *giG); g is the current generation number; G the maximum value
ofg. The value ofCI:' can be chosen from the range [0, 1] and that of T0 from the range [5, 100]. The final stare
is reached when computation approaches zero value ofT, i.e., the global solution is achieved at this point.
The probability that the best string is selected and inuoduced inro the mating pool is very high. However,
Elitism can be used to eliminate the chance of any undesired loss of information during rhe mutation stage.
Moreover, rhe execution rime is less.

l
 410 Genetic Algorithm

Pointer 1. Pointer 2 Pointer 3 Pointer 4 Pointer 5 Pointer 6

lndwidual 1 1 2 t ,1 4 t 5 t 7 J 9 10

0.0 1 0.18 0.34 0.49 0.62 0.73 0.82 0.95 1.0

Random number

Figure 1521 Stochastic universal sampling.

Elitism
The first best chromosome or the few best chromosomes are copied to the new population. The rest is done
in a classical way. Such individuals can be lost if they are not selected to reproduce or if crossover or mutation
destroys them. This significantly improves-the GA's performance.

15.9.2.6 Stochastic Universal Sampling

Stochastic uiiiversal sampling provides zero bias and minimum spread. The individuals are mapped to con~
riguous segmems of a line, such that each individual's segment is equal in size ro its fitness exactly as in
Roulen:e wheel selection. Here equally spaced pointers are placed over the line, as many as there are individ~
uals to be selected. Consider NPointer the number of individuals to be selected, then the distance between
"' the pointers are 1/NPointer and the position of the first pointer is given by a randomly generated number in
the range [0, liNPointer]. For 6 individuals to be selected, rhe distance between che pointers is 1/6 = 0.167.
Figure 15-21 shows the selection for the above example.
Sample of 1 random number in the range [0, 0.167]: 0.1.
After selection the mating population consists of the individuals,

1,2,3,4,6,8

Stochastic universal sampling ensures selection of offspring that is closer to what is deserved as compared
to Roulette wheel selection.

I 15.9.3 Crossover (Recombination)

Crossover is the process of taking two parent solutions and producing from them a child. After the selecrion
(reproduction) process, the population is enriched with bener individuals. Reproduction makes clones of
good strings but does not create new ones. Crossover operawr is applied to the mating pool with the hope
that it creates a better offspring.
Crossover is a recombination operator that proceeds in rhree steps:
1. The reproduccion operator selects at random a pair of rwo individual strings for the mating.
2. A cross site is selected ar random along the string length.
3. Finally, the position values are swapped between the rwo strings following the cross site.
That is the simplest way how to do that is to choose randomly some crossover poinr and copy everything
before this point &om the first parent and then copy everything after the crossover point from the other
parent. The various crossover techniques are discussed in the following subsections.
 15.9 Operalors in Genetic Algorithm 411

Parent1 10110:010

Parent2 10101:111

Child 1
t
10110:111

Child2 '
10101:010

Figure 1522 Single-point crossover.

15.9.3.1 Single-Point Crossover

The traditional genetic algorithm uses single-point crossover, where the rwo mating chromosomes are cur
once at corresponding points and che sections after the cuts exchanged. Here, a cross site or crossover point
is selected randomly along the length of the mated strings and bits next to the cross sites are exchanged.
If appropriate site is chosen, bener children can be obtained by combining good parents, else it severely
hampers string quality.
Figure 15~22 illustrates single~point crossover and it can be observed that the bits next to the crossover
point are exchanged to produce children. The crossover point can be chosen randomly.

15.9.3.2 Two~Point Crossover

Apart from single~poim crossover, many different crossover algorithms have been devised, often involving
more than one cut point. It should be noted that adding further crossover points reduces the performance of
the GA. The problem with adding additional crossover points is that building blocks are more likely to be
disrupted. However, an advantage of having more crossover points is that the problem space may be searched
more thoroughly.
In two-poimcrossover, two qossover points are chosen and the contents between these points are exchanged
berween two mated parents.
' In Figure 15-23 the dotted lines indicate the crossover points. Thus the coments benveen these points are
exchanged berween the parents to produce new children for mating in the next generation.
t

l Parent 1

Parent2
11:0 11:01 0

o1:1o1:1oo
'

I ~ ''.'
I Child 1 11:101:010
. .
I Child 2 01:011:100
' ' -- --

Figure 1523 Two-poim crossover.

412 Genetic Algorithm

Originally, GAs were using onepoim crossover which cuts rwo chromosOmes in one point and splices the
rwo halves to create new ones. But with chis one-point crossover, the head and the rail of one chromosome
cannot be passed together to the offspring. If both the head and the rail of a chromosome contain good
genetic information, none of the offspring obtained directly with one-point crossover will share the twO good
features. Using a rwo-poim crossover one can avoid this drawback, and so it is generally considered better than
one-point crossover. In fact, this problem can be generalized to each gene position in a chromosome. Genes
that are close on a chromosome have more chance ro be passed together to the offipring obtained ilirough
N-poims crossover. It leads to an unwanted correlation between genes next to each other. Consequendy, the
efficiency of an N-poim crossover will depend on ilie position of the genes wich..in the chromosome. In a
genetic representation, genes that encode dependent characteristics of the solution should be close together.
To avoid all the problem of genes locus, a good thing is to use a uniform crossover as :recombination operator.

15.9.3.3 Multipoint Crossover (NPoint Crossover)

There are rwo ways in this crossover. One is even number of cross sires and the other odd number of cross sites.
In the case of even number of cross sires, the cross sites are selected randomly around a circle and informacion
is exchanged. In the case of odd number of cross sites, a different cross point is always assumed at the string
beginning.

15.9.3.4 Uniform Crossover

Uniform crossover is quite different from the N~point crossover. Each gene in che offspring is created by
copying the corresponding gene from one or the other parent chosen according to a random generated binary
crossover mask of the same length as the chromosomes. Where there is a 1 in the crossover mask, the gene is
copied from the first parent, and where there is a 0 in the mask the gene is copied from the second parent. A
new crossover mask is randomly generated for each pair of parents. Offspring, therefore, contain a mixture
of genes from each parent. The number of effective crossing poim is not fiXed, but will average L/2 (where L
is the chromosome length).
In Figure 15-24, new children are produced using uniform crossover approach. It can be noticed that
while producing child 1, when there is a I in the mask, the gene is copied from parent 1 else it is copied from
parent 2. On producing child 2, when there is a 1 in the mask, the gene is copied from parent 2, and when
there is a 0 in the mask, che gene is copied from the parent 1.

15.9.3.5 ThreeParent Crossover

In this crossover technique, three parents are randomly chosen. Each bit of the first parent is compared with
the bit of the second parent. lfboth are the same, the bit is taken for the offspring, otherwise the bit from the
third parent is taken for the offspring. This concept is illustrated in Figure 15-25.

Parent1 10110011

Parent2 00011010

Mask 11010110

Child 1 1 0 0 1 1 0 1 0

Child2 00110011

Figure 1524 Uniform crossover.

15.9 Operators in Genetic Algorithm 413

Parenl1 11010001

Parent 2 .01101001

Parent3 01_.101100

Child 01101001

Figure 1525 Three~parentcrossover.

15.9.3. 6 Crossover with Reduced Surrogate

The reduced surrogate operator constraints crossover to always produce new individuals wherever possible.
This is implemented by restricting the location of crossover points such that crossover points Oniy occur where
gene values differ.

15.9.3. 7 Shuffle Crossover

Shuffle crossover is related ro uniform crossover. A single crossover position (as in single~point crossover) is
sdecred. But before the variables are exchanged, they are randomly shuffled in both parents. After recomb ina~
tion, the variables in che offspring are unshuffled. This removes positional bias as the variables are randomly
reassigned each rime crossover is performed.

15.9.3.8 Precedence Preservative Crossover

Precedence preservarive crossover (PPX) was independenrly developed for vehicle routing problems by Blanton
and Wainwright (1993) and for scheduling problems by Bierwirth et al. (1996). The operator passes on
precedence relations of operations given in rwo parental permutations to one offspring at the same race, while
no new precedence relations are introduced. PPX is illustrated below for a. problem consisting of six operations
A-F. The operator works as follows:
l. A vector oflength Sigma, sub i == 1 to mi, representing rhe number of operations involved in the problem,
is randomly filled with elements of the set {1, 2).
i

2. This vector defines the order in which the operations are successively drawn from parent I and parent 2.
3. We can also consider the parent and offspring permutations as lists, for which the operations "append"
and "delete'' are defined.
I
4. First we sra.n by initializing an empty offspring.
5. The lefrmost operation in one of the (WO parents is selected in accordance with the order of parents given
in the vector.
6. After an operation is selected, it is deleted in both parents.
7 Finally the selected operation is appended to the offspring.
8. Step 7 is repeated until both parents are empty and the offspring comains all operations involved.
Note that PPX does nor work in a uniform~crossover manner due to rhe "deletion-append" scheme used.
Example is shown in Figure 15-26.

15.9.3.9 Ordered Crossover

Ordered two-point crossover is used when the problem is order based, for example in U~shaped assembly
line balancing, er~ Given two parent chromosomes, two random crossover points are selected partitioning
 414 Genetic Algorithm

Parent permutation 1 ABCDEF

Parent permutation 2 CABFDE

Select parent no. (1/2) 2 2 2

Offspring permU1ation ACBDFE

Figure 1526 Precedence preservative crossover (PPX).

Parent1:4 211 3I65Child1:4 2131165

Parent2:2 311 4j56Child2:2 3141156

Figure 1527 Ordered crossover.

them imo a left, middle and right portions. The ordered two~point crossover behaves in the following way:
child 1 inherits its lefc and right section from parent l, and its middle section is determined by the genes in
the middle section of parent 1 in the order in which the values appear in parent 2. A similar process is applied
to determine child 2. This is shown in Figure 15~27.

~ 15.9.3. 10 Partially Matched Crossover

PaHially marched crossover (PMX) can be applied usefully in the TSP. Indeed, TSP chromosomes are simply
sequences of integers, where each integer represents a different city and the order represents the time at which a
city is visited. Under this representation, known as permutation encoding, we are only interested in labels and
not alleles. Ir may be viewed as a crossover of permutations that guarantees that all positions arc found exacdy
once in each offspring, i.e., borh offspring receive a full complement of genes, followed by the corresponding
filling in of alleles from their parents. PMX proceeds as follows:

I. The two chromosomes are aligned.

2. Two crossing sires are selected uniformly at random along the strings, defining a marching section.
3. The matching section is used to effect a cross through position-by-position exchange operation.
4. Alleles are moved to their new positions in the offspring.
The following illustrates how PMX works.

Name984 .567.13210 Allele1 01.001.1100

Name871.2310.9546 Allele111.011.1101

Figure 1528 Given suings.

Consider the rn'O strings shown in Figure 15-28, where rhe dots mark the selected cross poinrs. The marching
section defines the position-wise exchanges that must rake place in both parenrs ro produce the offspring.
The exchanges are read from the marching section of one chromosome ro that of the other. In the example
illustrate in Figure 15-28, We numbers that exchange places are 5 and 2, 6 and 3, and 7 and 10. The resulting
offspring are as shown in Figure 15-29. PMX is dealt in derail in the nexr chapter.
 15.9 Operators in Genetic Algorithm 415

Name 984.2310.1657 Allele 101.010.1001

Name 8101.567.9243 Allele 111.111.1001

Figure 1529 Partial!)' matched crossover.

15.9.3.11 Crossover Probability

The basic parameter in crossover technique is the crossover probability (Pt). Crossover probability is a param-
eter to describe how often crossover will be performed. If there is no crossover, offspring are exact copies
of parents. If there is crossover, offspring are" made from pam of both parents' chromosome. If crossover
probability is 1OOo/o, then all offspring are made by crossover. If it is Oo/o, whole new- generation is made from
exact copies of chromosomes from old population (but this does not mean that the new gene radon is the
same!). Crossover is made in hope that new chromosomes will contain good pam of old chromosomes and
therefore the new chromosomes will be better. However, it is good to leave some part of old population survive
to next generation.

I 15.9.4 Mutation

After crossover, the strings are subjected ro mutation. Mutation prevents the algorithm to be trapped in a local
minimum. Mutation plays the role of recovering the lost genetic materials as well as for randomly distributing
generic information. It is an insurance policy against the irreversible loss of genetic material. Mutation has been
traditionally considered as a simple search operator. If crossover is supposed to exploit rhe current solution
I to find better ones, mutation is supposed ro help for the exploration of the whole search space. Mutation is
vie1ed as a background operator to maintain genetic diversity in the population. It introduces new generic

! structures in the population by randomly modifying some of irs building blocks. Mutation helps escape from
local minima's trap and maintains diversity in the population. It also keeps the gene pool well stocked, rhus

l
ensuring ergodicity. A search space is said to be ergodic if there is a non-zero probability of generating any
solution from any population state.
There are manydifferem forms of mutation for the different kinds of representation. For binary representa-

! tion, a simple mutation can consist in inverting the value of each gene with a small probability. The probability
is usually taken about 1/ L, where L is the length of the chromosome. It is also possible to implement kind
I of hill climbing mutation operators that do mutation only if it improves the quality of the solution. Such an
operawr can accelerate the search; however, care should be taken, because it might also reduce the diversity
\ in the population and make rhe algorithm converge toward some local optima. Mutation of a bit involves
flipping a bit, changing 0 to I and vice-versa.

15.9.4. 1 Flipping
Flipping of a bit involves changing 0 to 1 and 1 ro 0 based on a mutation chromosome generated. Figure 15-30
explains mutation~flipping concept. A parent is considered and a mutation chromosome is randomly gener-
ated. For a 1 in mutation chromosome, the corresponding bir in parent chromosome is flipped (0 to 1 and
1 to 0) and child chromosome is produced. In the case illustrated in Figure 15-30, 1 occurs at 3 places of
mutation chromosome, the corresponding bits in parent chromosome are flipped and the child is generated.

15.9.4.2 Interchanging
Two random positions of the srring are chosen and the bits corresponding to those positions are interchanged
(Figure 15.31).
416 Genetic Algorithm

Parent 1 0 1 1 0 1 0 1

Mutation 1 0 0 0 1 0 0 1
chromosome

Child 0 0 1 1 1 1 0 0

Figure 1530 Mmacion flipping.

10110101
1 1 1 0 0 0 1

Figure 1531 Imerchanging.

Parent 1 0 1 1 o: 1 0 1

Chitd 1 a 1 t a: 1 1 o

Figure 1532 Reversing.

15.9.4.3 Reversing
A random position is chosen and rhe bits next tO rhat position are reversed and child chromosome is produced
(Figure 15-32).

15.9.4.4 Mutation Probability

An imponanr parameter in rhe mmation technique is the mutation probabilicy(P,).lrdecides how often parts
of chromosome will be mutated. If there is no mutation, offspring are generated immediately after crossover
I
(or directly copied) withom any change. If mumion is performed, one or more parts of a chromosome are
f
changed. If mutation probability is 100%, whole chromosome is changed; if ir is 0%, nothing is changed.
Mmarion generally prevents rhe GA from falling into local exrremes. Mmation should not occur very often,
[
because then GA will in fact change to ralidom search.

115.10 Stopping Condition for Genetic Algorithm Flow

In short, the various stopping condition are listed as follows:

1. Maxim 11m generations:. The GA stops when rhe specified number of generations has evolved.
2. Elapsed time: The generic process will end when a specified rime has elapsed.
Note: If the maximum number of generation has been reached before the specified rime has elapsed, the
process will end.
3. No change in fitness: The genetic process will end if rhere is no change to rhe population's best fimess for
a specified number of generations.
Note: If the maximum n~.mber of generation has been reached before the specified number of generation
I
with oo changes has been reached, the process will end.

t
 15.11 Constraints in Genetic Algorithm 417

4. Stailgenerarions: The algorithm stops if there is no improvement in the objective function for a sequence
of consecutive generations of length "Stall generations."
5. Stall time limit. The algorithm srops if there is no improvemenr in the objective function during an
imerval oftime in seconds equal to "Stall time limit."
The termination or convergence criterion finally brings the search to a halt. The following are rhe few
methods of termination techniques.

115.10.1 Best Individual

A best individual convergence criterion stops the search once rhe minimum fitness in the population drops
below the convergence value. This brings rhe search w a faster conclusion, guaranteeing at least one good
solmion.

I 15.1 0.2 Worst Individual

Worst individual terminates the search when the least fir individuals in the population have fitness less rhan
me convergence criteria. This guarantees rhe entire population w be of minimum srandard, although the
best individual may not be significantly better than the worst. In this case, a stringent convergence value may
never be met, in which case the search will terminate after rhe maximum has been exceeded.

I 15.1 0.3 Sum of Fitness

In this termination scheme, the search is considered to have satisfaction converged when rhe sum of the
fitness in rhe emire population is less rhan or equal to the convergence value in the population record. This
guarantees rhat virtually all individuals in the population will be within a particular fitness range, although
it is bener to pair rhis convergence criteria with weakest gene replacement, mherwise a few unfir individuals
in rhe population will blow our the fitness sum. The population size has to be considered while setting rhe
convergence value.

115.1 0.4 Median Fitness

Here ar least half of the individuals will be better than or equal to the convergence value, which should give
a good range of solutions to choose from.

115.11 Constraints in Genetic Algorithm

If the GA considered consists of only objective function and no information about rhe specifications of
variable, then it is calli:d unconstrained optimiztJtion problem. Consider, an unconstrained optimization problem
of me form
Minimire j(x) = 2- (15.11)

and there is no information about "x" range. GA minimizes this function using irs operators in random
specifications.
In rhe case of constrained optimization problems, the information is provided for the variables under
consideration. Constraints are classified as:
1. Equality relations.

t
2. Inequaliry relations.
 418 Genetic Algori~hm

A GA genemes a st=quence of parameters w be rested using the system under consideration, objective
function (to be maxiniized or minimized) and rhe constraints. On running. the system, the objective function
is evaluated and conmaim_s are cht=cked to see if there are any violations. If there are no violations, the parameter
set is assigned the fitness value corresponding to the objective function evaluation. When the constraints are
violated, the solution is infeasible and rhus has no firness. Many practical problems are constrained and it
is very difficult tO find a feasible point that is best. As a result, one should ger some information out of
infeasible solutions, irrespective of their fitness ranking in relation to rhe degree of constraint violation. This
is performed in penalry method.
Penalcy method is one where a consuained optimization problem is transformed to an unconstrained
optimization problem by associating a penalty or cost wirh all constraint violations. This penalty is included
in the objective function evaluation.
Consider the original constrained problem in maximization form:

Maximize J(x)

Subjecrro g;(x) ~ 0, i = 1, 2, 3, ... , n (15.12)

. ~- where xis a k-vector. Transforming this to unconmained form:

"
Maximizef(x) + PL<I>[g;(x)] (15.13)
.. ;,.... i=l

: ~3 where is the penalty function and P is the penalty coefficient. There exist several alternatives for this
- penalcy funaion. The penalty function can be squared for all violated constraints. In cerrain situations,
the unconstrained solution converges ro rhe constrained solution as the penalry coefficient p rends to
infiniry.

115.12 Problem Solving Using Genetic Algorithm

I 15.12.1 Maximizing a Function

Consider the problem of maximizing the function,

j(x)=f (15.14)

where xis permitted ro vary berween 0 and 31. The sreps involved in solving this problem are as follows:

I Step I: For using GA approach, one must first code the decision variable "x" into a finite length string. I
Using a five bit (binary integer) unsigned integer, numbers berween 0(00000) and 31(11111) can
be obtained.
The objective function here isf(x) =,?which is robe maximized. A single generation ofa GA
is performed here with encoding, selection, crossover and mutation. To start with, select initial
population ar random. Here initial population of size 4 is chosen, but any number of populations
can be selected based on the requirement and application. Table 15-4 shows an initial population
randomly selected.
15.12 Problem Solving Using GeneticAigori!lvn 419

Table 154 Selection

Suing no. Initial xvalue Fitness Prob; Percentage Expected AcruaJ
population j(x)=~ prob~ count ooum
(randomly ability
selected) (%)
I 0 I I 00 12 144 0.1247 11.47 0.4987 I
2 II 00 I 25 625 0.5411 54.11 2.1645 2
3 00 I 0 I 5 25 0.0216 2.16 0.0866 0
4 I 00 I I 19 -361 0.3126 31.26 1.2502
Sum 1155 1.0000 100 4.0000 4
Average 288.75 0.2500 25 1.0000 I
Maximum 625 0.5411 54.11 2.1645 2

Step 2: Obtain the decoded x values for the initial population generated. Consider ming I,

01100 = 0 <2 + I
4
* 23 + I * 22 + 0 * 2 1 + 0 * 2
=0+~+4+0+0
= 12
Thus for all the four strings the decoded values are obtained .
Step 3: Calculate the fimess or objective function. This is obtained by simply squaring the "x'" value,
since the given function is /(x) = ...? . When x = 12, the fitness value is

j(x) = .! = (12) 2 = 144

For x = 25, j(x) = x2 = (25) 2 = 625

and so on, unril rhe em ire population i.~ computed.

Step 4: Com pure the probability of selection,

Probi = ftx), (15.15)

"
/:_j(x);
1=1

where 11 is rhe number of populations; j(x) is the fitness value corresponding to a particular
individual in the population;

:E J(x) is the summation of all the fitness value of rhe enrire population.

Considering string l,

Fitnessj(x) = 144
~/(x) = 1155

The probability that string 1 occurs is given by

P1 = 14411155 = 0.1247
420

The percentage probability is obtained as

0.1247. 100 = 12.47%
Genetic Algorithm

l
The same operation is done for all the strings. It should be noted that summation of probability
select is l.
Step 5: The next step is to calCulate rhe expected coum, which is calculated as
f(x);
Expected count (1).16)
[Avgf(x)];

where

[ ]
'f:J(x);
(Avgf(x)); = "
;:,1 n

For string 1,
Expected count= Fimess/Average = 144/288.75 = 0.4987

We rhen com pure rhe expecred count for the entire population. The expected count gives an idea
of which population can be selected for further processing in the mating pool.
Step 6: Now the actual count is to be obtained tO select the individuals who would parricipate in
rhe crossover cycle using Rouleue wheel selection. The Roulette wheel is formed as shown
Figure IS-33.
The entire Raul we wheel covers 1OOo/o and rhe probabilities of selection as calculated in step 4
for the entire populations are used as indicators tO fit into the Roulette wheel. Now the wheel
may be spun and rhe number of occurrences of population is nored to get actual count.
1. String I occupies 12.47%, so there is a chance for it to occur ar least once. Hence irs actual
count may be I.
2. With string 2 occupying 54.11 o/o of the Roulette wheel, it has a fair chance of being selected
cwice. Thus irs acmal count can be considered as 2.
3. On the other hand, string 3 has the least probabilicy percentage of2.16o/o, so their occurrence
for next cycle is very poor. As a result, ir actual count is 0.

"12.47%

Figure 1533 Selection using Roulette wheel.

l
~
 15.12 Problem Solving Using Genetic Algorithm 421

Table 155 Crossover

String no. Mating Pool Crossover Offspring x value Fitness value
point after fix)=,?-
crossover
I 0 I I $0 4 0 I I0 I 13 169
2 I I 0 iii 4 I I 000 24 576
3 I j00 I 2 II0I I 27 729
4 I~ 0 I I 2 10001 17 21:l9
Sum 1763
Average 440.75
Maximum 729

4. String 4 with 31.26% has at least one chance for occurring while Roulette wheel is spun, rhus
irs actual count is I.

The above values of actual count are tabulated as shown is Table 15-S.
Step 7: Now, write the mating pool based upon rhe actual count as shown in Table 15-S.
The actual count of string no. 1 is I, hence it occurs once in the mating pool. The actual count
of string no. 2 is 2, hence it occurs rwice in the mating pool. Since the actual count of string no.
3 is 0, it does not occur in rhe mating pool. Similarly, the actual count of string no. 4 being I, it
occurs once in rhe mating pool. Based on this, rhe mating pool is formed.
Step 8: Crossover operation is performed w produce new offspring (children). The crossover point is
specified and based on the crossover point, single-point crossover is performed and new offspring
is produced. The parents are

Parent 1 0 I I00
Parent 2 I I00I

The offspring is produced as

Offspring 1 0 I I 0 I
Offspring 2 I I000

In a similar manner. crossover is performed for the nt:xt strings.

Step 9: Afrer crossover operations. new offspring are produced and "x .. value.\ are decoded and ~I mess is
calculated.
Step 10: In this step, mutation operation is performed to produce new offspring. Jfter crossover operation.
As discussed in Section 15.9.4.1 mutarion-Aipping operation is pcrtOrmcd :md new offspring are
produced. Table 15-6 shows the new offspring after mutation. Once the offspring are obtained
L after mutation, they are decoded ro x value and the firness values are computed. I
This completes one generation. The mutation is performed on a bir-bit by basis. The crossover probability
and mutation probability were assumed to be 1.0 and 0.001, respectively. Once selection, crossover and

~
 422 Genetic Algorithm

Table 1~-6 Murarion

String no. Offspring Mutation Offipring xvalue Fitness
afr:er chromosomes after f(x)= ,(-
crossover for flipping mutation
0 I I0 I 10000 I I I0 I 29 841
2 I 10 0 0 00000 1I000 24 576
3 1 10 1 I 00000 I I0I1 27 729
4 10001 00100 I0 I00 20 400
Sum 2546
Average 636.5
Maximum 841

mutation are performed, rhe new popular ion is now ready ro be resred. This is performed by decoding the
new srrings created by the simple GA afrer mutation and calculares rhe firness function values from the x
values rhus decoded. The results for successive cycles of simulation are shown in Tables 15-4 and 15~6.
From the rabies, it can be observed how GAs combine high-performance notions to achieve bercer per-
formance. In rhe rabies, it can be nored how maximal and average performances have improved in rhe new
popularion. The population average fitness has improved from 288.75 to 636.5 in one generation. The max-
imum firness has increased from 625 to 841 during the same period. Ahhough random processes make this
best solution, irs improvement can also be seen successively. The best string of the initial population (1 1 0 0
~ I) receives nvo chances for its existence because of its high, above-average performance. When this combines
at random with the next highest string (I 0 0 1 I) and is crossed at crossover point 2 (as shown in Table 15-5),
one of ilie resulting strings (1 1 0 I I) proves to be a very best solution indeed. Thus after mutation ar random,
a new offspring (l 1 I 0 I) is produced which is an excellenr choice.
This example has shown one genemion of a simple GA.

/
115.13 The Schema Theorem
In this section. we will formulate and prove rhe fundamemal resuh on the behavior of GAs- the so-called
Schema Theorem. Although being completely incomparable with convergence resuhs for convemional opti-
mization methods, it still provides valuable insight imo the intrinsic principles of GAs. Assume a GA with
proportional selection t~nd an arbitrary bur fixed flmess function f Let us make the following notations:
I. The number of individuals which fulfill H ar rime step tare denoted as

rH,r = IBr n HI
2. The expression J (t) refers to the observed average firness at time t:
l m
J (r) = - 'L,J(b;)
m i""l

3. The term J (H, t) stands for the observed average fimess of schema H in time step r:
f- (H,r) = -
1 '\'
L..., f(b;)
'H'' iE!)lbj,,EH]
 15.13 The Schema Theorem 423

Theorem (Schema Theorem- Holland 1975). Assuming we consider a simple GA. the following inequality
holds for eveq schema H:

f(H,t) ( o(H)) O<HI

E[rH.t+l] 2:. rn.r - - - - I - p,.-- (1- PM)
j(t) n- I

Proof. The probability that we select an individual fulfilling His

L, fib,,)
iE[Jlb,,.,,,l
,
"Jib,)
i=l

This probability does nor change throughout rhe execution of rhe selection loop. Moreover, each of rhe m
individuals is selecr~::d independent of the others. Hence. the number of selected individuals. which tlllfill H,
is binomially distributed with sample amount m and the probability. We obtain, 1h~rdOre. that the expecred
number of selected individuals fulfilling His

'L J(b,.,) 'L J(b,,,) L f(b;,,)f,H,,

= rH.t ''olb.f f- (H t)
iE [j1f'j.r~H~ 'H.t iE [.llbj.,~lfJ
m 111
=m , '''"H
m - r
'H.r Lfib;,)lm - H.> fir)
L,J(b,,) L,Jtb;,)
i=l i=l i=l

If rwo individuals ar~ crossed, which bmh fulfill H, the rwo offsprings again fulfill H. The number of strings
fulfilling H can only decrease if one srring. which fu]f,l!s H, is crossed with a string which does not fulfill H.
but, obviously, only if the cross sire is chosen somewhere in between the specifications of H. The probability
that rhe cross sire is chosen within the detining length of His
o(Hl
n-1
Hence the survival probabiliry ps of H, i.e., the probability thar a srring fulfilling H produces an offspring
also fulfilling H. can be estimated as follows (crossover is only done wirh probability pc):
o(Hi
ts:::. 1-pc--
n-J
Selection and crossover are carried our independently, so we may compute the expected number of strings
fulfilling H after crossover simply as

f(H.tl,
- H.<PS _
. >f(H.ti,
~-
(I- 8(HJ)
ft. I
! (r) U) H.,
n-

I Afrercrossover, the number of srrings fulfillingH can only decrease if a suing fulfilling His ahered by mutation
ar a specification of H. The probability that all spfc.ifications of H remain untouched by mutation is obviously

I (I - PM)(){HI

I The arguments in the proof of the Schema Theorem can be applied analogously ro many other crossover and
murarion operations.

J
424

I 15.13.1 The Optimal Allocation of Trials

Genetic Algorithm
1
The Schema Theorem has provided the insight that building blocks receive exponentially increasing trials
in fmure generations. The question remains, however, why this could be a good srraregy. This leads to an
important and well~analyzed problem from statistical decision theory- rhe two-armed bandit problem and
ir.s generalization, che k-armed bandit problem. Although this seems like a derour from our main concern, we
shall soon understand the corinectiori m GAs.
Suppose we have a gambling machine with two slots for coins and two arms. The gamMer ~n deposit rhe
coin either imo rhe left or rhe right slot. Afrer pulling rhe corresponding arm, either a reward is given or the
coin is lost. For mathematical simpliciry, we just work with outcomes, i.e., rhe difference berween rhe reward
(which can be zero) and the value of the coin. Let us assume that the left arm produces an outcome with mean
value /1-1 and a variance af while rhe right arm produces an outcome with mean value J12 and variance af.
Without loss of generality, aldtough the gambler does not know this, assume rhat J11 ~ 112
Now the question arises which arm should be played. Since we do not know beforehand which arm is
associated with the higher outcome, we are faced with an interesting dilemma. Not only must we make a
sequence of decisions about which arm to play, we have to collect, at the same rime, information about which
is the bener arm. This trade-offberween exploration of knowledge and irs exploitation is rhe key issue in this
problem and, as rums out later, in GAs, too.
A simple approach to rhis problem is to separate exploration from exploitation. More specifically, we could
perform a single experiment at the beginning and thereafter make an irreversible decision that depends on the
results of the experiment. Suppose we haveN coins. If we tlrsr allocate an equal number n {where 2n ~ N)
of trials ro both arms, we could allocate the remaining N- 2n uials to the observed bener arm. Assuming
we know all involved parameters, rhe expected loss is given as

L(N. n) = IJ.<i _,,,){(N- n)q(n) + n[l - q(n)l)

where q(n) is the probability that the worst arm is rhe observed best arm after 2n experimental trials. The
underlying idea is obvious: In case that we observe rhar the worse arm is the best, which happens wirh
probability q(n), rhc total number of trials allocHed ro the right arm is N - 11. The loss is, therefore,
(J1 1 -Jl2 )(N- n). In the reverse case where we acrually observe that rhe best arm is the best, which happens
with probability I - q(n), rhe loss is only whar we get less because we played rhe worse arm 11 rimes, i.e.,
(Ill -112 )11. Taking the cenrrallimir theorem inro account, we can approximate q(n) with the rail of a normal
distribution:

e_,.!n.

I
I
q(n) "" Jiir -,

where

..;n i
c- Jl.i -JJ.z
- Jaf+af
II
Now we have m specify a reasonable experiment size n. Obviously, if we choose n = 1, the obtained
information is potentially unreliable. If we choose, however, n = N/2 there are no trials left to make use of the !
information gained through rhe experimental phase. What we see is again the tradeoff between exploitation
with almost no exploration (n = I) and exploration without exploitation {n ~' N/2).lt does not take a Nobel

l
1 15.13 The Schema Theorem

prize winner ro see that the optimal way is somewhere in the middle. Holland has studied this problem in
derail. He came to the conclusion that the optimal strategy is given by the following equation:
425

n
4 "'b 1n( Brr b'lnN'
2
N' )

where

b=--
a,
111 -J12
Making a few transformations, we obtain that

N-u 4 .:::::: JBrrb4lnN21N21J

That is, rhe optimal strategy is m allocate slighdy more than an exponentially increasing number of trials
ro the observed best arm. Although no gambler is able to apply this strategy in practice, because it requires
knowledge of the mean values Jll and JLz, we still have found an imponanr bound of performance a decision
strategy should try to approach.
A GA, although the direct connection is nor yet fully clear, actually comes close to this ideal, giving at
least an exponentially increasing number of trials ro the observed best building blocks. However, one may still
wonder how rhe rwo-armed bandit problem and GAs are related. Let us consider an arbitrary string position.
Then there are rwo schemata of order one which have their only specification in this position. According to
the Schema Theorem, rhe GA implicitly decides between these rwo schemata, where only incomplete data are
available (observed average fitness values). In this sense, a GA solves a lor of rwo-armed problems in parallel.
The Schema Theorem, however, is not restricted to schemata of order one. Looking at competing schemata
(different schemata which are specified in the same positions). we observe that a GA is solving an enormous
number of k-armed bandit problems in parallel. The k-armed bandit problem, although much more com-
plicated, is solved in an analogous way - the observed better ahernatives should receive an exponentially
increasing number of trials. This is exactly what a GA does!

115.13.2 Implicit Parallelism

So far we have discovered two distinct, seemingly conflicting views of generic algorithms:
1. The algorirhmic view that GAs operate on strings;
2. the schema-based interpretation.

I
So, we may ask what a GA really processes, strings or schemata? The answer is surprising: Both. Nowadays,
the common interpretation is chat a GA processes an enormous amounr of schemata implicitly. This is accom-
plished by exploiting rhe currently available, incomplete information about these schemata continuously, while
trying to explore more information about them and other, possibly better schemata.
This remarkable property is commonly called the implicit parallelism of GAs. A simple GA has only m
structures in one time step, without any memory or bookkeeping about rhe previous generations. We will
now ny to gee a feeling how many schemata a GA actually processes.
Obviously, there are 3n schemata of length 11. A single binary string fulfills n schema of order I, ( il
schemata of order 2, in general, <k) schemata of order k. Hence, a ming fulfills

t(:)=2"

l
k:=::l
 r
426 Genetic Algorithm

Theorem. Consider a randomly generated starr population of a simple GA and let e E (0, 1) be a fixed error
bound. Then schemata of length

i1 <E(n-l)+l

have a probability of at least (1-e) to survive one-point crossover (compare with the proof of rhe Schema
Theorem). If the population size is chosen as m = 2~/2, rhe number of schemata, which survive for the nexr
generation, is of order O(m3).

115.14 Classification of Genetic Algorithm

There exist wide variety of GAs including simple and general GAs discussed in Sections 15.4 and 15.5,
respectively. Some or her variants of GA are discussed below.

I 15.14.1 Messy Genetic Algorithms

In a "classical" GA, rhe genes are encoded in a fixed order. The meaning of a single gene is determined by
irs position inside the string. We have seen in the previous chapter that a GA is likely to converge well if the
optimization rask can be divided imo several short building blocks. What, however, happens if the coding is
chosen such that couplings occur between distant genes? Of course, one-point crossover rends to disadvantage
long schemata {even if they have low order) over short ones.
Messy GAs try w overcome this difficulty by using a variable-length, position-independent coding. The
key idea is to append an index to each gene which allows identifying irs position. A gene, therefore, is no longer
represented as a single allele value and a fixed position, bur as a pair of an index and an allele. Figure 1S-34(A)
shows how rhis "messy" coding works for a string oflength 6.
Since with the help of rhe index we can identify the genes uniquely, genes may be swapped arbitrarily
without changing the meaning of the string. With appropriate genetic operations, which also change the
order of rbe pairs, the GA could possibly group coupled genes roger her automatically.

GillfiJi\[ji;?;:qX';:':I&W;'iii:;;.I~:N~ilt:,c? d. 4 1 Pl
l l I I I I
(A)

(B)

Figure 1534 (A) Messy coding and (B) positional preference; Genes with indices 1 and 6 occur twice,
the fim occurrences are used.

L
r
15.14 Classification of Genetic Algorithm
427

CUI Splice

A B

i-ii
C D
lmti!lt~:"'ai\gt~'t"l
rfp:.jg:....~ - . ~ ~--
'
1(2.0) I I
(4.0) (2.0) I (3.0) I
DC

Figure 1535 The cur and splice operation.

Owing to the free arrangement of genes and rhe variable length of rhe encoding, we can, however, run into.
problems, which do nm occur, in a simple GA. First of all, it can happen that there are cwo entries in a string,
which correspond to rhe same index bur have conflicting alleles. The most obvious way to overcome rhis
"over-specification" is positional preference- the first entry, which refers to a gene, is taken. Figure 15-34(B)
shows an example. The reader may have observed that the genes with indices 3 and 5 do nor occur at all in
the example in Figure 15-34(B). This problem of "under specification" is more complicated and irs solution
is not as obvious as for ovei=-fpecificarion. Of course, a lor of variants are reasonable.
One approach could be to check a!l possible combinations and to rake the best one (fork missing genes,
there are 2k combinations). Wirh the objective ro reduce this effort, Goldberg ct al. have suggested to use
so-called competitive templates for finding specifications fork missing genes. lr is nothing else rhan applying
a local hill climbing method with random initial value to rhe k missing genes.
While messy GAs usually work wirh the same mutadon operator as simple GAs (every alielc is altered
with a low probabiliry pM), rhe crossover operator is replaced by a more general cut and splice operaror
which also allows to mate parents wirh different lengths. The basic idea is to choose em sires for both parents
independently and to splice the four fragmems. Figure 15-35 shows an .:xample.

115.14.2 Adaptive Genetic Algorithms

Adaptive GAs are those whose parameters, such as the population si2.e, rhe crossing over probability, or
the mutation probability, are varied while the GA is running. A simple variant could be the following: The
mutation rate is changed according to changes in the population- rhe longer rhe population does nor improve,
rhe higher the mutation rare is chosen. Vice versa, it is decreased again as soon as an improvement of rhe
population occurs.

t
i 15.14.2.1 Adaptive Probabilities of Crossover and Mutation
I
~tis essenrial to have two characteristics in GAs for optimi2.ing multi modal functions. The first characteristic
I 15
the capacity to converge wan optimum (local or global) after locating the region containing the optimum.
! The second characteristic is the capacity to explore new regions of rhe solution space in search of the global
I optimum. The b:dance between rhese characteristics of the GA is dictated by the values of p111 and Pn and the
type of crossover employed. Increasing values ofp 111 and Pr promote explorario11 ar rhc expense of exploitation.

! Moderately large values of Pc (in rhe range 0.5-1.0) and small values of p111 (in rhe range 0.001-0.05) are
commonly employed in GA practice. In rhis approach, we aim at achieving this tradeoffbernoeen exploration

L
 =I

428 Genetic Algorithm

J
!
and exploitation in a different manner, by varyingp, and Pm adapcively in response to the fitness values of
the solutions; Pr and Pm are increased when the population tends to gee stuck at a local optimum and are
decreased when the population is scattered in rhe solution space.

15.14.2.2 Design of Adaptive p, and Pm

To vary Pr and Pm adaprively for preventing premature convergence of the GA to a local optimum, it is
essential w identify wheWer the GA is converging to an optimum. One possible way of detecting is to observe
J
average fitness value of the population in relation to the maximum fitness value [m3Y. of the population. The
value[rrux-J is likely to be less for a population that has converged to an optimum solution than char for a
population scattered in the solution space. We have observed the above property in all our experiments with
GAs, and Figure 15-36 illumates rhe property for a typical case. In Figure 15-36 we notice that /max -J
decreases when the GAconverges to a local optimum wirh a fitness valueof0.5. (The globally optimal solution
has a fitness value of 1.0.) We use the difference in the average and maximum firness value,[max -],as a
yardstick for detecting the convergence of the GA. The values ofp, and p111 are varied depending on the value
-f.
of [mxs. Since p, and Pm have to be increased when the GA converges to a local optimum, i.e., when
-J
/mv. decreases, p, and Pm will have to be varied inversely with/max -f.
The expressions that we have
chosen for p, and Pm are of rhe form

Pc = M/mu -J)
Pm = k,!lfmu -J)
It has to be observed in rhe above expressions rhat p, and Pm do nor depend on the firness value of any
particular solmion, and have the same values for all the solution of the popularion. Consequendy, solmions
with high fitness values as well as solutions wirh low firness values are subjected to rhe same levels of mutation
and crossover. When a population converges to a globally optimal solurion (or even a locally optimal solurion),
p, and p111 increase and may cause the disruption of rhe near-optimal solutions. The population may never

0.6

0.5
I Best

0.4
0
0
~
0 0.3
5 r:-1 ', Pop. Max. Avg.
'
" 0.2

,_,

0 5 10 15 20 25 30 35 40
Generation
Figure 1536 Variation of/mu. -J and /bcs (besr firness).

I
1

I
k
15.14 Classification of Genetic Algorithm 429

converge to the global optimum. Though we may prevent the GA from getting stuck ar a local optimum, the
performance of the GA (in terms of the generations required for convergence) will certainly deteriorate.
To overcome the above-stated problem, we need to preserve "good" solutions of the population. This can
be achieved by having lower values of p, and p,1 for high fitness solutions and higher values of p, and Pm
for low fitness solutions. While the high fitness solu~ions aid in the convergence of the GA, the low fitness
solutions prevem rhe GA from getting stuck ;:n a local optimum. The value of Pm should depend not only
on fmn -1 but also on the fitness value /of rhe solution. Similarly, p, should depend on the fitness values
of both the parent solutions. The closer f is to /m3'1. the smaller Pm should be, i.e., Pm should vary directly
as [m'XI. - f Similarly, p, should vary directly asfmax - f', where f' is rhe larger of rhe fimess value of the
solutions to be crossed. The expressions for Pc and Pm now rake r~e forms

p, = h [(Jm,. - J')i(/m,. -J )], k, -" 1.0

Pm = k,[(Jm,.- fJ/(fmu -f)], k2-" 1.0
(Here k, and kz have to be less rhan 1.0 to constrain Pr and Pm to the range 0.0-1.0.)
Note that Pc and Pm are zero for the solution with the maximum fitness. Alsop, = k1 for a solution with
f.
f ::.f, andpm = /q for a solution with/= For solution with subaverage f1mess values, i.e.,/< f,p, and
Pm might assume values larger than 1.0. To prevent the overshooting of Pc and Pm beyond 1.0, we also have
the following constraints:

= k,, J' -"f

p,
Pm = k4,js_J
where k3, k4 :S I .0.

15.14.2.3 Practical Considerations and Choice of Values for k1, k2, ka and k4
In rhe previous subsection, we saw that for a solurion with the maximum firness value Pc and Pm are borh
zero. The best solution in a population is transferred undisrupted into the next generation. Together wirh
the selection mechanism, this may lead to an exponential growth of the solution in the population and may
cause premature convergence. To overcome the above-mued problem, we introduce a default mutation rate
(ofO.OOS) for every solurion in the Adaptive Generic Algorithm (AGA).
We now discuss the choice of values for k1, kz, k3 and k4. For convenience, the expressions for Pc and Pm
are given as

p, = k, (j;,, - j')i(j;,,. -l ), f "l

p, = k,.J<l
Pm = k,(Jm,. - J')/(J~u - Jl, J 00 J
p,,;:::: k4, !<?
where k1, ~' k3 , k4 ::; 1.0.
It has been well established in GA literature that moderately large values of p, (0.5 < p, < 1.0) and small
values ofPm (0.001 < Pm < 0.05) are essential for the successful working of GAs. The moderately large values
of Pc promote the extensive recombination of schemata, while small values of Pm are necessary to prevent the
disruption of rhe solutions. These guidelines, however, are useful and relevant when the values of p, and Pm
do not vary.
 430 Genetic Algorithm

One of the goals of rhe approach is to prevent the GA from gerring sruck at a local optimum. To achieve
this goal, we employ solurions with sub\average firnesses to search the search space for the region containing
the global optimum. Such solmions nerd robe completely disrupted, and for this purpose we we a value of
0.5 for k4. Since solutions with a fitness value of[ should also be disrupted completely, we assign a value of
0.5 to k2 as well.
Based on similar reasoning, we assign k1 and k.~ a value of 1.0. This ensures that all solutions with a fitness
J
value less than or equal to compulsorily undergo crossover. The probability of crossover decreases as the
fimessvalue (maximum of the fitness values of the parent solutions) tends to /max and is 0.0 for solutions with
a fimess value equal to [mn

I 15.14.3 Hybrid Gene1ic Algorithms

k they use the firness function only in rhe selection step, GAs are blind oprimizers which do not use
any auxiliarl information such as derivatives or other specific knowledge about rhe special strucrure of the
objective function. If there is such knowledge, however, ir is unwise and inefficient not ro make use of ir.
Several investigations have shown that a lot of synergism lies in the combination of generic alj!orirhms and
conventional methods.
The basic idea is co divide rhe optimization rask into twO complementary pam. The GA does rhe coarse,
global optimiz.arion while local refinement is done by rhe convemional method (e.g. gradient-baseJ, hill
climbing, greedy algorithm, simulated annealing, ere.). A number of variants are reasonable:

~ I. The GA performs coarse search first. Afrer the GA is completed, local refinement is done.
2. The local method is integrated in the GA. For instance, every K generations, the pcpulation is doped with
. a locally optimal individual .
' 3. Both methods run in parallel: All individuals are continuously used as initial values for the local method.
The locally optimized individuals are re-implanred into the current genemion.

In this section a novel optimization approach is used that switches bmveen global and local search methods
based on the local topography of the design space. The global and local optimizers work in concert to efficiently
locate quality design points better than either could alone. To determine when it is appropriate to execute
a local search, some characteristics about the local area of the design space need m be determined. One
good source of information is contained in the population of designs in rhe GA. By calculating the relarive
homogeneity of the population we can get a good idea of whether there are multiple local optima located
within this local region of the design space.
To quantify the relative homogeneiry of the population in each subspace, the coefficient of variance of the
objective function and design variables is calculated. The coefficient of variance is a normalized measure of
variation, and unlike the actual variance, is independent of the magnitude of the mean of the population.
A high coefficient of variance could be an indication that there are multiple local optima present. Very low
values could indicate that the GA has converged to a small area in the design space, warranting the use of a
local search algorithm to find the best design within this region.
By calculating the coefficient of variance of the both the design variables and the objective function as the
optimization progresses, it can aJso be used as a criterion to switch from me global ro the local optimirer.
As ilie variance of the objective values and design variables of the population increases, it may indicate that
the optimizer is exploring new areas of the design space or hill climbing. If the variance is decreasing, the
optimizer may be converging toward local minima and the oprimiz.ation process could be made more efficienr
by switching to a local search algorithm.
 15.14 Classification of Genetic Algorithm 431

The second method, regression analysis, used in this section helps us determine when w switch berween
the global and local optimizer. The design data present in the current population of ilie GA can be used to
provide information as to the local topography of the design space by attempting ro fit models of various
order to ir.
The use of regression analysis to augment optimizatiOn algorithms is not new. In problems in which the
objective function or consrrainrs are computationally expensive, approximations to the design space are created
by sampling the design space and then using regression or other methods to create a simple mathematical model
that closely approximates the actual design space, which may be highly nonlinear. The design space can then
be explored to find regions of good designs or Gptimized to improve the performance of the system using the
predictive surrogate approximation models instead of the computarionaJly expensive analysis code, resulting
in large computational savings. The most common regression models are linear and quadratic polynomials
created by performing ordinary least squares regr~ssion on a set of analysis data.
To make dear the use of regression analysis in this way, consider Figure 15-37, which represents a complex
design space. Our goal is to minimire this function, and as a first step the GA is run. Suppose that afrer a
certain number of generarions the population consists of the sampled points shown in the figure. Since the
population of the GA is spread rhroughour the design space, having yet ro converge into one of the local
minima, it seems logical to continue the GA for additional generations. Ideally, before the local optimizer is
run it would be beneficial to have some confidence that irs starring point is somewhere within the mode that
comains rhe oprimum. Fitting a second-order response surface to rhe data and noting the large error (the R2
value is 0.13), ther~ is a dear indication that the GA is currently exploring multiple modes in rhe design space.
In Figure 15-38, rhe same design space is shown bur afrer the GA has begun ro converge inro the part
of the design space containing rhe optimal design. Once again a second-order approximation is fir to GA's
population. The cloned line connects the points predicted by the response surface. Note how much smaller
the error is in rhe approximation (the R2 is 0.96), which is a good indication that the GA is currently exploring
a single mode within the design space. At this point, rhe local optimizer can be made to quickly converge
ro the best solution within this area of the design space, rhereby avoiding the slow convergence propenies
of ,he GA.
After each generarion of the global optimizer. the values of rhe coefficient of determination and rhe
coefficient of variance of the enrire population are compared wirh the designer specified threshold levels.

15

.
,.---v:
10

'5
"'

-5
0 ~
' ..

- - I
. '
,
:
'

I I
'
'
:
'

-
e Sampled designs
Second-order lit
True design space

-10
..
0 1 2 3 4 5 6 7 B 9 10

'
I Figure 1537 Approximating mulriple modes with a second-order model.

J
 ~

!
432 Genetic Algorithm

8
6
4 '
d\ '
"
"
"'
0
-2
./ii - Sampled designs
Second-order fit
True design space

~
-4 j

-
-6 '
-6
-10
5 5.5 6 6.5 7 7.5 8 8.5 9.0
X

Figure 1538 Approximating a single mode with a second-order model.

The first threshold simply states that if coefficient of determination of the popuJarion exceeds a designer set
value when a second-order regression analysis is performed on the design data in the current GA population,
then a local search is started from the current 'best design' in the population. The second threshold is based
on the value of the coefficient of variance of the entire population. This threshold is also set by the designer
and can range upwards from Oo/o. If it increases at a rate greater than the threshold level then a local s~arch is
execuced from rhe best point in rhe population.
The flowchart in Figure 15-39 illustrates the stages in the algorithm. The algorithm can switch repeatedly
between the global search (Stage 1) and the local search (Stage 2) during execution. In Stage I, the global
search is initialized and chen monitored. This is also where the regression and sratisrical analysis occurs.
In Stage 2 the local search is executed when the threshold levels are exceeded, and then chis solution is
passed back and imegrared imo the global search. The algorithm scops when convergence is achieved for the
global optimization algorithm.

115.14.4 Parallel Genetic Algorithm

GAs are powerful search techniques char are used successfully ro solve problems in many different disciplines.
Parallel GAs (PGAs) are particularly easy co implemenr and promise substantial gains in performance. As such,
there has been extensive research in chis field. The section describes some of the most significant problems in
modeling and designing multi-population PGAs and presents some recent advancemenrs.
One of the major aspects ofGA is their ability to be parallelized. Indeed, because nawral evolution deals with
an entire population and not only with pan:icular individuals, it is a remarkably highly parallel process. Except '
i11 the selection phase, during which there is competition between individuals, the only interactions between
rr.embers of the population occur during the reproduction phase, and usually, no more than two individuals I
are necessary to engender a new child. Otherwise, any ocher operations of the evolution, in particular the
evaluation of each member of the population, can be done separately. So, nearly all the operations in a genetic
I
algorithm are implicitly pacallel.
PGAs simply consist in distributing the task of a basic GA on different processors. As those casks are I
implicitly parallel, little time will be spent on communication; and rhus, the algorithm is expected to run
much faster or to find more accurate resulrs.
It has been established chat GA's efficiency co find optimal solution is largely determined by the population
I
size. With a larger population size, the genetic diversity increases, and so the algorithm is more likdy to find

I
j..
 15.14 Classification of Genetic Algorithm 433

----------------------------------------
lnilialize global search methods
STAGE 1

Begin global search

-----
'

~&es
~---- I
Monitor global search Thresholds Trigger local
Collect statistics exceeded? : search
Perform regression analysis
': ''------r-----
'
'
-_-_-_-_-_-___.J_-------

STAGE2
Execute local
search
, ___________ ( __________________ _
,I I

Check global convergence No

Yes Exit

Figure 1539 Steps in twostage hybrid optimization approach. i

j,

~
a global optimum! A large population requires more memory to be scored; it has also been proved char it
takes a longer time to converge. If 11 is the population size, the convergence is expected aft:er n log(n) function
evaluations.
The use of mday's new parallel computers not only provides more storage space but also allows the use
tl~
of several processors w produce and evaluate more solutions in a smaller amount of rime. By parallelizing
,,
\!
_the algorithm, it is possible D increase population size, reduce rhe computational cost, and so improve rhe
performance of the GA. li
Probably the fim auempt to map GAs to existing parallel computer archirecrures was made in 1981 by 1:
John Grefensrerre. Bur obviously today, with the emergence of new high-performance computing (HPC), I!
PGA is really a flourishing area. Researchers try ro improve performance of GAs. The stake is to show that
I!

Ii;
GAs are one of the besr optimization methods to be used with HPC.
I
15.14.4.1 Global Parallelization
The first attempt ro parallelize GAs simply consists of global parallelization. This approach nics to explicitly !"
parallelize the implicit parallel tasks of the "sequential" GA. The nature of the problems remains unchanged.
1:
The algorithm still manipulates a single population where each individual can mare with any ocher, but the
breeding of new children and/or their evaluation are now made in parallel. The basic idea is rhar different
,,
I

processors can create new individuals and compme their fir ness in parallel almost without any communication I
among each other. '
To starr with, doing rhe evaluation of the population in paraHel is something really simple co implement.
Each processor is assigned a subset of individuals to be evaluated. For example, on a shared memory computer,

.
434 Genetic Algorithm

individuals could be stored in shared memory, so that each processor can read the chromosori:tes assigned and
c:an write back the resnlr of the firness computation. This method only supposes iliat the GA works with a
generational update of the population. Of course, some synchronization is needed between generations.
Generally, most of the computational time in a GA is spent calling the evaluation function. The time
spent in manipulating the chromosomes during the selection or recombination phase is usually negligible.
By assigning to each processor a subset of individuals m evaluate, a speed~up proportional to the number of
processors can be expeaed if there is a good load balancing between them. However, load balancing should
not be a problem as generally the rime spent for the evolution of an individual does not really depend on dle
individual. A simple dynamic scheduling algorithm is usually enough to share the population between each
processor equally.
On a distribmed memory compUter, we can smre che population in one "master" processor responsible
for sending che individuals to the other processors, i.e., "slaves." The master processor is also responsible
for collecting the result of the evaluation. A drawback of this distributed memory implementation is that
a bottleneck may occur when slaves are idle while only the master is working. But a simple and good use
of the master processor can improve the load balancing by distributing individuals dynamically to rhe slave
processors when they finish their jobs.
A further seep could consist in applying rhe generic operators in parallel. In fact, the interaction inside
the population only occurs during selection. The breeding, involving only two individuals to generate ~he
offspring, could easily be done simultaneously over n/2 pairs of individuals. Bur it is not chat clear if it worth
doing so. Crossover is usually very simple and not so rime-consuming; the point is nor rhat roo much rime will
be lost during the communication, bur that the time gain in the algorithm will be almost nothing compared
to the effort produced to change the code.
This kind of global parallelization simply shows how easy it can be to transpose any GA onto a parallel
machine and how a speed-up sublinear to the number of processors may be expected.

15.14.4.2 Classification of Parallel GAs

The basic idea behind most parallel programs is to divide a cask into chunks and co solve the chunkssimulrane-
ously using multiple processors. This divide-and-conquer approach can be applied to GAs in many different
ways, and che literature contains many examples of successful parallel implementations. Some parallelizacion
methods use a single population, while others divide the population into several relatively isolated subpopu-
lacions. Some methods can exploit massively parallel computer architectures, while ochers are better suited to
multicomputers with fewer and more powerful processing elements.
There are three main cypes ofPGAs:
1. global single-population master-slave GAs,
2. single-population fine-grained,
3. multiple-population coarse-grained GAs.
In a master-slave GA there is a single panmicric population (just as in a simple GA), but the evaluation
of fitness is distributed among several processors (see Figure 15-40). Since in rhis type of PGA, selection
and crossover consider the entire population it is also known as global PGA. Fine-grained PGAs are suited
for massively parallel computers and consist of one spatially structured population. Selection and mating are
resrricred to a small neighborhood, but neighborhoods overlap permitting some interaction among all the
individuals (see Figure 15-41 for a schematic of this class of GAs). The ideal case is co have only one individual
for every processing element available.
Multiple-popuJarion (or multiple-deme) GAs are more sophisticated, as they consist in several subpopu-
lacions which exchange individuals occasionally (Figure 15-42 has a schematic). This exchange of individuals
15.14 Classification of Genetic Algorithm 435

Master

Workers

Figure 1540 A schematic of a master-slave PGA. The master stores the population, executes GA operations
and distributes individuals ro the slaves. The slaves only evaluate the fimess of the individuals.

Figure 1541 A schematic of a fine-grained PGA. This class ofPGAs has one spadally distributed popularion,
and ir can be implemented very efficienrly on massively parallel compmers.

Figure 1542 A schematic of a mulciple-popularion PGA. Each process is a simple GA, and rhere is
(infrequent) communicadon between the populations.

is called migration and, as we shall see in later sections, it is controlled by several parameters. Multiple-deme
GAs are very popular, but also are the class ofPGAs which is most difficult to understand, because the effecrs
of migration are not fully understood. Mulriple-deme PGAs introduce fundamental changes in the operation
of the GA and have a different behavior than simple GAs.
Multiple-deme PGAs are known with different names. Sometimes they are known as "distributed" GAs,
because they are usually implemented on distributed memory MIMD computers. Since ilie computation to
communication ratio is usually high, they are occasionally called coarse-grained GAs. Finally, mulriple~deme
 ~..

436 Genetic Algorithm

l
~
y
GAs resemble the "island model" in Population Genetics which considers relatively isolated demes, so the 11
PGAs are also known as "island" PGAs. Since the size of the demes is smaller than the population used by a ~.
serial GA, we would expect that lhe PGA converges faster. However, when we compare the performance of ~
the serial and the parallel algorithms, we must also consider the qualicy of the solurions found in each case.
Therefore, while it is true that smaller demes converge faster, it is also true iliar the qualicy of the solution
mighr be poorer.
It is impon:am to emphasize that while the master-slave parallelizarion method does not affect the behavior
of the algorithm, the last rwo methods change the way the GA works. For example, in master-slave PGAs,
selection takes into account all the population, but in the ocher rwo PGAs, seleccion only considers a subset
of individuals. Also, in the mascer~slave any two individuals in che population can mare (i.e., there is random
mating), but in the other methods mating is restricted to a subset of individuals.
The final merhod to parallelize GAs combines mulriple demes with masrer~slave or fine~grained GAs. We
call this class of algorithms hierarchical PGAs, because at a higher level they are multiple~deme algorithms with
single-population PGAs (either master-slave or fine~grained) at the lower level. A hierarchical PGA combines
the benefits of its components, and it promises bener performance than any of rhem alone.

Master-slave parallelization: This section reviews the master~slave (or global) parallelization method. The
algorithm uses a single population and the evaluation of the individuals and/or the application of generic
operators are done in parallel. As in the serial GA, each individual may compete and mate with any other (thus
selection and mating are global). Global PGAs are usually implemented as masrer-slave programs, where the
master stores the population and the slaves evaluate the fitness.
The most common operation iliac is parallelized is rhe evaluation of the individuals, because rhe fitness of
an individual is independent from the rest of the population, and there is no need to communicme during
chis phase. The evaluation of individuals is parallelizcd by assigning a fraction of the population to each
of the processors available. Communication occurs only as each slave receives irs subset of individuals to
evaluate and when rhe slaves return the firness values. If the algorithm stops and waits to receive the fitness
values for all the population before proceeding inro rhe next generation, then the algorithm is synchronous.
A synchronous mastcr~slave GA has exacdy rhe same properties as a simple GA, with speed being the only
difference. However, ir is also possible to implement an asynchronous master-slave GA where the algorithm
does nor stop to wait for any slow processors, bur it does nor work exactly like a simple GA. Most global PGA
implementations are synchronous and the rest of the paper assumes that global PGA~ carry our exactly rhe
same search of simple GAs.
The global paralleliz.acion model does nor assume anything about the underlying computer architecture,
and it can be implemented efficiently on shared~ memory and distributed-memory computers. On a shared-
memory multiprocessor, the popul:.tion could be slOred in shared memory and each processor can read rhe
individuals assigned co it and write the evaluation results back without any conflicts.
On adisrribured-memorycomputer, the population can be scored in one processor. This "master" processor
would be responsible for explicidy sending the individuals to rhe oilier processors {the "slaves") for evaluation,
collecting the results and applying rhe generic operators ro produce the next generation. The number of
individuals assigned to any processor may be constant, bur in some cases (like in a multiuser environment
where the uciliz.acion of processors is variable) ir may be necessary ro balance the computational load among
the processors by using a dynamic scheduling algorithm (e.g., guided self~scheduling).

Multiple-deme parallel GAs: The imponant characteristics of mulriple-deme PGAs are che use of a few i
relarively large subpopulations and migration. Mulciple-deme GAs are rhe most popular parallel method, and
many papers have been written describing innumerable aspects and derails of their implementation. l

l
 15.14 Classification of Genetic Algorithm 437

Probably the first systematic srudy of PGA<i with mulriple populations was Grosso's dissertation. His
objective was to simulate the interaction of several parallel subcomponents of an evolving population. Grosso
simulated diploid individuals (so there were rwo subcomponentS for each "gene"), and the population was
divided into five demes. Each deme exchanged individuals with all the others with a fixed migration rate.
With controlled experiments, Grosso found cha~ the improvement of the average population fitness was
fasrer in the smaller demes rhan in a single large panmictic population. This confirms a long~held principle in
Population Genetics: favorable traits spread faster when che demes are small chan when che demes are large.
However, he also observed that when che demes were isolated, the rapid rise in fitness stopped at a lower fitness
value than with the large population. In other words, the quality of che solution found after convergence was
worse in the isolated case chan in the single population.
With a low migration rate, the demes still behaved independently and explored different regions of the
search space. The migrants did nor have a significant effect on the receiving deme and the quality of the
solutions was similar to the case where the demes were isolated. However, at intermediate migration rates the
divided population found solutions similar to those found in the panmictic population. These observations
indicate that there is a critical migration rate below which the performance of the algorithm is obstructed by
rhe isolation of the demes, and above which the partitioned population finds solutions of the same quality as
rhe panmictic population.
It is interesting rhar such important observations were made so long ago, at the same time that other
systematic studies ofPGAs were underway. For example, Tanese proposed a PGA with the demes connected
on a four~dimensional hypercube topology. In Tanese's algorithm, migration occurred at fixed intervals between
processors aJong one dimension of rhe hypercube. The migrants were chosen probabilistically from the best
individuals in the subpopulation, and they replaced the worst individuals in the receiving deme. Tanese carried
out three sees of experiments. In the first, the interval between migrations was ser ro five generations, and
the number of processors varied. In tests wirh two migration rates and varying the number of processors, the
PGA found results of the same quality as the serial GA. However, it is difficulc ro see from the experimental
results if the PGA found the solutions sooner than the serial GA, because the range of the cimes is roo large.
In the second set of experiments, Tanese varied the murarion and crossover rates in each deme, attempting
ro find parameter values to balance exploration and exploitation. The third set of experiments studied the
effect of the exchange frequency on the search, and the results showed thar migrating roo frequendy or roo
infrequently degraded the performance of the algorithm.
The multideme PGAs are popular due ro rhe following several reasons:

l. Multiple-deme GAs seem like a simple extension of the serial GA. The recipe is simple: take a few
conventional (serial) GAs, run each of them on a node of a parallel computer, and at some predetermined
times exchange a few individuals.
2. There is relatively little extra effort needed ro convert a serial GA inro a mulriple-deme GA. Most of the
program of the serial GA remains the same and only a few subroutines need to be added co implement
migration.
3. Coarse-grain parallel computers are easily available, and even when they are not, it is easy co simulate one
with a network of workstations or even on a single processor using free software (like MPI or PVM).

There are a few important issues noted from rhe above sections. For example, PGAs are very promising in

i termsofthegains in performance. Also, PGAsare more complex than their serial counterpartS. In particular, rhe
migration of individuals from one deme ro anorher is controlled by several p:uameters like (a) the ropology that

l defines the connections between rhe subpopulations, (b) a migr;uion r;Ht:: rh.lt controls how m;my individuals
migrate and (c) a migration inrerval char affecrs rhe freqU<'lK~ of mi~r.1~inn~. In rht.' btl' 11lS(h .ullllarl~ 1990~ .

l .
438 Genetic Algorithm

the research on PGA:; began ro explore alternatives ro make PGAs faster and to understand better how they
worked.
Around this time the first theorecical srudies on PGAs began to appear and the empirical research attempted
to identify favorable parameters. This section reviews some of that early theoretical work and experimental
srudies on migration and ropologies. Also in this period, more researchers began to use mulciple~population
GAs co solve application problems, and this section ends with a brief review of their work.
One of rhe directions in which the field matured is that PGAs began to be tested with very large and
difficult test functions.

Fine-grained PGAs: The devdopment of massively paraHel compmers triggers a new approach of PGAs.
To l:ak.e advantage of new archirecrures with even a greater number of processors and less communication
coslS, fine-grained PGAs have been devdoped. The population is now partitioned into a la..tge number of very
smaJl subpopulations. The limit (and may be ideal) case is to have just one individual for every processing
element available.
"Basically, the population is mapped onto a connected processor graph, usually, one individual on each
processor. (But it works also more than one individual on each processor. In this case, it is preferable to choose
a multiple of the number of processors for the population size.) Mating is only possible between neighboring
individual, i.e, individuals stored on neighboring processors. The selection is also done in a neighborhood of
each individual and so depends only on local information. A motivation behind local selection is biological.
In nature there is no global selection, instead natural selection is a local phenomenon, raking place in an
individual's local environment.
If we want to compare this model to the island model. each neighborhood can be considered as a different
deme. But here, the demes overlap providing a way w disseminate good solutions across the entire population.
Thus, the topology does not need w explicitly define migration roads and migration rare.
Ir is common to place the population on a two-dimensional or three-dimensional torus grid because in
many massively parallel computers the processing elements are connected using this topology. Consequently
each individual has four neighbors. Experimentally, it seems that good results can be obtained using a topol-
ogy with a medium diameter and neighborhoods nor roo large. Like the coarse-grained models, it worth
trying to simulate this model even on a single processor to improve the results. Indeed, when the popula-
tion is stored in a grid like this, after few generations, different optima could appear in different places on
rhe grid.
To sum up, with parallelization of GA, all the different models proposed and all the new models we can
imagine by mixing those ones, can demonstrate how well GA are adapted to parallel compmarion. In fact, the
too many implementations reponed in the literature may even be confusing. We really need to understand
what truly affects the performance ofPGAs.
Fine-grained PGAs have only one population, bur have a spatial structure that limits the interactions
between individuals. An individual can only compere and mate with its neigh~ors; bur since the neighborhoods
overlap good solutions may disseminate across the entire population.
Robertson parallelized the GA of a classifier system on a Connection Machine 1. He paraJielized the
selection of parents, the selection of classifiers to replace, mating, and cl-ossover. The execution time of his
implementation was independent of ilie number of classifiers (up to 16K, the number of processing elements
in the CM-1).

Hierarchical parallel algorithms: A few researchers have cried to combine two of the methods to pacalleliz.e
GAs, producing hierarchical PGAs. Some of these new hybrid algorithms add a new degree of complexity to
.the already complicated scene ofPGAs, but other hybrids manage tO keep the same complexity as one of their
 15. t 4 Classification of Genetic Algorithm 439

components. When two methods of pacallelizing GAs are combined they form a hierarchy. At the upper level
most of the hybrid PGAs ace multiple-population algorithms.
Some hybrids have a fine-grained GA at the lower level (see Figure 15-43). For example Gruau invented a
"mixed" PGA. In his algorithm, the population of each deme was placed on a two-dimensional grid, and the
demes themselves were connected as a rwo-dimensior:tal tOM. Migration between demes occurred at regulae
intervals, and good results were reported for a novel neucal network design and uaining application.
Another type of hierarchical PGA uses a master-slave on each of the demes of a multi-population GA (see
Figure 15-44). Migration occurs between demes, and the evaluation of the individuals is handled in paraJlel.
This approach does not introduce new analytic problems, and it can be useful when worlt:ing with complex
applications with objective functions that need a considerable amount of compurarion rime. Bianchini and

HE

HE HE
Figure 15-43 Hierarchical GA combines a multiple-deme GA (ar rhe upper level) and a fine-grained GA
{at rhe lower level).

'
I
Figure 15-44 A schematic of a hierarchical PGA. At the upper level rhis hybrid is a mulci-deme PGA where
each node is a master-slave GA.

\
l.
440 Genetic Algorithm

Figure 1545 This hybrid uses mulciple-deme GAs ar both the upper and the lower levels. At the lower level
the migration rate is faster and the communicarions topology is much denser than at rhe
upper level.

Brown presented an example of iliis method of hybridizing PGAs, and showed that it can find a solution of
the same qualiry as of a masrer~slave PGA or a multiple~deme GAin less time.
Interestingly, a very similar concept was invented by Goldberg in the context of an objecroriented imple-
mentation of a "community model" PGA. In each "community" there are multiple houses where parents
reproduce and the offsprings are evaluated. Also, rhere are mulriple communities and ir is possible that
individuals migrate to other places.
A third method of hybridizing PGAs is to use mulriple-deme GAs at both rhe upper and the lower levels
(see Figure 15-45). The idea is ro force panmiaic mixing ar the lower level by using a high migration rate and
a dense topology, while a low migration rate is used at the high level. The complexity of this hybrid would
be equivalent ro a multiple~popularion GA if we consider the groups of panmicric subpopularions as a single
deme. This method has nor been implemented yet. Hierarchical implementations can reduce the execution
time more than any of their components alone.
15.14.4.3 Coarse Grained PGAs- The Island Model
The second class ofPGA is once again inspired by nature. The population is now divided inro a few subpopu~
lations or demes, and each of these relatively large demes evolves separately on different processors. Exchange
between subpopularions is possible via a migration operator. The term island model is easily understandable;
the GA behave as if rhe world was constituted of islands where populations evolve isolated from each other.
On each island the population is free to converge mward different optima. The migration operator allows
"merissage" of the different sub populations and is supposed to mix good features that emerge locally in the
different demes.
We can notice chat this time the nature of the algorithm changes. An individual can no longer breed
with any other from the entire population, but only with individuals of the same island. Amazingly, even
if this algorithm has been developed to be used on several processors, it is wonh simulating it sequentially
on one processor. Ic has been shown on a few problems that better results can be achieved using this model.
This algorithm is able ro give different sub~optimal solutions, and in many problems, it is an advantage if
we need to determine a kind of landscape in the search space to know where the good solutions are located.
Anoilier great advantage of the island model is iliat cite population in each island can evolve wiili different
''
'\
I
........
 15.14 Classification of Genetic Algorithm 441

rules. That can be used for multicriterion optimization. On each island, selection can be made according
ro different fimess functions, representing different criterions. For example it can be useful to have as many
islands as criteria, plus another central island where 'selection is done with a multicriterion fitness function.
The migration operator allows individuals ro move betw~en islands, and therefore, m mix criteria.
In lirerarure iliis model is sometimes also referred as the coarse~grained PGA. (In parallelism, grain size
refers m ilie ratio of time spent in computation and time spent in communication; when rhe ratio is high the
processing is called coarse~grained). Sometimes, we can also find the rerm "distribmed" GA, since iliey are
usually implemented on distributed memory machines (MIMD Computers).
Technically iliere are three important featureS in the coarse~grained PGA: the topology char defines connec~
tions between sub populations, migration rare that controls how many individuals migrate, migration intervals
chat affect how often the migration occurs. Even if a lor of work has been done ro find optimal mpology and
migration parameters, here, intuition is still used more often than analysis wirh quite good results.
Many topologies can be defined m connect rhe demes, but the most common models are rhe island model
and the srepping~stones model. In ilie basic island model, migration can occur between any subpopularions,
whereas in the S(epping stone demes are disposed on a ring and migration is restricted to neighbouring demes.
Works have shown that cite topology of rhe space is nor so important as long as ir has high connectivity and
small diameter to ensure adequate mixing as rim.:! proceeds.
Choosing the right rime for migration and which individuals should migrate appears to be more com~
plicated. Quite a lor of work is done on this subject, and problems come from the following dilemmas. We
can observe that species are converging quickly in small isolated populations. Nevertheless, migrations should
occur after a rime long enough for allowing the development of goods characteristics in each subpopulation.
1r also appears that, immigration is a trigger for evolutionary changes. If mjgrarion occurs after each new
generation, the algorithm is more or le~ equivalent to a sequencia\ GA with a larger population. In praaice,
migration occurs either after a fixed number of iterations in each deme or at uniform periods of rime. Migrants
are usually selected randomly from the best individuals in rhe population and they replace rhe WOfS( in the
receiving deme. In fact, intuition is srill mainly used to fix migration rare and migration intervals; there is
absolurely nothing rigid, each personal cooking recipe may give good results.

115.14.5 Independent Sampling Genetic Algorithm (ISGA)

In the independent sampling phase, we design a core scheme, named rhe "Building Block DerccringSrrategy"
(BBDS), to extract relevam building block information of a fitness landscape. In this way, an individual is able
to sequentially construct more highly fir partial solutions. For Royal Road Rl, the global oprimum can be
attained easily. For other more complicared fitness landscapes, we allow a number of individuals to adopt the
BBDS and independently evolve in parallel so that each schema region can be given samples indepcndendy.
During this phase, the population is expected to be seeded with promising genetic material. Then follows the
breeding phase, in which individuals are paired for breeding based on rwo mate-selection schemes (Huang,
2001): individuals being assigned mates by natural selection only and individuals being allowed to actively
choose their mares. In the Iauer case, individuals are able to distinguish candidate mates rhar have the same
fitness yet have different string structures, which may lead to quite different performance after crossover.
This is nor achievable by natural selection alone since it assigns individuals of the same fitness the same
probability for being mares, without explicitly raking inro account string suucrures. In short, in the breeding
phase individuals manage to construct even more promising schemata through ilie recombination of highly
fir building blocks found in the first phase. Owing co the characteristic of independent sampling of building
blocks that distinguishes the proposed GAs from conventional GAs, we name this type of GA independent
sampling genetic algorithms (ISGAs).

....
 442 Genetic Algorithm

15.14.5.1 Comparison of /SGA with PGA

The independent sampling phase ofiSGAs is similar m the fine-grained PGAs in the sense that each individual
evolves autonomously, although ISG.As do flO[ adopt the population scrucrure. An initial population is ran-
domly generated. Then in every cycle each individual does local hill climbing, and creates rhe next population
by mating with a parmer in its neighborhood and replacing parents if offsprings are better. By contrast, ISGAs
partition the genetic processing into rwo phases: che independent sampling_ phase and the breeding phase as
described in the preceding section. Third, the approach employed by each individual for improvement in
IS GAs is different from that of the PGAs. During the independent sampling phase of ISGAs, in each cycle,
through the BBDS, each individual attempts to extract relevant informacion of potential building blocks
whenever irs fimess increases. Then, based on the schema information accumulated, individuals continue to
construct more complicated building blocks. However, the individuals of fine-grained PGAs adopt a local hill
climbing algorithm that does not manage to extract relevant information of potential schemata.
The motivation of the two phased ISGAs was partially from the messy genetic algorithms (mGAs). The
rwo stages employed in the mGA.s are "primordial phase" and "juxtaPositional phase," in which the mGAs
first emphasize candidate building blocks based on the guess at the order k of small schemata, then juxtaposing
them to build up global optima in the second phase by "cut" and "splice" operators. However, in the first phase,
the mGAs still adopt centralized selection to emphasize some candidate schemata; this in rum results in the
loss of samples of ocher potentially promising schemata. By contrast, IS GAs manage to postpone the emphasis
of candidate building blocks co the latter stage, and highlight the fearure of independent sampling of building
blocks to suppress hitchhiking in the first phase. As a result, population is more diverse and implicit parallelism
can be fulfiUed to a larger degree. Thereafter, during the second phase, ISGA.s implement population breeding
through rwo mate~selecrion schemes as discussed in the preceding section. In the following subsections, we
present the key componenrs of ISGAs in detail and show the comparisons between the experimental results
of the ISGAs and those of several other GAs on two benchmark test functions.

15.14.5.2 Components of ISGAs

ISGAs are divided into rwo phases: the independent sampling phase and the breeding phase. We describe
;; rhem as follows.

Independent sampling phase: To implement independent sampling of various building blocks, a number
of strings are allowed w evolve in parallel and each individual searches for a possible evolutionary path entirely
independent of others.
In this section, we develop a new searching strategy, BBDS, for each individual to evolve based on the
accumulated knowledge for potentially useful building blocks. The idea is ro allow each individual co probe
valuable information concerning beneficial schemata through resting irs fimess increase since each time a fitness
increase of a string could come from the presence of useful building blocks on it. In short, by systematically
resting each bit to examine whether this bit is associated with che fitness increase during each cycle, a cluster
of bits constituting potentially beneficial schemata will be uncovered. Iterating this process guarantees the
formation oflonger and longer candidate building blocks.
The operation ofBBDS on a string can be described as follows:
1. Generate an empty set for collecting genes of candidate schemata and create an initial string with uniform
probability for each bit until its fitness exceeds 0. (Record the current fitness as Fit.)
2. Except the genes of candidate schemata c;ollecced, from lefr to right, successively all rhe other bits, one at
a time, evaluate the resuhing string. If the resulting fitness is less than Fit, record this bit's position and
original value as a gene of candidate schemata.
 15.14 Classification of Genetic Algorithm 443

3. Except the genes recorded. Randomly generate all the orher bits of the string until the resulting string's
fitness exceeds Fit. Replace Fit by the new fitness.
4. Go to steps 2 and 3 until some end criterion. The idea of this strategy is that the cooperation of certain
genes (bir:s) makes for good fitness.
Once these genes come in sight simultaneously, [hey contribute a fimess increase w the string containing
them; thus any .loss of one of rhese genes leads to che fitness decrease of che string. This is essentially what
step 2 does and after this step we should be able to collect a set of genes of candidate schemata. Then at step 3,
we keep the collected genes of candidate schel)lata fixed and randomly generate other bits, awaiting other

II
building blocks tO appear and bring forth another fitness in crease.
However, step 2 in this strategy only emphasizes the f1mess drop due to a particular bit. It ignores the
possibility that the same bit leads to a new fitness rise because many loci could interact in an exuemely
nonlinear fashion. To rake this into account, the second version ofBBDS is introduced through the change
in seep 2 as follows.
Step 2: Except the genes of candidate schemata collected, from left co right, successively all the other bits,
t one at a time, evaluate the resulting string. If the resulting fimess is less than Fit, record chis bit's position and
original value as a gene of candidate schemata. If the resulting fitness exceeds Fit, substitute this bit's 'new'
value for the old value, replace Fit by this new fitness, record chis bit's posicion and 'new' value as a gene of

I
candidate schemata, andre-execute chis step.
Because this version ofBBDS cakes into consideration the fitness increase resulted from that particular bit,
iris expected to cake less time for detecting. Ocher versions ofRBDS are of course possible. For example, in
step 2, if the same bit resuhs in a fitness increase, ir can be recorded as a gene of candidate schemata, and the
procedure continues to test the residual bits yetwithour completely traveling back to the first bit ro reexamine
each bit. However, rhe empirical results obtained rhus far indicate that the performance of this alternative is
quire similar ro that of the second version. More experimental results are needed ro distinguish the difference
between them.
The overall implementation of the independem sampling phase ofiSGAs is through the proposed BBDS
ro get auronomous evolution of each string until all individuals in rhe population have reached some end
cnrenon.

Breeding phase: After the independent sampling phase, individuals independendy build up their own
evolutionary avenues by various building blocks. Hence rhe population is expected to contain diverse beneficial
schemata and premature convergence is alleviated to some degree. However, factors such as deception and
incompatible schemata (i.e., two schemata have different bit values ar common defining positions) srill could
lead individuals co arrive at suboptimal regions of a fitness landscape. Since building blocks for some strings
ro leave suboptimal regions may be embedded in other srrings, the search for proper maring partners and then
exploiting rhe building blocks on them are critical for overwhelming the difficulty of strings being trapped in
undesired regions. In Huang (2001) the importance of mate selection has been investigated and the results
showed that the GAs is able to improve rheir performance when the individuals are allowed to select maces to
a larger degree.
In this section, we adopt rwo mate-selection schemes analyzed in Huang (2001) w breed the population:
individuals being assigned mates by natural selection only and individuals being allowed co actively choose
their mares. Since natural selection assigns strings of the same fitness the same probability for being parents,
individuals of identical fitness yet distinct string structures are treated equally. This may result in significant
loss of performance improvement after crossover.
We adopt the tournament selection scheme (Mitchell, 1996) as the role of natural selection and the
mechanism for choosing mates in the breeding phase is as follows:
444 Genetic Algorithm

During each mating evem, a binary tournament selection with probabilicy 1.0 is performed to select the
first individual our of the two fittest randomly sampled individuals according to the following schemes:
1. Run the binary tournament selection again to choose the partner.
2. Run another two rimes of the binary tournament selection to choose two highly fit candidate partners;
then the one more dissimilar to the first individual is selected for mating.
The implementation of the breeding phase is through iterating each breeding cycle which consists of
(a) two parents obtained on the basis of the mate~seleccion schemes above. (b) Two-point crossover operator
(crossover rate 1.0) is applied to these parents. (c) Both parents are replaced with both offsprings if any of the
two offsprings is better than them. Then steps (a), (b) and (c) are repeated until ilie population size is reached
and this is a breeding cycle.

I 15.14.6 Real-Coded Genetic Algorithms

The variant of GAs for rea.l~valued optimization that is closest to the original GA are so~called real~coded GAs.
Let us assume that we are dealing with a free N~dimensional real~valued optimization problem, which means
X = RN without constraints. In a real~coded GA, an individual is then represented as an N~dimensional
vector of real numbers:

b = (XJ, .. ,XN)
ru selection does not involve the particular coding, no adaptation needs tO be made- all selection schemes
discussed so far are applicable withour any restriction. What has to be adapted to his special structure are the
genetic oper.uions crossover and mutation.

15.14.6.1 Crossover Operators for Real-Coded GAs

So far, the following crossover schemes are most common for reakoded GAs:
Flat crossover: Given two parents b1 = (x~, ... , x~) and f? = (x?, ... , xJv), a vector of random values from
the unit interval (AJ , ... , AN) is chosen and the offspring b = (x{, ... , xfv) is computed as a vector oflinear
combinations in the following way (for all i = 1, ... , N):

~ =Ai x} + (1-A,) xl
BLX-a crossover is an extension of flat crossover, which allows an offspring allele ro be also located outside
the interval
[min(x}, x?), max(x,!, x?)]
In BlX~acrossover, each offspring allele is chosen as a uniformly disuibuted random value from the imerval
[min(x} ,x[) - fa, max(x}, xl) + f.ct]
where l = max(x},xf)- min(x},xj). The parameter a has to be chosen in advance. For a= 0, BLX~a
crossover becomes identical to flat crossover.

Simple crossover is nothing else bur classical onepoint crossover for real vectors, i.e., a crossover site
k E 2{ 1, ... , N- 1} is chosen and cwo offspring are created in the following way:
bl -- ( 1 I 2 2)
XJxk,xk-HxN
II'= (x? ... ,xf,xl+ 1 , ,x~)
15.15 Holland Classifier Systems 445

Discrete crossover is analogous to classical uniform crossover for real vectors. An offspring b of rhe two parents
b1 and fJ is composed from alleles, which are randomly chosen either as x} or x[.

15.14.6.2 Mutation Operators for Real-Coded GAs

The following mutation operators are most common fo.r real-coded GAs:
1. Random mutation: For a randomly chosen gene i of an individual b = (xl, ... , XN), the allele x; is replaced
by a randomly chosen value from a predefined interval Ia, b,].
2. Nonuniform mutation: In nonuniform murarion, the possible impact of mutation decreases with the
number of generations. Assume that tm4X is the predefined maximum number of generations. Then, with
the same setup as in random mumion, rhe allele x 1 is replaced by one of the rwo values

~ = x1+A (t,b;- x1)

:if= x;-A (r,x;- a;)
The choice as to which of the two is taken is determined by a random experiment with two outcomes rhat
have equal probabilities 1/2 and I /2. The random variable A (t, x) determines a mutation step from the range
]0, xl in the following way:

D. (t,x) = x(J-),IHdmIJ')

In this formula, A is a uniformly distributed random value from the unit interval. The parameter r
determines the influence of rhe generation index ton the disrribution of mutation step sizes over the imerval
IO,xl.

115.15 Holland Classifier Systems

A Holland classifier system is a classifier system of the Michigan type which processes binary messages of a
fixed length through a rule base whose rules are adapted according to response of the environment.

I 15.15.1 The Production System

Fim of all, rhe communication of rhe production system with the environment is done via an arbitrarily long
list of messages. The derecrors translate responses from the environmem imo binary messages and place them
l)n the message list which is then scanned and changed by the rule base. Finally, the effectors translate output
messages imo actions on the environment, such as forces or movements.
Messages are binaty strings of the same length k. More formally, a message belongs w {0, l}k. The rule
base consists of a fixed number (m) of rules (classifiers) which consist of a fixed number (r) of conditions and
an acrion, where both conditions and actions are strings oflength k over the alphabet {0, 1, *).The asterisk
plays the role of a wildcard, a 'don't care' symbol.
A condition is matched if and only if there is a message in the list which matches the condition in all
nonwildcard positions. Moreover, conditions, except the first one, may be negated by adding a'-' prefix. Such
a prefixed condition is satisfied if and only if there is no message in the list which marches the string associated
with the condition. Finally, a rule fires if and only if all the conditions are satisfied, i.e., the conditions are
connected wirh AND. Such 'firing' rules compere to put their action messages on the message list.
In the action pans, the wildcard symbols have a different meaning. They take rhe role of 'pass through'
elemem. The outpm message of a firing rule, whose action parr contains a wildcard, is composed from the
 446 Genetic Algorithm

nonwildcard positions of the action and the message which satisfies the fim condition of the classifier. This is
actually the reason why !legations of the first conditions are not allowed. More formally; the outgoing message
mis defined as
a[i] ifa[i];O. .
(!!= { ['] if['] =l, ... ,k
m' a' = *
where a is the action part of the classifier and m is the.(Ilessage which matches the first condition. Formally,
a classifier is a suing of the form

Cond,, 1'-' IICond,, ... , 1'-'Cond,/Acrion

where the brackets shouJd express the optionalicy of the "-" prefixes. Depending on the concrete nee; of the
task to be solved, it may be desirable ro allow messages to be preserved for the next step. More specifically, if
a message is not interpreted and removed by the effectors interface, it can make another classifier fire in the
next step. In practical applications, this is usually accomplished by reserving a few birs of the messages for
identifying the origin of the messages (a kind of variable index called tag).
Tagging offers new opportunities to transfer information about the current step into rhe next step simply
~ by placing ragged messages on the list, which are not interpreted, by the output interface. These messages,
which obviously contain information about the previous step, can support the decisions in the next step.
Hence, appropriate use of rags permits rules to be coupled to act sequenrially. In some sense, such messages
are rhe memory of the system.
A single execmion cycle of the production system consists of the following steps:

1. Messages from the environment are appended to rhe message list.

2. A!! the conditions of all classifiers are checked against rhe message list w obtain the set of firing rules.
3. The message list is erased.
-.' .
~
/

,,;
4. The firing classifiers participate in a competition ro place their messages on rhe list.
::~ / 5. The winning classifiers place their actions on the list.
6. The messages directed to the effectors are executed.
This procedure is repeated iteratively. How step 6 is done, if these messages are deleted or nor, and so on,
depends on ilie concrete implementation. It is, on rhe one hand, possible to choose a representation such that
the effectors can interpret each output message. On the other hand, it is possible to direct messages explicitly
to ilie effectors with a special tag. If no messages are directed to ilie effectors, ilie system is in a iliinking phase.
A classifier Rl is called consumer of a classifier R2 if and only if there is a message mO which fulfills at
least one ofRl's conditions and has been placed on rhe list by R2. Conversely, R2 is called a supplier ofRl.

I 15.15.2 The Bucket Brigade Algorithm

As already mentioned, in each rime step t, we assign a suengrh value u;, 1 to each classifier Ri. This strength
value represents ilie correctness and importance of a classifier. On ilie one hand, the strengrh value influences
ilie chance of a classifier to place its action on the output list. On the oilier hand, the suength values are used
by the rule discovery system, which we will soon discuss.
In Holland classifier systems, the adaptation of the strength values depending on the feedback (payoff)
&om the environment is done by the so.called bucket brigade algorithm. It can be regarded as a simulared
15.15 Holland Classifier Systems 447

economic system in which various agents, here the classifiers, participate in an auction, where the chance to
buy rhe right to post the action depends on the strength of the agents.
The bid of classifier Ri at timet is defined as

B;,, = CLrJ;,,S;

where CL E [0, 1] is a learning parameter, similar to learning rates in anificial neural nets, and s,- is the
specificity, the number of nonwildcard symbols in the condition pan of the classifier. If CL is chosen small,
the system adapts slowly. If it is chosen too high, the strengths rend to oscillate chaotically. Then the rules
have ro compete for the right for placing ilieir"output messages on the list. In the simplest case, this can be
done by a random experiment like the selection in a genetic algorithm. For ~i:h bidding classifier it is decided
randomly if it wins or not, where the probability that it wins is proportional to its bid:

B;,,
P[R; wins]= " B
L.., ").1
jES3{i

In rhis equation, Sar1 is the set of indices of all classifiers which are satisfied at rime t. Classifiers which get
the right to post their output messages are called winning classifiers.
Obviously, in this approach more than one winning classifier is allowed. C f course, or her selection schemes
are reasonable, for instance, the highest bidding agent wins alone. This is necessary to avoid ilie conflict
between two winning classifiers. Now let us discuss how payoff from the environment is disrribmed and how
the strengths are adapted. For this purpose, let us denme rhe set of classifiers, which have supplied a winning
~gent R; in step t with 5;, 1 Then the new strength of a winning agent is reduced by irs bid and increased by
its portion of the payoff P1 received &om the environment:

P,
fli,t+l = lli,r +-
w,
~ B;,r

where w1 is the number of winning agents in the actual time step. A winning agent pays its bid to its suppliers
which share the bid among each other equally in the simplest case:

B,-,,
lli,r+l = u;, 1 + - I for all R; E 5;, 1
15
"'
If a winning agem has also been active in the previous step and supplies another winning agent, the value
above is additionally increased by one portion of the bid the consumer offers. In the case that two winning
agents have supplied each orher mutually, the portions of the bids are exchanged in rhe above manner. The
SHengrhs of all orher classifiers Rm which are neither winning agents nor suppliers of winning agents, are
reduced by a certain factor (they pay a rax):

u,,r+I = u,,r(l ~ T)

Tis a small value lying in the interval [0, 1]. The intention of taxation is to punish classifiers which never
contribute anything to the outpurof rhesystem. With this concept, redundant classifiers, which never become
active, can be filtered out.
The idea behind credit assignment in general and bucket brigade in particular is w increase the strengths
of rules, which have ser the stage for later successful actions. The problem of determining such classifiers,
which were responsible for conditions under which it was later on possible ro receive a high payoff, can be
 448 Genetic Algorithm

-
;t execution
60 I+- Payoff
'20 ~ '20 ~ '20 f.- '20 f.- -
-- - F r-----
80 80 80 80 80

Strengths 100 100 100 100 140

Payoff
Second execution

80 80 80 80 112

Strengths 100 100 100 108 172

Figure 1546 The bucker brigade principle.

very difficuh. Consider, for instance, the game of chess again, in which very early moves can be significant
for a late success or failure. In fact, the bucker brigade algorithm can solve this problem, although strength
is only transferred tO the suppliers, which were active in rhe previous step. Each rime the same sequence is
activated, however, a little bir of the payoff is transferred one step back in rhe sequence. It is easy ro see that
repeated successful execution of a sequence increases the mengrhs of all involved classifiers.
Figure 15-46 shows a simple example of how rhe bucker brigade algorithm works. For simpliciry, we
consider a sequence of five classifiers which always bid 20o/o of their strength. Only after the fifth step, after
the activation of the fifth classifier, a payoff of 60 is received. The further development of the strengths in this
example is shown in the Table lS-7. It is easy to see from this example that the reinforcemenr of the strengths
is slow at the beginning, but it accelerates later. Exaccly this properry contributes much to the robustness
of classifier systems- they tend to be cautious at the beginning, trying not to rush conclusions, but, after a
certain number of similar siruations, the system adopts the rules more and more.
It might be clear that a Holland classifier system only works if successful sequences of classifier activations
are observed sufficiently often. Otherwise the bucket brigade algorithm does not have a chance to reinforce
the strengths of the successful sequence properly.

I 15.15.3 Rule Generation

The purpose of the rule discovery system is to eliminate low-firred rules and to replace them by hopefully
..._ better ones. The fitness of a rule is simply irs strength. Since the classifiers of a Holland classifier system
themselves are strings, the application of a GA to the problem of rule induction is straightforward, though
many variants are reasonable. Almost all variants have one thing in common: the GA is nor invoked in each
time step, but only every nth step, where 11 has to be set such that enough information about the performance
15.16 Genetic Programming 449
Table 157 An example for repeated propag:j.rion of payoffs
Strength after the
3rd 100.00. 100.00 101.60 120.80 172.00
4rh 10o.oo 1oo:32 !03.44 136.16 197.60
5rh 100.06 101.34 111.58 152.54 234.46
6rh 100.32 103.39 119.78 168.93 247.57

tom ta6.56 124.17 I64 .44 224.84 2 78.52

25rh 215.86 253.20 280.36 294.52 299.24

execurion of rhe sequence

of new classifiers can be obtained in the meantime. A. Geyer-Schuh., for instance, suggests the following
procedure, where the strength of new classifiers is initialized with the average strength of rhe current rule base:

I. Select a subpopulation of a certain size at random.

2. Compute a new set of rules by applying the generic operations- selection, crossingover and muration -
to this subpopularion.
3. Merge the new sub population with the rule base omitting duplicates and replace the worst classifiers.

This process of acquiring new rules has an interesting sideffecr. Iris more rhan just the exchange of pans of
conditions and actions. Since we have nor stared restrictions for manipulating rags, the GA can recombine
parts of already existing rags m invent new tags. In the following. rags spawn related rags establishing new
couplings. These new tags survive if rhey conrribure ro useful interactions. In this sense, the GA additionally
creates experience-based internal structures autonomously.

115.16 Genetic Programming

Genetic programming (GP) is also part of rhe growing set of evolutionary algorithms rhar apply the search
principles of natural evolution in a variety of differem problem domains, notably parameter optimization.
Evolutionary algorithms, and GP in particular, follow Danvin's principle of differential natural selection.
This principle states that the follow"ing preconditions must be fulfilled for evolution to occur via (natural)
selection:

I. There are entities called individuals which form a population. These entities can reproduce or can be
reproduced.
2. There is herediry in reproduction, rhat is to say that individuals produce similar offspring.
3. In the course of reproduction, there is variery which affects the likelihood of survival and therefore of
reproducibility of individuals.
4. There are finite resources which cause the individuals to compete. Owing ro over reproduction of indi-
viduals nor all can survive the struggle for cx..isrence. Differential natural selections will exert a continuous
pressure towards improved individuals.
450 Genetic Algorithm

In rhe long run, GP and other .evolutionary computing technologies will revolutionize program devel~
opmem. Present methods are not mamre enough for deploymem as auromatic programming systems.
Nevertheless, GP has already made inroads imo automatic programming and will continue ro do so in
the foreseeable fmure. Likewise, the application of evolu(ion in machine-learning problems is one of rhe
potentials we will exploit over rhe coming decade.
GP is parr of a more general Held known as evolutionary computation. Evolutionary computation is based
on the idea that basic concepts of biological reproduction and evolution can serve as a metaphor on which
computer-based, goal-directed problem solving can be based. The general idea is that a computer program can
maintain a population of artifacts represented using some suitable computer-based data structures. Elements
of that population can then mare, mutate, or otherwise reproduce and evolve, directed by a fimess measure
that assesses the quality of the population with respect ro the goal of the rask at hand.
GP is an automated method for creating a working computer program from a high-level problem statement
of a problem. GP starts from a high-level statement of 'what needs to be done' and auromarically creates a
computer program to solve the problem.
One of the central challenges of computer science is to get a computer to do what needs to be done,
without telling it how to do it. GP addresses this challenge by providing a method for automatically creating
a working compmer program from a high-level problem statement of the problem. GP achieves this goal of
azttomatic programming (also sometimes called program synthesis or program induction) by generically breeding a
population of computer programs using the principles of Darwinian natural selection and biologically inspired
operations. The operations include reproduction, crossover, mutation and architecture-altering operations
patterned after gene duplication and gene deletion in nature.
GP is a domain-independent method rhar generically breeds a population of computer programs to solve
a problem. Specifically, GP iteratively transforms a population of computer programs into a new generation
of programs by applying analogs of naturally occurring generic operations. The generic operations include
crossover, mutation, reproduction, gene duplication and gene deletion. GP is an excellent problem solver,
a superb function approximator and an effective tool for writing functions to solve specific tasks. However,
despite all these areas in which it excels, it still does not replace programmers; rather, it helps them. A human
still must specify the fitness function and identify the problem to which GP should be applied.

I 15.16.1 Working of Genetic Programming

GP typically startS with a population of randomly generated com purer programs composed of the available
programmatic ingredients. GP iteratively transforms a population of computer programs into a new generation
of rhe population by applying analogs of naturally occurring genetic operations. These operations are applied
to individual(s) selected from the population. The individuals are probabilisrically selected to participate in
rhe genetic operations based on their fitness (as measured by the f1tness measure provided by the human user
in rhe third prepararory step). The iterative transformation of the population is executed inside the main
generational loop of the run of G P.
The executional steps of GP (i.e., the flowchart of GP) are as follows;
1. Randomly create an initial population (generation 0) of individual computer programs composed of the
available functions and terminals.
2. Iteratively perform the following subsreps (called a genemtion) on the population until the termination
criterion is satisfied:
Execute each program in the population and ascertain its fitness (explicitly or implicirly) using the
problem's fitness measure.
15.16 Genetic Programming 451

Select one or rwo individual program(s) from the population with a probability based on fitness (with
reselecrion allowed) to participate in the generic operations in rhe next subsrep.
Create new individual program(s) for the populaiion by applying the following generic operations with
specified probabilities:
(a) Reproduction: Copy the selected individual program to the new population.
(b) Crossover:. Create new offspring program(s) for the new population by recombining randomly
chosen pam from rwo selected programs.
(c) Mutation: Create one new offspring program for the new population by randomly mutating a
randomly chosen part of one selected program.
(d) Archirecrure-alren"ngoperatiom-. Choose an architecture~alteringoperarion from the available reper-
toire of such operations and create one new offspring program for the new population by applying
the chosen architecture-altering operation to one selected program.
3. After the termination criterion is satisfied, the single best program in the population produced during the
run (the besr-so-far individual) is harvested and designated as rhe result of rhe run. If the run is successful.
the result may be a solution (or approximate solution) to the problem.
GP is problem-independent in the sense that the flowchart specifying the basicsequenceofexecmional steps
is nor modified for each new run or each new problem. There is usually no discretionary human intervention
or interaction during a run of generic programming (although a human user may exercise judgment as to
whether to terminate a run).
Figure 15-47 below is a flowchart showing the executional steps of a run ofGP. The flowchart shows the
generic operations of crossover, reproduction and mutation as well as the architecrure~alrering operations.
This flowchart shows a two-offspring version of the crossover operation.
The flowchart of GP is explained as follows: GP stans with an initial population of com purer programs
composed of functions and terminals appropriate to the problem. The individual programs in rhc initial
population are typically generated by recursively generating a roared point-labeled program tree composed of
random choices of rhe primitive functions and terminals (provided by rhe human user as part of the first and
second preparatory steps, a run ofGP). The initial individuals are usually generated subject to a pre-established
maximum size (specified by the user as a minor parameter as pan of rhe founh preparatory step}. In general,
the programs in the population are of different sizes (number of functions and terminals) and of differenr
shapes (the particular graphical arrangement of functions and terminals in the program tree).
Each individual program in the population is executed. Then, each individual program in the population
is either measured or compared in rerms of how well it performs the task at hand (using the fitness measure
provided in the third preparatory step). For many problems, this measurement yields a single explicit numerical
value called fitness. The fitness of a program may be measured in many different ways, including, for example,
in terms of the amount of error between its output and the desired output, the amount of rime (fuel, money,
etc.) required to bring a system to a desired target stare, the accuracy of the program in recognizing patterns
or classifying objects into classes, the payoff that a game-playing program produces, or rhe compliance of a
complex structure (such as an antenna, circuit, or controller) with user-specifted design criteria. The execution
of the program sometimes returns one or more explicit vaJues. Alternatively, the execution of a program may
consist only of side effecrs on rhc stare of a world (e.g., a robot's actions). Alternatively, the execution of a
program may produce both return values and side effects.
The fitness measure is, for many practical problems, mulriobjecrive in rhe sense that it combines rwo or
more differem elements. The different elements of the fitness measure are often in competition with one
another to some degree.
452 Genetic Algorithm

~
No'>
Yes
' --
I Run,=O H Gen,=O f-- Create initial random
population for run
( Run-N? )~I Run:- Run+ 1 I
Termination criterion
salislied for run?

~ No
Apply fitness measure to every individual in the population

No

Gen := Gen + 1

No
( Select Genetic Operation )

Select one individual Copy into new

based on litness Perform reproduction
population
Insert offspring
Select two individuals
based on fitness into new
population

Select one individual Insert mutant into

Perform mutation
based on fitness new population

Select an architecture altering operation

based on its specified probability

Perform the Insert offspring into

based on fitness architecture altering new population
operation

Figure 15-47 Flowchart of gene[ic programming.

For many problems, each program in the population is executed over a representative sample of different
fituess cases. These fitness cases may represent different values of the program's inpur(s), differem initial
conditions of a system, or different environments. Sometimes the firness cases are constructed probabilisticaHy.
The creation of the initial random population is, in effect, a blind random search of the search space of
the problem. It provides a baseline for judging future search effons. Typically, the individual programs
in generation 0 all have exceedingly poor fitness. Nevertheless, some individuals in the population are
{usually) more fir than odters. The difference.~ in fitness are dten exploited by GP. GP applies Darwinian
selection and che generic operations co create a new population of offspring programs from the current
population.
The generic operations include crossover, mutation, reproduction and the architecture-altering operations.
These generic operations are applied to individual(s) that are probabilistically selected from the population
based on fitness. In this probabilistic selection process, better individuals are favored over inferior individuals.
15.16 Genetic Programming 453

However, the best individual in the population is not necessarily selected and rhe worst individual in the
population is not necessarily passed over.
After the generic operations arc performed on the current population, the population of offspring (i.e.,
the new generation) replaces the current population {i.e., rhe now-old generation). This iterative process of
measuring fimess and performing the genetic operations is re~eated over many generations.
The run of GP terminates when the termination criterion (as provided by the fifth preparatory step) is
satisfied. The outcome of the run is specified by the method of result designation. The best individual ever
encountered during the run (i.e., rhe best-so-far individual) is rypically designated as rhe result of the run.
All programs in the initial random population {generation 0) of a run of GP are symacrically valid,
executable programs. The generic operations that are performed during the run (i.e., crossover. mutation,
reproduction and rhe architecture-altering operations) are designed to produce offspring that art: synracrically
valid, executable programs. Thus, ever~ individual created during a run of genetic programming (including,
in pmicular, the best-of-run individual) is'' synracrically valid, executable program.

15.16.2 Characteristics of Genetic Programming

GP now routinely delivers high-return human-compeririv[: machine inrelligence, the next four subsections
explain what we mean by the terms human-competitive, high-return, routine and machine intelligence.

15.16.2.1 Human-Competitive
In attempting ro evaluate an automated problem-solving method, the question arises as ro wherhcr there
is any rl"a] substance to rhe demonstr;Hive probl!"ms rhar are published in connection wirh the method.
Demonstrative problems in the fields of anificial intelligence and machine learning are often conuived ro
problems that circulate exclusively inside academic groups that study a particular methodology. These problems
typically have little relevance to any issues pursued by any scienrist or engineer outside the fields of :mitlcial
intelligence and machine learning.
ln his 1983 talk enrided "A!: \'(Ibm' lr Hm Been and Whert !tIs Going," machine learning pionl"er Anhur
Samuel said:
1/Jt ,1i111 i.( . 10 ger mathhw ro exhibir hchwior, which ifdone by lmmt/W, 1/!0IIId be IIWmud 10 in!!oh'e rh.:
usc oj"inrdligenu.
S,unuel., statement retlecrs rhe common go:tl olrriculatcd by the pio11cers of rhe J 950s in rhc f1elds of ;Lrtitlcial
intelligence and machine learning. Indeed, getting machines ro produce human-like results is rhc reason for
the exisrcnce of the fields of anificioll inrelligence and machine learning. To make chis goal more concrete, we
say th;It a result is "human-comperirive" if it satisfies one or more of the eight criteria in Table 15-8. These
eight criteria have rhe desirable attribute of being at arms-length from rhe fields of artificial intelligence,
machine learning :tnd GP. That is, a rc~ult cannot acquire rhe raring of'human-comperitive' merely because
it is endorsed by researchers imide rhe specialized fields rhat are arrempting to create machine intelligence.
lnsread, a result produced by an auromared method must earn the raring of human-competitive independem
of the tact rhat it wa.~ generated by an automared method.

15. 16.2.2 High-Return

What is delivered by the acru:tl auronuted operat!on of an artificial method in comparison to rhe amounr of
knowledge, information, analysis and intelligence char is pre-supplied by the human employing the method?
We define rhe AI ratio (the 'arrificial-ro-inrelligence' ratio) of a problem-solving method as the ratio of
~har which is delivered by the auromated operation of the artificial method to the amounr of intelligence rim
lS supplied by the human applying the method to a particular problem.
454 Genetic Algorithm

Table 158 Eight criteria for saying that an automatically created resuh is human-competitive
Criterion
A The rcsulr was patented as an invemion in the past, is an improvement over a parented invention, or would
quality roday as a patemable new invention.
B The result is equal to or beuer than a result rhar was accepted as a new scientific resuh ar rhe rime when it
was published in a peer-reviewed sciemific journal.
C The resulr is equal to or better rhan a result that was placed into a darabase or archive of results maintained
by an inrernarionally recognized panel of scientific experrs.
D The resulr is publishable in its own right as a new scienrific resulr-independent of the facr rhar rhe result
was mechanicallv created.
E The resulr is eq~al w or bencr than rhe most recem human-created solurion ro a long-standing problem for
which there has been a succession of increasingly better human-created solutions.
F The result is equal to or better than a resuh that was considered an achievement in irs field at the rime it was
first discovered.
G The resulr solves a problem of indisputable difficulty in irs field.
H The result holds its own or wins a regulated competition involving human contestants (in the form of either
live human players or human-wriuen computer programs).

The AI ratio is especially pertinent to methods for getting computers to automatically solve problems
because it measures the value added by the artificial problem-solving method. Manifestly, rhe aim of the fields
of artificial intelligence and machine learning is to generate human-competitive results with a high AI ratio.
Deep Blue: An Arnficin/ lme//igence Milestone (Newborn, 2002) describes the 1997 defeat of the human
world chess champion Garry Kasparov by the Deep Blue computer system. This oumanding example of
machine imdligence is clearly a human-competitive result (by virtue of satisfying criterion H of Table 15-8).
Feng-Hsiung Hsu (the system architect and chip designer for the Deep Blue project) recounts the intensive
work on the Deep Blue project at IBM's T. J. Watson Research Center between 1989 and 1997 {Hsu, 2002).
The team of scientists and engineers spent years developing the software and the specialized computer chips
to efficiently evaluate large numbers of alternative moves as parr of a massive parallel state-space search. In
short, rhe human developers invested an enormous amount of"!" in the project. In spire of rhe fact rhat Deep
Blue delivered a high {human-competitive) amount of "A," the project has a low rerurn when measured in
terms of rheA-to-! ratio.
The aim of rhe fields of artificial intelligence and machine learning is ro get computers to automatically
generate human-competitive results with a high AI ratio- not to have humans generate human-competitive
results themselves.

15.16.2.3 Routine
Generality is a precondition ro what we mean when we say that an automated problem-solving method is
"romine." Once the generality of a method is established, "routineness" means cllar relatively little human
effon is required to get rhe method to successfully handle new problems within a particular domain and to
successfully handle new problems from a different domain. The ease of making rhe transition to new problems
lies at the hearr of what we mean by routine. A problem-solving method cannot be considered routine if iS
executional steps must be substantially augmented, deleted, rearranged, reworked or customized by the human
user for each new problem.
15. 16.2.4 Machine Intelligence
We use cll,e term machine it;ttelligence to refer to the broad vision articulated in AJan Turing's 1948 paper
emided "fnu:!iigent Machinery" and his 1950 paper entitled "Computing Machinery and Intelligence."
 15.16 Genetic Programming
455

In rhe 1950s, the terms machine intelligence, artificial intelligence and machine learning all referred to the
goal of getting "machines to exhibit behavior, which if done by humans, would be assumed to involve the use
of intelligence" {to again quote Arthur Samuel).
However, in the intervening five decades, the terms "arrif!Cial intelligence" and "machine learning" pro-
gressively diverged from their original goal-oriented meaning. These terms are now primarily associated with
particular methodologies for attempting to achieve the goal of getting computers ro automatically solve prob-
lems. Thus, the term "artificial intelligence" is today primarily associated with attempts to get computers
ro solve problems using methods that rely on knowledge, logic, and various analylical and mathematical
methods. The term "machine learning" is today primarily associated with attempts to get computers to solve
problems that use a particular small and somewhat arbitrarily chosen set of methodologies (many of which
are statistical in nature). The narrowing of these terms is in marked contrast to the broad field envisioned
by Samuel at rhe time when he coined the term "machine learning" in rhe 1950s, the charter of the original
founders of rhe field of artificial imdligence, and the broad vision encompassed by Turing's term "machine
intelligence." Of course, the shift in focus from broad goals to narrow methodologies is an all roo common
sociological phenomenon in academic research.
Turing's term "machine intelligence" did not undergo this arteriosclerosis because, by accident of history, it
was never appropriated or monopolized by any group of academic researchers whose primary dedication is to
a particular methodological approach. Thus, Turing's term remains catholic today. We prefer to use Turing's
term because it still communicates the broad goal of getting computers ro automatically solve problems in a
human-like way. ,
In his 1948 paper, Turing identified three broad approaches by which human competiti\'e machine inrel-
ligencc might be achieved: The first approach was a logic-driven search. Turing's interest in rhis approach is
nor surprising in lighr ofTuring's own pioneering work in rhe 1930s on the logical foundations of computing.
The second approach for achieving machine intelligence was what he called a "cultural search" in which
previous]~, acquired knowledge is accumulated, stored in libraries and brought to bear in solving a problem
-the approach taken by modern knowledge-based expert systems. Turing's first two approaches have been
pursued over the pasl 50 years by the \'aSt majority of restarchers using the methodologies that are roday
primarily associated with the term "arrificial imelligence.''

I 15.16.3 Data Representation

Without any doubt, programs can be considered :IS strings. There are, however, rwo important limiwrions
which make it impossible to use the representations and operations from our simple GA:
l. It is mosrly inappropriate to assume a fixed length of programs.
2. The probability to obtain syntactically correct programs when applying our simple initialization. crossover
and mutation procedures is hopelessly low.
lr is, therefore, indispensable to modify the data representation and the operations such that syntactical
correctness is easier to guarantee. The common approach tu represent programs in GP is to consider programs
as trees. By doing so, initialization can be done recursively, crossover can be done by exchanging subtrees and
random replacement of subtrees can seJVe as mutation operation.
Since their only construct are nested lists. programs in LISP-like languages already have a kind of tree-like
Structure. Figure 15-48 shows an example how the function 3x + sin(x + I) can be implemented in a LISP-
like language and how such an LISP-like Fnncrion can he split up inro a tree. lr can be nored that the tree
n:presenration corresponds to the nesred lists. The program consists of ramie expressions, like variables and
constants, which act as leave nodes while Functions acr as nonleavc nodes .

.1.I.
456 Genetic Algorithm
1

Figure 15-48 The tree representation of3x+ sin(x + 1).

There is one importanr disadvanrage of rhe LISP approach-iris diflicult ro inrroduce rype checking. In
case of a purely numeric function like in rhe above example, there is no problem at all. However, ir can be
desirable to process numeric clara, .mings and logical expressions simulraneously. This is difficult ro handle if
we use a rree representation like rhar in Figure 15~48.
A. Geyer-Schulz bas proposed a very general approach, which overcomes rhis problem allowing maximum
flexibility. He suggested representing programs by rheir synractical derivation trees wirh respea to a recursive
'definition of underlying language in Backus-Naur form (BNF). This works for any context-free language. h
is far beyond the scope of this lecmre to go into much derail about formal languages. We will explain the
basics with rhe hdp of a simple example. Consider the following language which is suitable for implementing
binary logical exp~essions:

S :=: <exp>
<exp> :== (var) I "("<neg> <exp>")" I "("<exp> <bin> <exp>")";
<v~lr> := '"x" l'"r'";
<neg> :="NOT"
<bin> :="AND" I "OR":

The BNF description wns'1sts of so-called syntacrica.l rules. Symbols in angular brackets < > are called
nomerminal symbols, i.e .. symbols which have to be expanded. Symbols between quotation marks are called
terminal symbols, i.e., they cannot be expanded any further. The first rule S:=<exp> defines rhe staning
symbol. A BNF rule of the gener;:~l shape,
\
<nonrerminab := <deriv, >I <deriv2> <deriv 11 >;
1 ... 1
i
defines how a non-terminal symbol may be expanded, where the differem variams are separated by vertical I
bars.
In order to get a feeling of how lO work wirh rhe BNF grammar description, we will now show step-by-step
how rhe expression (NOT (x OR y)) can be derivared from the above language. For simplicity, we omir
q_uotadon marks for the terminal symbols:
I. We have ro begin with rhe start symbol: <exp>
2. We replace hexpi wirh rhe second possible derivation_:
.

l'
<exp> --i- (<neg> <exp>)
 15.16 Genetic Programming 457

3. The symbol <neg> may only he expanded with the terminal symbol NOT:

{<neg> <t:xp>) - (NOT <c:xp> i

4. Next. we replace: <exp> with the rhird possible derivar-ion:

(NOT <exp>)-.. (NOT {<exp> <bin> <exp>))

S. We exp~md rhe second pmsihle deri1'arion f(lr <bin>:

(NOT l<exp::o .:bin> <~xp>))-. (NOT (<exp> OR <exp>))

6. Tht tirst ou.urrc:n<.:c of <exp.> i~ expanded wirh rhe first derivation:

tNOT(<exp> OR -:.:exp>J)- (NOT (<var> OR <exp>)J

The .,tcond l)o..<.:urrencc of <exp> i~ expanded with rhc fiN derivation. ro11:

{~( lT! <v;tr> OR <txp> )) -. (NOT ( <v<~r'> OR <var> l)

l
H. )\;ow ,,.l. LtpLh:e thl fiN <lar> wirh the wrre~ponding, tlr~t ;thernari1e:
j
9. hna!lv. rlll' b~t
!NOT ( <lar> ( lR qa1

non-terminal 1ymbol i~
> )) ---- (NOT t.d)R <var> ))

cxpamkd wid1 rhc: second .than;Hive: i

(~()Tt.d)l\ ~-~ar:.>)) --+ (J\'(lT!xlli{y)l

Such a recursive derivation ha~ an inhr:rent nee structure. hn tht ;~bo1e cx:~mple, this derivation tree
h.1~ hecn l'isualized in hgure l )-49. The -~~max of modern progr~unming Llllguages can be specified in
HNE Hcnct. <llll' data moJd would he ;Lpplicahlc to all of them. rhe quesrioli is whether rhis is useful. ...i
K1lZ,t's hypothesis include~ d1;1t rhe ;Hogr:unmin~ hnguag.e h;ts m he dwstn such rhar rhe given problem
i., ~oktble. Thi~ does tHH neccssMil~- imply dut we h,tlc w . .:home: rhc: language such rhar virtually any
~olvablt prohkm com he: .~olvcd. h i~ o!wiou., that rhc si-,T of thl lt<lrch space ~rows with the complexity of

}

the l.tn!!.ua~c. \X'e know thou the ~i1x of rhl 'l"Mch 'pace inthrence.' the performance of ;t GA- the larger rhe
\1111\'C:r.

lt is. rherdOre, recommendahte w rt~rrio rlw bnguagl w nt:n~-~ar~- . :omrntcr~ and to avoid superfluous
lon.,rrucr~.Assume. t;1r example. rhon we want ro do symbolic regre~_,i,Jn. bur we .tre only interested in
polynomials with integer coe~l'icic:nrs. hn ~uch ;Ill applicarion. it would he ~m mc:rkill to introduce rational
constants or w indude exponenriJ.l function~ in rhc- languagt. A g;ood choi<.:e muld be the following:

.'-! := <func:.-:
.. fun<.:> := <var> I "('.<coLI.\t> I '"("'<tirnc> -chin> <func>"')":
.-.var> := "'x":
<cnmt> := <inr> I <const> <int>:
.c:im> := "'0'"] ... 1 "9":
<hin> ::o "+"'1"-"1"+"';

For repre~eming rational funcrions with imq,~,r codficients, ir is sufficienr ro add the division symbol "/"

~"~""""'""""'"~ .~.
 458 Genetic Algorithm

<BXp:>

2nd of 3 possible derivations

3rd of 3 possible derivations

Figure 1549 The derivation uee of (NOT (x ORy)).

Anorher example: The following language could be appropriate for discovering trigonometric identities:
'-
S := <func>;
<func> := <var> I <canst> I <trig> "("<func>")" I "("<func> <bin> <func>")"
<var> := "x";
<canst>:= "0" I "I" I "rr";
<trig> :="sin" I "cos";
<bin> :="+"["-"["+";

~:
There are basically two different variants of how w generate random programs with respect to a given BNF
grammar:

l. Beginning from the starring symbol, it is possible to expand nonterminal symbols recursively, where we
have to choose randomly if we have more than one alternative derivation. This approach is simple and
fast, but has some disadvantages: First, it is almost impossible to realize a uniform distribution. Second,
one has to implement some constraints with respect ro the depth of the derivation trees in order to avoid
excessive growth of the programs. Depending on the complexity of the underlying grammar, this can be
a tedious task.
2. Geyer~Schulz has suggested to prepare a list of all possible derivation trees up to a certain depth and to
select from this list randomly applying a uniform distribution. Obviously, in this approach, the problems
in terms of depth and the resulting probability dimibmion are elegantly solved, but these advantages go
along with considerably long computation rimes.

15. 16.3. 1 Crossing Programs

It is trivial to see that primitive string-based crossover of programs almost never yields syntactically correct
program~. Instead, we should use the perfea syntax information a derivation tree provides. Already in ilie USP
times ofGp, sometime before the BNF-based repr~entation was known, crossover was usually implemented

l
 15.16 Genetic Programming 459

as the exchange of randomly selected subtrees. In case that the subtrees (subexpressions) may have different
types of return values (e.g., logical and numerical), it is not guarameed iliar crossover preserves syntactical
correctness.
The derivation tree~based representation overcomes this p~oblem in a very elegamway. If we only exchange
subtrees which start from the same nomerminal symbol, croSsover can never violate syntactical correctness. In
this sense, the derivation tree model provides implicit type checking. In order to demonstrate in more detail
how this crossover operation works, let us reconsider rhe example of binary logical expressions. k parents,
we take rhe following expressions:

(NOT (xORy))
((NOT x) OR (x ANDy))

Figure l 5-50 shows graphically how the two children (NOT (xOR (x ANDy))) ((NOT x) ORy) are obtained.

c _ _ _ _ __ _ _ _ _j

l Figure 15-50 An rxample for crossing two binary logical expressions.

 "I
460 Genetic Algomhm
!
I

Figure 1551 An example fur mmaring ;\ dt:ri1atinn rrc~:.

15.16.3.2 Mutating Programs

We havt: always considered murarion as rhc random deformation of .1 _,n1.1ll parr of .1 chrorncl.~onlt:. r. ~~
therefore, nor surprising rhar rhe mosr common mur;nion in gt:nt'tiL pro~r:1111111ing is rhc random rlpb<-l'll\l"LH
of a randomly sdecred subuee. Thr only moditictrion i~ rhar we do not n~:ccs.~aril~ stan from rhl ~\;In ~\'lnhol.
bur from rhe non terminal symbol <H rhc root ol' rhc subtrt:c we consider. hgurl I 'i-S I shtiW., ,Hllx:ILllpk \1 hrrc
in rhe logical expression {NOT (x ORy)). dtt ~ari:1hk I' is replao:d h~- (NOT y).

15.16.3.3 The Fitness Function

There is no common recipe for spn.:i~ing an .1ppropri:1te times~ funninn which ~rrongl~- dqwnd.' ,m dw
given problem. lr is, however, worrh emphasizill~ rhar iris nr:ct~.'>ar~ ILl prmidl' enou~h inf~nfll.lllllll tu ~uu.k
rhe GA to the solution. More specifically. ir i~ not .HtHicir:m ro Jdint .1 tirnt . . ~ timtrion whid1 .t~,Jgl'-' 0 w
a program which does not solve rht problr:m and I ro <1 program 1\'hich >fll\'t'., rht" probkm >mh .I litll<'~'
function would correspond ro a nr:eJlr:-in-hay~rack problem. In rh~,., >t'll.'it' ..1 propt'r firnc.>~ tllt\hLtrl' .,Jwuld
be a gradual com:epr for judging rht' corrcclncss of pmgr:um.
In many .lpplit:ations, the tirness t'uncrion i~ h;tst'd on <I tompari . .on tlt' desin:d and atw.lll~- ,,hl.llllt'll
ompu1. Koza, f(u instance. use . . rlh ~impiL' sum oi' tjlladr;ui .. tTrur~ t<ll 'i\'rllhiJlit- rcgrts~ion ;tnd rht tli~t<'\'l'll
of rrigonomerric iJL'nrities:

fl!-'i--:::: L!_l'. - 1-'(x,)['

lOCI

In rhis detlnirion, F i~ rhc marhcmant;d flln..:rion which t.:lll'rt'~pnnd~ \!) rht pro~ram undn ._-\,duartull. I he
lisr {x;,y;), I _s, :':: N consists of rder.:nLl' pJir.. . - ;I desirt'd OlilfHII y, i~ .~~~~gll~d ro L".ll'h input , . ( '\t.trh dl<'
samples have robe chosen such rhar rhc considered in pur space is covered ~utlicit'nrl~ well. _
Numeric error-based fitness function:. ttsu.tll1 implv minimization prnhlcn1.'-. ;-'Hllllt' clthcr .tpplil.JIU>n~ 111 1 ~
. . . . . . _ . . . ::.
m1ply max1m1zauon r~ks. There .tre bas1cally rwo well-known rramtonn.mons wh1ch allow w sran&trcll/.l1.'
firness functions such rhar always minimization or maximization rasks are obtained. -.L ...
. .

,.
 !__:>~ 17 Ad~anla_Me_s and Ltmitations of Genetic Algorithm 461

(:omider .111 .trbitrary "raw" fimess function/ Assuming that the number of individuals in the population
~ nut ti:.:t'd 1111, Jt rime t), the uandardizedfitness is compured as

[s(b;,,) = j(b;,,) - .;;h, j(bj.,)

F

if/ h;~~ w be maximized and as

[s(b;,l) = f(b;,,) -
.
;fn j(b;,r)
j=l

if_( has to be minimized. One possible variam is m consider rhe besr individual of the last k generations
instead of only considering the acrual generation.
Obviously, standardized fitness rransforrri.s any optimization problem inro a minimization task. Roulette
wheel selection relies on the fact that the objective is maximization of rhe firness function. Koza has suggested
a simple transformation such thar, in any case, a maximization problem is obtained.
W!rh the assumptions of previous definition, rhe adjusted fitness is computed as

jA(b;.,) = i=f [s(bj.,) - [s(bj.,)

Another variant of adjusted fitness is defined as

I
J;(b;.,) = I+ js(bj.,)
For applying GP w a given problem, rhe following points have to be satisfied.

I. An appropriate fitness function, which provides enough information to guide the GA to rhe solution
(mostly based on examples).
2. A symactical description of a programming language, which contains as much elements as necessary for
solving the problem.
3. An inrerprerer for rhe programming language.
The main application areas of GP include: Computer Science, Science, Engineering, An and Emer-
rainmem.

15.17 Advantages and Limitations of Genetic Algorithm

The advanrage.s of GA are as follows:
1. ParaHelism.
2 113bl
lity.
3. Solurion space is wider.
4. The frtness landscape is complex.
5 '(;'.,,.,,
t d" I bal
......,, o rscover go opnmum.
. . .G. Th
.'. . . . .
e problem has multtobjecnve funcnon.

<'
462 Genatic Algorithm

The limitations of GA are as follows:

1. The problem ofidemifying fitness function.
2. Definition of represemacion for the problem.
3. Premarure convoi'ge~ce occurs.
4. The problem of choosing various parameters such as rhe size of rhe population, murarion rare, crossover
rare, the selection method and its strength.

115.18 Applications of Genetic Algorithm

An effective GA representation and meaningful fitness evaluation are the keys of the success in GA applications.
The appeal of GAs comes &om their simplicicy and elegance as robust search algorithms as well as from their
power co discover good solutions rapidly for difficult high-dimensional problems. GAs are useful and effifiem
when
1. the search space is large, complex or poorly understood;
2. domain knowledge is scarce or expert knoWledge is difficult to encode to narrow the search space; .
3. no mathematical analysis is available;
4. traditional search methods fail.
The advantage of the GA approach is the ease with which it can handle arbitrary kinds of constraints and
I
objectives; all such things can be handled as weighted components of the fimess function, making it easy to
adapt ilie GA scheduler to the particular requirements of a very wide range of possible overall objectives. I
GAs have been used for problem-solving and for modeling. GA~ are applied co many scientific, engi~<:ering
problems, in business and emerrainmem, including:
1. OptimiZJ1t;on: GAs have been used in a wide varieryofoprimization casks, including numerical optimiza-
I
tion and combinacorial optimization problems such as traveling salesman problem (TSP), circuit design
(Louis, 1993), job shop scheduling (Goldstein, 1991) and video & sound qualicy optimization.
2. Automatic programming. GAs have been used to evolve computer programs forspeciftc casks and ro design
other compmario'nal structures, for example, cellular aucomara and sorting networks.
I
3. Machine and robot learning. GAs have been used for many machine-learning applications, including
classifications and prediction, and protein structure prediction. GAs have also been used to design neural
networks, to evolye rules for learning classifier systems or symbolic production systems, and to design and
control robots.
4. Economic models: GAs have been used to model processes of innovation, the development of bidding
strategies and the emergence of economic markets.
I
5. Immune system models: GAs have been used to model various aspects of the natural immune system,
including somatic mutation during an individual's lifetime and the discovery of multi-gene famit(es during
evolutionary time.
6. Ecokgjcal models: GAs have been used co model ecological phenomena such as biological arms races,
host-parasite co-evolutiOns, symbiosis and resource flow in ecologies.
7. Population genetics models: GAs have been used to study questions in population genetics, such as 'under
what conditions will a gene for recombination be evolutionarily viable?'
8. Interactions between evolution and learning. GAs have been used co study how individual learning and

L
species evolution affect' one another.

'
c
 463
15.19 Summary

9. Models ofsocial systems: GAs have been used to study evolutionary aspects of social systems, such as ilie
evolution of cooperation (Chughtai, 1995), ilie evolution of communication and trail-following behavior
in ants.

115.19 Summary
Generic algorithms are original systems based on rhe supposed functioning of the living. The method is very
different &om classical optimization algorithms as it:
1. Uses the encoding of the parameters, not the parameters themselves.
2. Works on a population of points, not a unique one.
3. Uses the only values of the function ro optimize, not their derived function or other auxiliary knowledge.
4. Uses probabilistic transition function and not determinist ones.
lr is important ro understand rhat the functioning of such an algorithm does not guarantee success. The
problem is in a stochastic system and a genetic pool may be too far from the solution, or for example, a too

I fast convergence may hair the process of evolution. These algorithms are, nevertheless, extremely efficient,
and are used in fields as diverse as stock exchange, production scheduling or programming of assembly robots
in the automotive industry.
GAs can even be faster in fmding global maxima chan conventional methods, in particular when derivatives

I provide misleading information. It should be noted chat in most cases where conventional methods can be
applied, GAs are much slower because they do not take auxiliary information such as derivatives into accounr.
In these optimization problems, there is no need tO apply a GA, which gives less accurate solutions after much

I longer computation rime. The enormous potencial ofGAs lies elsewhere- in optimization of non-differentiable
or even discontinuous functions, discrete optimization, and program inJucrion.
lr has been claimed char via the operations of selection, crossover and mutation, the GA will converge over
successive generations cowards the global (or near global) optimum. This simple operation should produce
I a fast, useful and robusr technique large!}' because of the face that GAs combine direction and chance in
the search in an effective and efficient manner. Since population implicidy contain much more information
rhan simply the individual fitness scores, GAs combine rhe good information hidden in a solution with good
information from another solution to produce new solutions with good information inherited from both
parents, inevitabl}' (hopefully) leading cowards optimality.

I
I
In this chapter we have also discussed rhe various classifications of GAs. The class of parallel GAs is very
complex, and its behavior is affected by many parameters. It seems iliar the only way ro achieve a greater
understanding of parallel GAs is to srudy individual facets independent!}', and we have seen char some of rhe
most influential publications in parallel GAs concentrate on only one ~lspect (migration rates, communicarion
topology or deme siz.e) either ignoring or making simplifying assumptions on rhe others. Also rhe hybrid GA,
adaptive GA, independent sampling GA and messy GA has been included with the necessary information.
Generic programming has been used to model and control a muhirude of processes and to govern their
behavior according co firnessbased automatically generared algorirhtm. lmplc:mc:ntarion of gcnc:ric progr.:tm
ming will benefit in rhe coming ye:m from new approaches which include research from developmental
biology. Also, it will be necessary tO learn co handle ilie redundancy forming pressures in the evolution of
\ code. Application of generic programming will continue to broaden. Many applications focus on controlling
I behavior of real or virtual agents. In chis role, genetic programming may contribute considerably to the grow-
ing field of social and behavioral simularions. A brief discussion on Holland classifier system is also included

L
in this chapter.
 CCI
4.64
Genetic Algorithm

115.20 Review Questions

1. State Charles Darwin's theory of evolmion. II. Write shan note on Holland classifier systems.
2. What is meant by genetic algorithm? 12. Differentiate between messy GA and parallel
3. Compare and contrast traditional algorithm and GA.
generic algorithm. 13. What is the importance ot hybrid GAs?
4. Stare the impon:ance of generic algorithm. 14. Describe the concepts involved in reakoded-
5. Explain in detail about che various operawrs genetic algorithm.
involved in genetic algorithm. J 5. What is generic programming?
6. What the various types of crossover and mura- 16. Compare generic algorithm and genetic pro-
tion techniques? gramming.
7. With a neat flowchart, explain the operation of 17. List the characteristics of generic programming.
a simple genetic algorithm.
18. With a neat flowchart, explain the operation of
8. State rhe general generic algorithm. genetic programming.
9. Discuss in derail about the various rypes of 19. How are data represented in genetic program-
generic algorithm in derail. ming?
10. State schema theorem.
20. Mention the applic"-rions of genetic algorithm.

115.21 Exercise Problems

I. Determine the maximum of function x x :? x 4. Solve the logical AND function using generic
(0.007x+ 2) using genetic algorithm by w iring a algorithm by writing a program.
program.
5. Solve the XNOR problem usinf.genericalgorithm
2. Determine rhe maximum of function exp( -3x) + by writing a program.
sin(6.7r x) using genetic algorithm. Given range::::
6. Determine the maximum of function exp(5x) +
[0.004 0.7]; bits :::: 6; population :::: 12; gen-
sin(7rr x) using generic algorithm. Given range::::
erations == 36; mutation ==- 0.005; matenum ::::
0.3. [0.002 0.6]; bits = 3; popu arion == 14; gen-
erations :::: 36; mutation :::: 0.006; matenum =
3. Optimize the logarithmic function using a 0.3.
generic aiLorithm by writing a program.

l. .
 Hybrid Soft Computing Techniques 16
Learning Objectives - - - - - - - - - - - - - - - - - - ,
Neuro-fuzzy hybrid systems. Properties of generic neuro hybrid systems.
Comparison of fuzzy systems with neural Genecic algorithm based back-propagacion
nerworks. network (BPN).
Properties of neuro-fuzzy hybrid systems. Advantages of neuro-genetic hybrids.
Characteristics of neuro-fuzzy hybrids. Genetic fuzzy hybrid and fuzzy genetic hybrid
Cooperative neural fuzzy systems. systems.

General neuro-fuzzy hybrid systems. Genetic fuzzy rule based systems (GFRBSs).

Adaptive Neuro-fuzzy Inference System Advantages of generic fuzzy hybrids.

(ANFIS) in MATLAB. Simplified fuzzy ARTMAP.
Generic neuro hybrid systems. Supervised ARTMAP system.

116.1 Introduction
In general, neural networks, fuzzy systems and generic algorithms are distinct soft computing techuiques
evolved from the biological computational strategies and nature's way to solve problems.
Neural networks are rhe simplified models of rhe human nervous sysrems mimicking our ability to adapt
w certain siruations and to learn from the past experiences. Chapters 2-6 of the book discuss the basics
of anificial neural networks, supervised and unsupervised learning neural networks, associative memory
networks, and few other special ner-.vorks. Fuzz.y logic or fuzzy systems deal with uncertainty or vagueness
existing in a system and formulating fuzzy rules ro find a solution to problems. Fuzzy logic does nor operate
on accurate boundaries and it provides a uansition between membership and non-membership of the variables
for a particular problem. Chapters 7-14 of rhe book discuss rhe basic concepts of fuzzy sers, fuzzy relations,
and methods for formulation of membership funC[ions and for convening fuzzy entities ro crisp entities, fuzzy
arithmetic, funy rule base, and fuzzy control sysrem with irs applications. Genetic algorithms inspired by the
natural evolution process are adaprive search and optimization algorithms. Chapter 15 of the book discusses
on the fundamental geiletic operators and general generic algorithm used for finding an optimal solution.
All the above three techniques individually have provided efficient solutions ro a wide range of simple
and complex problems pertaining to different domains ..As discussed in Section 1.5 of Chapter l, these
three techniques can be combined together in whole or in pan, and may be applied to find solution to the
problems, where rhe techniques do nor work indi\'idually. The main aim oF the concept of hybridi7...ation
is to overcome the weakness in one technique. while applying it and bringing out rhe strength of the orher

.
.
-
 .~!i
<
466 Hybrid Soft Computing Techniques

technique to find solution by combining them. Every soft computing technique has particular computational
parameters (e.g., ability to learn, decision making) which make rhem suited for a particular problem and not
for others. It has ro be noted that neural networks are good at recognizing panerns but they are nor good at
explaining how they reach their decisions. On the contrary, fuzzy logic is good at explaining the decisions
but cannot automatically acquire the rules used for making the decisions. Also, the tuning of membership
functions becomes an important issue in fuzzy modeling. Since this tuning can be viewed as an optimization
problem, eiilier neural network (Hopfield neural network gives solution ro optimization problem) or generic
algorithms offer a possibility ro solve this problem. These limirations act as a central driving force for the
creation of hybrid soft computing systems where rwo or more techniques are combined in a suitable manner
that overcomes the limitations of individual techniques.
The importance of hybrid system is based on the varied narure of the application domains. Many complex.
domains have several different component problems each of which may require different rypes of processing.
When there is a complex application which has twO distinct sub~problems, say for example, a signal processing
and serial shift reasoning, then a neural nerwork and fuzzy logic can be used for solving these individual
tasks, respectively. The use of hybrid systems is growing rapidly with successful applications in areas such
as engineering design, stock market analysis and prediction, medical diagnosis, process comrol, credit card
analysis, and few other cognitive simulations.
Thus, even though the hybrid soft computing systems have a great potential to solve problems, if not
applied appropriately they may result in adverse solutions. It is not necessary that when individual techniques
give good solution, hybrid systems would give an even ben:er solution. The key driving force is to build highly.
automated, intelligent machines for the future generations using all these techniques.

116.2 Neuro-Fuzzy Hybrid Systems

A neurofuzzy hybrid system (also called fuz.zy neural hybrid), proposed by]. S. R. Jang, is a learning mechanism
that utilizes the training and learning algorithms from neural networks to find parameters of a fuzzy system
(i.e., fuzzy sets, fuzzy rules, fuzzy numbers, and so on).lr can also be defined as a fuzzy system that determines
its parameters by processing data samples by using a learning algorithm derived from or inspired by neural
network theory. Alternately, it is a hybrid intelligent system that fuses anificial neural nerworks and fuzzy
logic by combining the learning and connectionist structure of neural net\vorks with human-like reasoning
sryle of fuuy systems.
Neuro-fuzz.y hybridization is widely termed as Fully Neural Net\vork (FNN) or Neuro-Fuzzy System
(NFS). The human~ like reasoning style of fully systems is incorporated by NFS (the more popular term is
used henceforth) through rhe use of fuzzy sets and a linguistic model consis(ing of a set oflF-THEN fuzzy
rules. NFSs are universal approximators with the ability to solicit imerpretable lF-THEN rules; this is their
main strength. However, ilie suengrh ofNFSs involves imerpretabiliry versus accuracy, requiremems that are
contradictory in fuzzy modeling.
In the field of fuzzy modeling research, the neuro-fuuy is divided into two areas:
l. Linguistic fuzzy modeling focused on imerpretabiliry (m:linly the Mamdani model).
2. Precise fuzzy modeling focused on accuracy [mainly the Takagi-Sugeno-Kang (TSK) model].

16.2.1 Comparison of Fuzzy Systems with Neural Networks

From :he existing literature, it can be noted thar neuraJ networks and fuzzy systems have some things in
common. If there does not.exist any mathematical model of a given problem, then neural networks and fuzzy
<
16.2 Neuro-Fuzzy Hybrid Systems 467

Table 161 Comparison of neural and fuzzy proces,sing

Neural processing Fuzzy processing

Mathematical model not necessary Mathematical model not necessary
Learning can be done from scmch A priori ~owledge is needed
There are severa1leaming algorithms Learning is not' possible
Black-box behavior Simple interpretation and implemenration

systems can be used for solving that problem (e.g:, pattern recognition, regression, or density escimation).
This is the main reason for the growth of iliese intelligent computing techniques. Besides having individual
advamages, they do have certain disadvantages that are overcome by combining both concepts.
When neural necworks are concerned, if one problem is expressed by sufficient number of observed
examples then only it can be used. These observations are used to train the black box. Though no prior
knowledge about rhe problem is needed extracting comprehensible rules from a neural necwork's structure is
very difficult.
A fuzzy system, on the other hand, does nor need learning examples as prior knowledge; fa[her linguistic
rules are required. Moreover, linguistic description of the input and output variables should be given. If
the knowledge is incomplete, wrong or contradicmry, then ilie fuzzy system must be runed. This is a time-
consuming process. Table 16.1 shows how combining both approaches brings our the advantages, leaving out
the disadvantages.

16.2.2 Characteristics of Neuro-Fuzzy Hybrids

The general architecture of neuro-fuuy hybrid system is as shown in Figure 16-I. A fuzzy system-based NFS
is trained by means of a data-driven learning method derived from neural net\vork theory. This heuristic
causes local changes in the fundamental fuzzy system. At any stage of the learning process- before, during,
or after- it can be represented as a set of fuzzy rules. For ensuring the semantic properties of the underlying
fuzzy system, the learning procedure is constrained.
An NFS approximates ann-dimensional unknown function, partly represented by training examples. Thus
fuzzy rules can be interpreted as vague prototypes of the training data. As shown in Figure 16-1, an NFS is

Inputs Outputs

I
I

L
Figure 161 Architecture of neuro-fuzzy hybrid system.
 ,,
<"-
468 Hybrid Soft Computing Techniques

given by a three-layer feedforward neural nerwork model. It can also be observed that the first layer corresponds
m the input variables, and the second and third layers correspond to the fuzzy rules and output variables,
respectively. The fu7zy sets are converted to (fuzzy) connection weights.
NFS can also be considered as a sy5tem of fuzzy rules wherein the system can be initialized in the form :~
;J!o
of fuzzy rules based on the prior knowledge available. Some researchers use five layers- the fuzzy sets being ,,;f
encoded in the units of the second and the fourth layer, respectively. It is, however, also possible for these
~~~
modds to be transformed into three-layer architecrure. 'r

16.2.3 Classifications of NeuroFuzzy Hybrid Systems

NFSs can be classified imo the following two systems:

l. Cooperative NFSs.
2. General neuro-fuzzy hybrid systems.

16.2.3.1 Cooperative Neural Fuzzy Systems

In this type of system, bmh artificial neural network (ANN) and fuzzy system work independently from each
other. The ANN attempts ro learn the parameters &om the fuzzy system. Four different kinds of coopemive
fuzzy neural networks are shown in Figure 16-2.
The FNN in Figure 16-2(A) learns fuzzy set from the given uaining data. This is done, usually, by fining
membership functions with a neural network; the fuzzy sets then being determined offline. This is followed
by their utilization m form the fuzzy system by fuu;y rules that are given, and not learned. The NFS in
Figure 16-2(8) determines, by a neural network, rhe fuzzy rules from the training data. Here again, the neural
networks learn offiine before the fuzzy system is initialized. The rule learning happens usually by clustering
on self-organizing feature maps. There is also the possibility of applying fuzzy clustering methods to obtain
rules.
For the neuro-fuz.zy model shown in Figure l6-2(C), the parameters of membership function are learnt
online, while the fuzzy system is applied. This means that, initially, fuzzy rules and membership functions musr
be defined beforehand. AJso, in order to improve and guide the learning srep, the error has to be measured.
The model shown in Figure 16-2(0) determines the rule weights for all fuzzy rules by a neural network. A
rule is determined by its rule weight-interpreted as the influence of a rule. They are then multiplied with the
rule output.

16.2.3.2 General NeuroFuzzy Hybrid Systems (General NFHS)

General neuro-fuzzy hybrid systems (NFHS) resemble neural networks where a fuzzy system is interpreted as
a neural nerwork of special kind. The architecture of general NFHS gives it an ad van rage because there is no
communication between fuzzy system and neural network. Figure 16-3 illustrates an NFHS. In this figure the
rule base of a fuzzy system is assumed to be a neural network; the fuzzy sets are regarded as weights and the
rules and the input and output variables as neurons. The choice m include or discard neurons can be made
in the learning step. Also, the fuzzy knowledge base is represented by the neurons of the neural network; rhis
overcomes the major drawbacks of both underlying systems.
Membership functions expressing the linguistic terms of the inference rules should be formulated for l
building a fuzzy controller. However, in fuzzy systems, no formal approach exists ro define these functions.
Any shape, such as Gaussian or triangular or bell shaped or rrapezoidal, can be considered as a membership I
l~
&merion with an arbitrary set of parameters. Thus for fuz:ty systems, the optimization of these functions
in terms of generalizing the data is very important; this problem can be solved l.ly using neural nernrorks.
 469
r
16.2 NeuroFuzzy Hybrid Systems

~~ L-..-..---'

(A)

Fuzzy Rules

=x>rrs~ L--..---'

(B)

,.....
X>r Fuzzv
membership J. 0
!_fuzzy Systems 1_ utput
function

\ Error computin9l
l module
(C)

X>r Rule 1 > ~Output

weights _ Fuzzy Systems ~
1

J Error computing \
module
(D)

Figure 162 Cooperative neural fuzzy systems.

Using learning rules, the neural nernrork must optimize cite parameters by fixing a distinct shape of the
membership functions; for example, triangular. But regardless of the shape of rhe membership functions,
training dara should also be available.
The neuro fuzzy hybrid systems can also be modeled in an another method. In cit's case, the training

l~
data is grouped into several clusters and each duster is designed to represent a particular rule. These rules are
defined by the crisp data points and are not defmed linguistically. Hence a neural necwork, in this case, might
 470

Control output
Hybrid Soft Computing Techniques
r
Neural Network Module

\Computing~
error
>
(Formulates rule base)
IF- THEN rules
I System unde~
consideration

System state

Figure 163 A general neuro-fuzzy hybrid system.

be applied ro train the defined dusters. The resting can be carried our by presenting a random resting sample
~- ... to the trained neural nerwork.. Each and every output unit will return a degree which extends to fit ro the
anr.ecedem of rule.

16.2.4 Adaptive Neuro-Fuzzy Inference System (ANFIS) in MATLAB

The basic idea behind this neuro-adaptive learning technique is very simple. This technique provides a method
for the fuzzy modeling procedure m learn information about a data ser, in order to compute the membership
function parameters rhar best allow rhe associated fuzzy inference system ro track the given input!ompurdata.
This learning method works similarly to that of neural nwvorks.
ANFIS Toolbox in MATLAB environmem performs the membership function parameter adjustments.
The function name used m activate this molbox in anfis. ANFIS toolbox can be opened in MATLAB
~:
either at command line prompt or at Graphical User Interface. Based on the given input-output dam set,
ANFIS mol box builds a Fuzzy Inference System whose membership functions are adjusted either using back
propagation network training algorithm or Adaline network algorithm, which uses least mean square learning
rule. This makes the fuzzy syHem ro learn from the data they model.
The Fuzzy Logic Toolbox function that accomplishes this membership function parameter adjwrmem
is called anfis. The acronym ANFIS derives its name from adaptive neuro-fuzzy inference system. The
anf is function can be accessed either from the command line or through dte ANFIS Editor GUI. Using
a given input/output data set, the toolbox function anfis constructs a fuzzy inference system {FIS) whose
membership function parameters are adjusted using either a back-propagation algorithm alone or in com-
bination with a least squares type of method. This enables fuzzy systems w learn from the data they are
madding.

16.2.4. 1 FIS Structure and Parameter Adjustment

A network-type structure similar to that of a neural network can be used to interpret the input/output. This
strucnue maps inputs through input membership functions and associated parameters, and then through
output membership functions and associated parameters to outputS. During the learning process, the param-
eters associated with the ~embership functions will change. A gradient vector facilitates the compuration
r 16.2 NeuroFuzzy Hybrid Systems
471

(or adjustment) of these parameters, providing a measure of hoW well the fuzzy inference system models the
input!ourput data for a ve!:! set of parameters. After obtaining the gradient vector, any of several optimiza-
tion routines could be applied!to adjust the parameters for reducing some error measure (defined usually
by the sum of the squared difference between the acrual.and desired outputs). anfis makes use of either
back-propagacion or a combination of adaline and back-propagacion, for membership function parameter
estini:i.tion.

16.2.4.2 Constraints of ANFIS

When compared to the general fuzzy infetence.sysrems anfis is more complex. It is not available for all of
the fuzzy inference system options and only supports Sugeno-rype systems. Such systems have the following
properties:
1. They should be the first- or zeroth-order Sugeno-type systems.
2. They should have a single ourpur that is obtained using weighted average defuzzificarion. All outpur
membership functions must be the same rype and can be either linear or constanr.
3. They do not share rules. The number of output membership functions must be equal to rhe number of
rules.
4. They must have unity weight for each rule.
IfFIS structure does not comply with these constraints then an error would occur. Also, all the cusromization
options that basic fuzzy inference allows cannm be accepted by anfis. In simpler words, membership
functions and defuzzification functions cannot be made according to one's choice, rather those provided
should be used.

16.2.4.3 The ANFIS Editor GUI

To ger started with the ANFIS Editor GUI, cype anf isedi tat the :MATLAB command prompt. TheGUI
as in Figure 16-4 will appear on your screen.
From this GUI one can:
1. Load data (training, resting and checking) by selecting appropriate radio buttons in the Load Data portion
of rhe GUI and then cliclcing Load Data. The loaded data is planed on the plot region.
2. Geneme an initial FIS model or load an initial FlS model using the options in the Generate FIS portion
of rheGUI.
3. View the FIS model structure once an initial FIS has been generated or loaded by clicking the Structure
" button.
4. Choose the FIS model parameter optimization method: back-propagation ora mixmreofback-propagation
and least squares (hybrid method).
5. Choose the number of training epochs and rhe training error tolerance.
6. Train th.e FIS model by clicking the Train Now button. This training adjusts the membership function
parameters and plots the training (and/or checking data) error plot(s) in rhe plot region.
7. View the FIS model output versus the training, checking, or testing data output by clicking the Test Now
button. This function plots the test data against the PIS output in the plot region.
One can also use the ANFIS Ediror GUI menu bar to load an FIS training initialization, save your trained

l
I.
].'.
FIS, open a new Sugeno system, or open any of the other GUis to interpret the trained FIS model.
472 Hybrid Soft Computing Techniqul_s-

Figure 16-4 ANFIS E.diror in MATLAB.

16.2.4.4 Data Formalities and the ANFIS Editor GU/

To scan training an FIS using either anfis or the ANFIS Ediror GUI, one needs co have a training data set
char contains desired inpudoutput clara pairs of the rarger sysrem to be modeled. In certain cases, optional
resting data set may be available that can check the generalization capability of the resuhing fuzzy inference
system, and/or a checking data sec that helps with model overfirring during the training. One can account
for overfiuing by resting rhe FIS trained on rhe training clara against the checking data and choosing rhe
membership function parameters to be those associated wirh rhe minimum checking error, if these errors
indicate model overfitting. To determine this, their training error plots have to be examined fairly closely.
Usually, these rraining and checking data sets are stored in separate files after being collected based on
observations of rhe target sysrem.

16.2.4.5 More on ANFIS Editor GU/

A minimum of rwo and maximum six arguments can be taken up by the command anfis whose gener~l
fOrmat is
[fismatl,trnError,ss,fismat2,chkError]= ...
anfis(trnData,fismat,trnOpt,dispOpt,chkData,method);
Here trnOpt (training options), dispOpt (display options), chkData (checking dara), and method
(training method) are optional. All of the output argwnems are also optional. In this section we will discuss
the arguments and range components of the command line function anfis as well as the analogous func~
tionaliry of the ANFIS Ediror GUI. Only the training data set must exist before implementing anfis when
rhe ANFIS Editor GUI is in'!oked using anfisedit. The srep-size will be fixed when the adaptive NFS is
trained using this GUI tool.

~
c

.
 16.2 NeuroFuzzy Hybrid Syslems 473

Training Data
Both anfis and ANFIS Edimr GUI require the training data, trnData, as an argument. For the target
system to be modeled each row of trndata is a dC5ired input/output pair; a row starts with an input vector
and is followed by an ourput value. So, the number of rows of trndata is equal to the number of training
data pairs. Also, because there is only one output, the number of columns of trndata is one more than the
number of inputs.

Input FIS Structure

The input FIS Suucrure, fismat, can be obtained from any of the following fuzzy edimrs:

1. The FIS Editor.

2. The Membership Func(ion Editor.
3. The Rule Editor from ilie ANFIS Editor GUI (which allows a FIS structure to be loaded from a file or
the MATLAB workspace).
4. The command line function, genfisl (for which one needs to give only numbers and cypC5 of
membership functions).

The FIS structure contains both the model structure (specifying, e.g., number of rules in the FIS, the number
i
1,
(

~
of membership functions for each input, etc.) and the parameters (which specify the shapC5 of the membership
functions).
For updating membership function parameters, anfis learning employs two methods:
.,

)
1. Back-propagacton for all parameters (a steepC5t descent method).
2. A hybrid method involving back-propagation for the parameters associated with the input membership
functions and least~squares estimation for the parameters associated with the output membership functions.

This means that throughout the learning process, at least locally, the training error decreases. So, as the initial d
;~~
membership functions increasingly rC5emble the optimal ones, it becomC5 easier for the model parameter ;;li
rraining to converge. In the setting up of thC5e tnitial membership function parameters in rhe FIS structure,
it may be helpful to have human expenise about the target system co be modeled.
Based on a fixed number of membership functions, the genfisl function produces a FIS mucrure. .,.,
This structure invokes the so~called mrse ofdimemio111dity and causes excessive propagation of the number
of rules when the number of inputs is moderately large (more than four or five). To enable some dimension
reduction in the fuzzy inference system, the Fuzzy Logic TOolbox software providC5 a method-a FIS structure
'i
can be generated using the clustering algorithm discussed in Subtractive Clustering. To use this clustering I
'

algorithm, select the Sub. Clustering option in the Generate FIS portion of the ANFIS Editor GUI, before
the FIS is generated. The data is partitioned by the subtractive clustering method into groups called dusters I
and generate$ a F1S with the minimum number of rules required to distinguish the fuzzy qualities associated i'l
with each of the clusters. ""
"
ti
1'!
Training Options
,,
;.,
One can choose a dC5ired error tolerance and number of training epochs in the ANFIS Editor GUI tool.
For the command line anf is, training option trnOpi: is a vector specifying the stopping criteria and the
ti
;I

step~size adaptation strategy: !'

1. trnopt ( 1): number of training epochs; default = 10

~
 ~

474 Hybrid Soft Computing Techniques

2. trnOpt ( 2): error. tolerance; default= 0

3. trnOpt ( 3): initial step-size; default= 0.01
4. trnOpt {4): stepsize decrease rate; default"' 0.9
5. trnOpt {5): step--size increase rate; default= 1.1

The default value is taken if any element of trnOpt is missing or is an NaN. The training process stops if
the designated epoch number is reached or the error goal is achieved, whichever comes first.
The srep-size profile is usually a curve that increases initiaHy, reaches a maximum, and then decreases for
the remainder of the training. This ideal step-size profile can be achieved by adjusting the initial step-size and
the increase and decrease rates (trnOpt ( 3) - trnOpt ( 5) ). The default values are set up to cover a
wide range oflearning taSks. These step-size options may have to be modified, for any specific application, in
order ro optimize the training. There are, however, no user.specilied step-size options for uaining the adaptive
neuro-fuzzy inference system generated using the ANFIS Editor GUI.

Display Options
They apply only to the command line function anfis. The display options argument, dispOpt, is a vector
of either ls or Os that specifies the information to be displayed (print in ilie MATLAB command window)
' before, during, and after the training process. To denote print this option, 1 is used and ro denote do not print
...... this option, 0 is used .

1. dispOpt ( 1): display ANFIS information; default"' 1

2. dispOpt ( 2): display error (each epoch); default"' I
3. dispOpt ( 3): display srep-size (each epoch); default= 1
4. dispOpt ( 4): display final results; default"' 1

All available information is displayed in the default mode. If any element of dispDpt is missing or is NaN,
;: the default value is used.

Method
To estimate membership function parameters, both the command line anfis and the ANFIS Editor GUI
apply either a back-propagation form of rhe steepest descent method, or a combination of back-propagation
and rhe least-squares method. The choices for this argument are hybrid or backpropagation. In the
command line function, anfis, these method choices are designated by 1 and 0, respectively.

Output FIS Structure for Traihing Data

The output FIS structure corresponding to a minimal training error is fismatl. This is the FIS structure
one uses to represent the fuzzy system when there is no checking data used for modd cross-validation. Also,
when the checking data option is not used, this data represents the FIS structure that is saved by the ANFIS
Editor GUL When one uses the checking data option, the output saved is that associated with the minimum
checking error.

Training Error
This is the difference betWeen the training data ourput value and the output of the fuzzy inference sysrem
corresponding to the same uaining data input value (the one associated with that training data output value.)
~

16.2 Neuro-Fuzzy Hybrid Systems 475

The root mean squared error (RMSE) of the training data set at each epoch is recorded by the training
error trnError; and fismatl is the snapshot of the FIS structure when the training error measure is
at irs minimum. As the system is trained, the ANFIS Editor GUI plots the training error versus epochs
curve.

Step-Size
With the ANFIS Editor GUI, one cannot control the step-size options. The step-size array ss records the
step-size during the uaining, using the command line anf is. If one plots ss, one gets the step-siz.e profile
which serves as a reference for adjusting the iniri:i.l step-size, and the corresponding decrease and increase rates.
The guidelines followed for updating the step-size (ss) for the command line function anfis are:

l. If the error undergoes four consecutive reductions, increase the step-size by multiplying it by a constant
(ssinc) greater than one.
2. If the error undergoes two consecucive combinations of one increase and one reduction, decrease the
step-size by multiplying it by a constant (ssdec) less than one.

For the initial step-size, the default value is 0.01; for ssinc and ssdec, they are 1.1 and 0.9, respectively.
All the default values can be changed via the training option for the command line anfis.

Checking Data
For testing the generalization capability of rhe fuzzy inference system at each epoch, the checking clara,
chkData, is used. The checking data and the training data have the same format and elements of the former
are generally distinct from those of the latter.
For learning tasks for which the input number is large and! or the data itself is noisy, the checking data is
important. A fuzzy inference system needs to track a given input!output data set welL The model mucmre
used for anfis is fixed, which means that rhere is a tendency for the model to overfit the data on which
it is trained, especially for a large number of training epochs. In case overfitting occurs, the fuzzy inference
system may not respond well to other independent data sets, especially if they are corrupted by noise. In these
situations, :1 validation or checking dam set can be usefuL To cross-validate the fuzzy inference model, this
d:~.ta set is used; cross-validation requires applying the checking data to the model :~.nd then seeing how well
the model responds to this data.
The checking data is applied to the model at each tmining epoch, when the checking data option is used
with anfis either via the command line or using the ANFIS Editor GUI. Once rhe command line anfis
is invoked, the model parameters hat correspond ro the minimum checking error are returned via the output
argument fismat2. The FIS membership function parameters computed using the ANFIS Editor GUI
when both training and checking data are loaded, are associated with the training epoch that has a minimum
checking error.
The assumptions made when using the minimum checking data error epoch to set the membership function
parameters are:

1. The similarity between checking data and the training clara means that the checking clara error decreases
as the training begins.
2. The checking data increases ar some point in the training after rhe data overfitting occurs.

The resulting FIS may or may not be rhe one which is required to be used, depending on the behavior of the
checking data error.

~
 "f
476 Hybrid Soft Computing Techniques

Output FIS Structure f~r Checking Daca

The output FIS structure with the minimum checking error is the output of the command line anfis,
fismat2. If checking clara is used for cross-validation, this FIS structure is the one rhat should be used for
I
further calculation.

Checking Error
This is the difference becween the checking data ourpuc value and the output of the fuzzy inference system
corresponding to the same checking dala input value, which is the one associated with that checking data
output value. The Root Mean Square Error (RMSE) is reCorded for clte checking data at each epoch, by the
checking error chkError. The snapshot of ilie FIS structure when the checking error has its minimum
value is fi sma t2. The checking error versus epochs curve is planed by the ANFIS Editor GUI, as the system
is trained.

11&.3 Genetic NeuroHybrid Systems

A neuro-genetic hybrid or a genetic-neurohybrid system is one in which a neural network employs a genetic
algorithm to optimize its structural parameters iliat define its architecture. In general, neural networks and
genetic algorithm refers to two distinct methodologies. Neural networks learn and execute different tasks
using several examples, classify phenomena, and model nonlinear relationships; that is neural networks solve
problems by self-learni~g and self-organizing. On the other hand, genetic algorithms present themselves as a
potential solution fOr the optimization of parameters of neural networks.

16.3.1 Properties of Genetic NeuroHybrid Systems

Certain properties of generic neuro-hybrid systems are as follows:

1. The parameters of neural networks are encoded by generic algorithms as a string of properties of the
network, that is, chromosomes. A large population of chromosomes is generated, which represent the
many possible parameter sets for the given neural network.
2. Genetic Algorithm- Neural Network, or GANN, has the abilicy to locate the neighborhood of the optimal
solution quickly, compared to other conventional search mategies.
Figure 16-5 shows the block diagram for the genetic-neuro-hybrid systems. Their drawbacks are: the large
amount of memory required for handling and manipulation of chromosomes for a given network; and also
the question of scalabiliry of this problem as the size of the networks become large.

I 16.3.2 Genetic Algorithm Based Back-Propagation Network (BPN)

BPN is a method of reaching multi-layer neural networks how to perfonn a given task. Here learning occurs
during this training phase. The basic algorithm with architecture is discussed in Chapter 3 (Section 3.5) of
this book in derail. The limitations ofBPN are as follows:
1. BPN do not have the abiliry to recognize new patterns; they can recognize patterns similar to those they
have learnt.
2. They must be sufficiently trained so that enough general features applicable ro both seen and unseen
instances can be extracted; there may be undesirable effects due to over training the network.
-"' ~'-
' t ..
,,,
'_
 16.3 Genetic Neuro-Hybrid Systems 477

ANN

Initial
population Fitness Reproduction Stopping
Selection crossover condition
calculation
mutation met?

New populi:ttion after generation

Properties

1 U''"'"'"uv! 1 Best
individual
ANN topology
and ANN
parameters
Figure 16-5 Block diagram of genetic-neuro hybrids.

Also, it may be nored that the BPN determines its weight based on gradient search technique and hence it
may encounter a local minima problem. Though genetic algorithms do nor guarantee to find global optimum
solution, they are good in quickly finding good acceptable solutions. Thus, hybridization ofBPN with generic
algorithm is expected to provide many advantages compared tO what they alone can. The basic concepts and
working of genetic algorithm are discussed in Chapter 15. However, before a genetic algorithm is executed,

l. A suitable coding for the problem has to be devised.

2. A fitness function has to be formulated.
3. Parents have to be selected for reproduction and then crossed over to generate offspring.

16.3.2.1 Coding
Assume a BPN configuration n-1-m where n is the number of neurons in the input layer, I is rhe number
of neurons in the hidden layer and m is the number of output layer neurons. The number of weights to be
determined is given by

(n+m)f

Each weight (which is a gene here) is a real number. Let dbe the number of digits (gene length) in weight. Then
a StringS of decimal values having string length (n + m)ld is randomly ge.nerared. It is a string that represenrs
weight matrices of inpur~hidden and the hidden-output layers in a linear form arranged as row-major or
column-major depending upon the sryle selected. Thereafter a population ofp (which is the population size)
chromosomes is randomly generated.

16.3.2.2 Weight Extraction

In order to determine the fitness values, weightsareexuacred from each chromosome. Lern,, fiJ., ... , llJ, ... , IlL
represent a chromosome and let apd+l, apd+Z ... , a(p+l)d represent pth gene (p ~ 0) in the chromosomes.

-
' ..
 478 Hybrid Soft Compuling Techniques

The actual weight Wp is given by

apd+210d-2 + apd+310d-3 + ... + a(p+I)d ifO :5 4pd+I < 5

!Qd 2
wp = { apd+Zlod-2 + apd+3wd-3 + ... + a(p+t)d ifS::: tlpd+l :59
+ 1nd 2

16.3.2.3 Fitness Function

A fimess has to be formulated for each and every problem to be solved. Consider ilie matrix given by

(Xli,X2.l>X31>, ,XIII) (y11 J2J,]3l ,y,J)

(xJ2, X2.2, x.n ... , x,z) (/'J2,Y22Y32 ,y,a)
(x\3,X'23> X'33 ,Xn3) (y\3}23>}33 >}n3)

(Xtm,X2m> XJm,. ,x,,m) (r!m>Y2m>]3m> ,y,m)

where X and Yare the inputs and targets, respectively. Compute initial population Io of size 'j'. Let
Ow, Ozo, ... , OjO represem 'j' chromosomes of the initial population /o. Let the weights extracted for each
of the chromosomes upto jth chromosome be w1o, 1Uzo. w3o, ... , WfJ- For n number of inputs and m number
of outputs, let the calculated output of the considered BPN be

(CJ 1> '21, C31, .. 'Cnd

(cJ2, CZ2> c32,. . , c~d
I
(cl3' l'23 CJ3, , c773)

j;
(qm, C"!m> C3m , Cmn)

k a result, the error here is calculated by

2
ER1 = (y11- c11) + (nl -CzJ) 2 + (y31- "31) 2 + + (y.,1- C J) 2 11

ERz = (y1z - c12)

2
+ (yzz - '22) 2 + (y32 - c3z)2 + + (y"z - cnz) 2

2
ERm = (ylm- CJm) + !Jzm- Czm) 2 + (r3m- C3m) 2 + + (rnm- Cnm) 2

The fitness function is further deriv~d &om this root mean square error given by

FF" = _1_
Em~u

The process has to be carried out for all the total number of chromosomes.
 16.4 Genetic Fuzzy Hybrid and Fuzzy Genetic Hybrid Systems 479

16.3.2.4 Reproduction of Offspring

In this process, before the parents produce the offspring with better fitness, the mating pool has to be
formulated. This is accomplished by neglecting the chromosome with minimum fitness and replacing it with
a chromosome having maximum fimess, In other words, the fittest individuals among clle chromosomes will
be given more chances to participate in the generations and the worst individuals will be eliminated. Once
the mating pool is formulated, parent pairs are selected' randomly and the chromosomes of respective pairs are
combined using crossover technique to reproduce offspring. The selection operator is suitably used to select
the best parem to participate in the reproduction process.

16.3.2.5 Convergence
The convergence for generic algorithm is the number of generations wilh which the fitness value increases
wwards the global optimum. Convergence is the progression towards increasinguniformiry. When about 95%
of the individuals in the population share the same fitness value then we say that a population has converged.

16.3.3 Advantages of NeuroGenetic Hybrids

The various advantages of neuro-genetic hybrid are as follows:

GA performs optimization of neural network parameters with simplicity, ease of operation, minimal
requiremems and global perspective.
GA helps to find om complex structure of ANN for given input and the ourput data set by using its
learning rule as a fimess function.
Hybrid approach ensembles a powerful model that could signif1camly improve the predicmbility of rhe
system under construction.
The hybrid approach can be applied to several applications, which include: load forecasting, srock forecasting,
cost optimization in textile industries, medical diagnosis, face recognition, multi-processor scheduling, job
shop scheduling, and so on.

lt6.4 Genetic Fuzzy Hybrid and Fuzzy Genetic Hybrid Systems

Curremly, several researches has been performed combining fuzzy logic and genetic algorithms (GAs), and
there is an increasing interest in the integration of these two topics. The imegration can be performed in the
following two ways:

1. By the use offuzzy logic based techniques for improving generic algorithm behavior and modeling GA
components. This is called fozzy genetic algorithms (FGk).
2. By the application of genetic algorithms in various optimization and search problems involving fuzzy
systems.

An FGA is considered as a genetic algorithm that uses techniques or tools based on fuzzy logic to improve
the GA behavior modeling. lr may also be defined as an ordering sequence of instructions in which some
of the instructions or algorithm components may be designed wirh tools based on fuzzy logic. For example,
fuzzy operators and fuzzy connectives for designing genetic operators with differ.;:nt properties, fuzzy logic
comml systems for coJl[rolling the GA parameters according to some performance measures, stop criteria,

lI representation tasks, ere.

hi
480 Hybrid Soft Computing Techniques

GAs are utilized for solving different fuzzy optimization problems. For example, fuzzy flowshop scheduling ~
problems, vehicle routing problems with fuzzyduNime, fuzzy optimal reliability design problems, fuzzy mixed -"<4
integer programming applied m resource distribution, jobshop scheduling problem with fuzzy processing "~
time, interactive fuzzy satisfYing method for multi-objective 0-1, fuzzy optimization of distribution nerworks,
etc.
";:

116.4.1 Genetic Fuzzy Rule Based Systems (GFRBSs) ";'.

For modeling complex systems in which classical tools are unsuccessful, due to them being complex or
imprecise, an imponant tool in the form of fuzzy rule based systems has been identified. In this regard, for
mechanizing the definicion of the knowledge base of a fuzzy controller G& have proven to be a powerful roo!,
since adaptive conrrOl, learning, and self-organization may be considered in a Joe of cases as optimization or
search processes. Over the last few years their advantages have extended the use of GAs in the development
of a wide range of approaches for designing fuzzy controllers. In particular, the application to the design,
learning and tuning of knowledge bases has produced quite good results. In general these approaches can be
termed as Grnttic Fuzzy Systems (GFSs}. Figure 16-6 shows a system where genetic design and fuzzy processing
are the two fundamental constituents. Inside GFRBSs, it is possible to distinguish between either parameter
optimization or rule generation processes, that is, adaptation and learning.
The main objectives of optimization in fuzzy rule based system are as follows:
l. The task of finding an appropriate knowledge base (KB) for a particular problem. This is equivalent to
parameterizing the fuzzy KB (rules and membership functions).
2. To find those parameter values that are optimal with respect to the design criteria.
Considering a GFRBS, one has to decide which parts of the knowledge base (KB) are subject to optimization
by the GA. The KB of a fuzzy system is rhe union of qualitatively different components and not a homogeneous

Genetic
Algorithms

Fuzzy processing

Figure 166 Block diagram of genetic fuzzy system.

l
t 6.4 Genetic Fuzzy Hybrid and Fuzzy ~enet1c Hybnd Systems 481

Table 162 "liming vrr~u~ l<:.1rning prohlcm'

Tuning Learning Problems
It is conc~rned with optimization of an .:xisring FRRS. h ~onsrirutt".~ an automated dt"~i~n mt:dmd t(lr fm.z~
rule sers. that ~t;lrt from s~.:rau,;h.
Tuning processes assume <1 predefined RB and have the Lcarni~g processes pt'rtOrm a more dabor;nt"d w<~rch
objective to find a ct of optimal parameters for the mem- in rhe spaCt' nf possible RRs or wholt" KB and do not
bership and/or the scaling functions, DB parameters. depend on a prt"dcfincll ser of rules.

structure. As an example, the K8 ,1fa descriptive Marhdani-cype fU1.zy system has two components: <l rule base
(RB) containing rhe collection offuzzy rules and a data base {DB} ; .mtaining rhe definitions of the seal in!;!,
factors and the member.~hip functions of the fuzzy .~ets associated with the Hnguisric labels.
In this phase. it is imponanr ro distinguish between tuning {ahernarivdy. adaptation) and learning
problems. See Table 16-2 for the differences.

16.4.1.1 Genetic Tuning Process

The task of tuning the scaling furlC[ions and fuu.y membership funcriom is imporram in FRBS design.
The adoption of parameteri7.cd scaling functions <Jr.d membership functions hy rhe GA is based on the
fimess function rhat specifies the design criteria quantitatively. The responsibility of finding a set of optimal
parameters for the member.~hip andfor the scal!ng functions rests with the tuning proce!'ses \o,.-hich assume a
predefined rule ba.~e. The tuning process can he performed a priori also. This can be done if <1 mbsequent
process derives the RB once rhe DR has been oht;lined, char i.~. a priori generic DB learning. Figure 16-7
illustrates the process of genc1!.._ tuning.

Trming Scaling Functions

The universes of discourse whcrt' fuzzy mcmbc:rship funniom arc ddincd are normalized by scaling functions
applied to the i1put and omput variables of FRBSs. In case of linear scalin~. the scaling functions arc
parameterized by a single ~caling factor or cirher hy specifYing a lower and upper bound. On rhc orher hand.
in case of non-linear scaling, the scaling functions are paramt"terized by one or several contracciotlfdilation
parameters. These parameters are adapted such thar rhc scaled universe of diswurse marches the underlying
variable range.
Ideally, in these kinds of proces.~es the approach is to adapt one ro four parameters per variable: one when :.1
using a scaling facror, rwo for linear scaling, and three or four :Or non-linear scaling. This approach leads to
a fixed length code as the number of variables is predefined as is rhe number of parameters required ro code
each scaling function.

Figure 167 Process of ntning rhe DB.

 482 Hybrid Soft Computing Techniques

Tuning Membership Functions

It can be no[ed that during the tuning of Il).embership funcrions, an individual represents the entire DB. This
is because irs chromosome encodes the parameterized membership functions associated to the linguistic terms
in every fuzzy partition considered by the fuzzy rule based system. Triangular (either isosceles or asymmetric),
uapez.oidal, or Gaussian functions are rhe most common shapes for ilie membership functions (in GFRBSs).
The number of parameters per membership flmcrion can vary from one to four and each parameter can be
either binary or real coded.
For FRBSs of rhe descriptive (using linguistic variables) or the approximate (using fuzzy variables) cype, the
structure of the chromosome is different. In the process of runing the membership functions in a linguistic
model, the entire fuzzy panirions are encoded into the chromosome and in order tO maintain the global
semantic in the RB, it is globally adapted. These approaches usually consider a predefined number of linguistic
terms for each variable- with no requirement to be the same for each of them -which leads to a code of fixed
length in what concerns membership functions. Despite this, it is possible to evolve the number of linguistic
terms associated to a variable; simply define a maximum number (for the length of the code) and let some
of the membership functions be located out of the range of the linguistic variable (which reduces the actual
number oflinguistic terms).
Descriptive fu.u.y systems working with strong fuzzy partitions, is a particular case where the number of
~
parameters to be coded is reduced. Here, the number of parameters to code is reduced to the ones defining the
i core regions of the fuzzy sw: the modal point for triangles and the exueme poinrs of the core for trapezoidal
.,, shapes .
Tuning the membership functions of a model working with funyvariables (scarrer partitions), on the other
hand, is a particular instance of knowledge base learning. This is because, instead of referring to linguistic
terms in the DB, the rules are defined completely by their own membership functions.

16.4. 1.2 Genetic Learning of Rule Bases

As shown in Figure 16~8, genetic learning of rule bases assumes a predefined set of fuzzy membership functions
in the DB to which rhe rules refer, by means of linguistic labels. As in the approximate approach adapting
rules, it only applies to descriptive FRBSs, which is equivalent to modifying the membership functions. %en
i considering a rule based system and focusing on learning rules, there are three main approaches that have
been applied in the literature:
1. Piusburgh approach.
2. Michigan approach.
3. Iterative rule learning approach.

Predefined set of fuzzy

membership !unctions
(Database)

Figure 168 Genetic learning of rule base.

 h
16.5 Simplified Fuzzy ARTMAP 483

Gene)ic knowledge
base learning
process

Knowledge base I
Icomputing. module
Knowledge base ~
with data base and
rule base

Figure 169 Generic learning of the knowledge base.

The Piusburgh approach is characterized by representing an entire rule ser as a generic code {chromosome),
maintaining a population of candidate rule sers and using selection and generic operators to produce new
generations of rule sets. The Michigan approach considers a different model where the members of the
population are individual rules and a rule set is represented by the entire population. In the third approach,
rhr iterative one, chromosomes code individual rules, and a new rule is adapted and added to the rule set, in
an iterative fashion, in every run of the genetic algorithm.

16.4. 1.3 Genetic Learning of Knowledge Base

Genetic learning of a KB includes different generic represenrarions such as variable length chromosomes,
multi-chromosome genomes and chromosomes encoding single rules instead of a whole KB as it deals wirh
heterogeneous search spaces. As rhe complexity of the search space increases, rhe computational cost of the
generic search also grows. To combat this issue an option is to maintain a GFRBS that encodes individual
rules rather than entire KB. In this manner one can maintain a flexible, complex rule space in which rhe search
for a solution remains feasible and efficient. The three learning approaches as used in case of rule base can
also be considered here: Michigan, Pittsburgh, and iterative rule learning approach. Figure 16-9 illustrates the
generic learning ofKB.

116.4.2 Advantages of Genetic Fuzzy Hybrids

The hybridization between fuzzy systems and GAs in GFSs became an important research area during the
last decade. GAs allow us to represent different kinds of structures, such as weights, features together with
rule parameters, ere., allowing us m code multiple models of knowledge representation. This provides a
wide variety of approaches where it is necessary m design specific generic components for evolving a specific
representation. Nowadays, it is a grOwing research area, where researchers need to reflect in order to advance
towards strengths and distinctive features of the G FSs, providing useful advances in the fuzzy systems theory.
I Generic algorithm efficiently optimizes the rules, membership functions, DB and KB of fuzzy systems. The
methodology adopted is simple and the fittest individual is identified during the process.

I
I 116.5 Simplified Fuzzy ARTMAP
The basic concepts of Adaptive Resonance Theory Neural Nerworks are discussed in Chapter 5. Both the

L
types of ART Nerworks, ART-1 and ART~2, are discussed in derail in Section 5.6.

.. -~ .
484 Hybrid Soft Computing Techn1ques

Apan from rheR rv.o ART m~nvnrb, the other two maps are ARTMAP and fuzzy ARTMAI~ ARTMAP
is also known as Pn:Jicrive ART. It combines rwo slightly modified ART-1 or ART-2 units iLHo a supervised
learning mucture. Here. the first unit rakes the inpm data and the second unit rakes the correct outpur data.
Then minimum po~~ible adjusrmem of the vigilance parameter in rhe fim unit is made using the correct
output data .~o rhar correct classification can be made.
The Fuzzy ARTMAP model has fuzzy-logic-based computations incorporated in rhe ARTMAP model.
a
fuzzy ARTMAP is neural nw.,.ork architecture for conducting supervised learning in multidimensional
serring.. When Fuzzy ARTMAP is used on learning problem, it is trained till it correctly classifies all uaining
data. Thi:> feature causes Fuzzy ARTMAP ro "overfir" some darasers, especially those in which the underlying
panern ha.~ m overlap. To avoid the problem of "overfiuing" one must allow for error in rhe training
process.

I 16.5.1 Supervised ARTMAP System

Figure 16-10 shows rhe super\'ised ARTMAP system. Here, two ART modules are linked by an inrer-ART
module called rhe Map Field. The Map Field forms predictive associations berween categories of the ART
modules and realizes a march tracking rule. If ARTa and ARTb are disconnected, then each module would be
of self-organize category, grouping their respective in pursers. In supervised mode, rhe mappings are learned
berween input vecmrs a and b.

I 16.5.2 Comparison of ARTMAP with BPN

1. ARTMAP nerworks are self-stabilizing, while in BPNs the new information gradually washes away old
information. A consequence of rhis is rhat a BPN has separate training and performance phases while
ART MAP systems perform and learn at the same time.

Map field
orientating
subsystem

Art 8
I
8 I
l
Figure 1610 Supervised ARTMAP system.

l "
 16.7 Solved Problems us1ng MATlAB 485
\I
2. ART MAP networks arc designed ru work in real-rimt. while BPNs ;Ire rypictlly Jc!>igned w work ofl.Jint:,
ar lea.sr during rhcir rraining pha~e.
3. ART MAP sysrems r:;mlearn hmh in;\ t~tst as wdf;tS inslow m;Hdl configuration, whik rhe HPN em on!~
learn in slow misnutch confi~ur;Hion. Thi.~ means th;,u an ARTMAP ~~srcm learns, or adapt.~ it~ wciglus,
only when the inpm matchc:~ <llll'.q,tbli~hed cnq;or.~. while HPNs learn wh~n the input docs not march
;tn establi~hcd carcgorr
4. In HPN~ there is ;Jiw:l~'s ;t ch:mtc nfdw s~s~~;:m geuing rrappni i11 ;\local minimum while this is impossible
tilT ART system~.
However, rhl S\'.qem~ h;tscd on ART module.~ lc;trnin~ ma~ tlcp~nd upon rhc llrdcring of rho.: inpur
jl:\Hl'rrl~.

1 16.6 Summary

ln rhis .:h;ljlteT. rhc vuiom h~brids ofindividu;Jineur:Jlnctwnrks, fuuy log_i( and gplctic ;Ji~orirhm h:1vc: bc(n
discussc:d in detail. The :H.lvamagc.~ of c:tch of theM: techniques an combined mgedll'r ro give ;1 bcncr solnrion
to tht problem under (unsidcr;lrion. Ltch of these s~stcm~ posst~.~l~ .:lrr:1in limiratimb wllt'n rht~ optr:uc
individual!~ and thc~t' limit;uinn~ ;Ht' llll'l hy bringing nut rht' ;tdvant;lgt'.' of cmnhining rhc.w ~ysttm.,_ Till'
h~brid systems an: t(mnd to provide herter ~olurio11 for wmpkx prohletm ;md dtl ,tlh t'IH of hybrid s~.-;rcnh
":!..
makl~ it .tpplicahlc to bo.: .tpplio.:d i11 \';lfium applicuiun dom.1im.
,,

116.7 Solved Problems using MATLAB

I. \X'ritl' a :\1:\Tl :-\B pmgram to .tdapr dll' giwn inpu1 tn ~llll" w;tvl t~mn min~ ,ltbpri"~ lll"llfll-fllz7Y h~brid
tedlniqtlt'.
Soun:~ todc
'') "'-!) ,}- "iJ: :;, .J'V<~r :r:;_.:.;c 1.' 0'::. In'.' ;,; C.U).' ,, ;:

1i<..::.::y ,,: H: I ,_Cl.J:.C lc

!t:dl ci:
' ... :-;,: :1.

: :!i-J<: .~cl: "'

~:. : [,: G. : :: ,l:
~is;) I '!'h(- : ::~1.. ~lc. ~ ... ,--.r: ~= j s
;; i Jo;;_: I:.; i ;

' ' <': .;;v~

: ~sin I~:'
: ~;' .. ,...
' ,._,: .,, ... )s

' : .1.1 ,;o ~ "

r-nda.;_('j- .:.:..;
,-,_fs=7;

"
 486 Hybn'd Sot! Compuling Techniques

epochs=570;

%creating fuzzy inference engine

fis=genfisl(trndata,mfs);
plotfis(fis);
figure
r=showrule(fis);

%creating adaptive neuro fuzzy inference engine

nfis=anfis(trndata,fis,epochs);
rl=showrule(nfis);

%evaluating anfis with given input

y=evalfis(x,nfis);
disp{'The output data from anfis : ');
disp(y);

%calculating error rate

e=y-t;
plot (e);
title{'Error rate')
,,. figure

%plating given training data and anfis ouLput

plot(x,t, 'o',x,y, '*');
title('Training data vs Output data');
legend('Training data', 'ANFIS Output');

,. Output
The input data given x is:
> 0
0.3000
0.6000
0.9000
1. 2000
1. 5000
1.8000
2.1000
2.4000
2.7000
3.0000
3.3000
3.6000
3.9000
4.2000
4.5000
4.8000
5.1000
5.4000
5.7000
 16.7 Solved Problems using MATLAB 487

6.0000
6.3000
6.6000
6.9000
7.2000
7.5000
7.8000
8.1000
8.4000
8.7000
9.0000
9.3000
9.6000
9.9000
10.2000
10.5000
10.8000
11.1000
11.4000
11.7000
12.0000
12.3000
12.6000
12.9000
13.2000
13.5000
13.8000
14.1000
14.4000
14.7000
15.0000
15.3000
15.6000
15.9000
16.2000
16.5000
16.0000
17.1000
17.4000
17.7000
18.0000
18.3000
18.6000
18.9000

l
19.2000
 f
488 Hybrid Soft Computing Techmques

C9. SOOt,
:-! RliQ(:

Ttif : argo,n dat_a give:1

(1
."!955
;j .
t)_',646
\), I fl) ~

.:lL'(J
L:.491'J
'~ . ') ': 8
. rd; ;,

-" .''J._
. -L

h\.1

'''""

.,,
.. t -~

. -~'
; I ~-,

..,.

,")

...,,,_
:_-1,;."
'J,

. 0.

-,: J ;,0

'J 9 ~~ ::.
U.R038
16.7 Solved Problems using MATtAB 489

0.9437
0.9993
0.9657
0.8457
0.6503
0.3967
0.1078
-0.1909
-0.4724
-0.7118
-0.8876
-0.9841
-0.9927
-0.9126
-0.7510
-0.5223
-0.2470
0.0504
0.3433
0.6055
0. 8137

ANFIS info:
Number of nodes: 32
Number of linear parameters: 14
Number of nonlinear parameters: 21
Total number of parameters: 35
Number of training data pairs: 67
Number of checking data pairs: 0
Number of fuzzy rules: 7

Start training ANFIS

1 0.0517485
2 0. 0513228
3 0.0508992
4 0.0504776
5 0.0500581

Step size increases to 0.011000 after epoch 5.

6 0.0496406
7 0.0491837
a o.o4S7291

568 0.00105594
 490 Hybrid Soft Comp"t;og Techo;q,es

Step size increases to 0.000450 after epoch 568.

569 0.0010555
570 0. 0.0105516

Designated epqch number rea~hed---> ANFIS training completed at epoch 570.

The output data from anfis':

-0.0014
0.2981
0.5647
0.7817
0.9314
0.9984
0.9747
0.8629
0.6746
0. 4271
0.1416
-0.1571
-~ -0.4425
~-. -0.6884
~ .....
~ -0.8720
-0.9772
-0.9955
-0.9260
-0.7735
-0.5509
-0.2788
0.0174
I
0. 3112
0. 5777

" 0.7935
0.9387
0.9991
0.9697
0.8540
0.6627
0.4122
0.1247
-0.1741
-0.4574
-0.7000
-0.8801
-0.9812
-0.9941
-0.9189
-0.7623
-0.5371
-0.2629
0.0346
r 16.7 Solved Problems using MATLAB

0.3277
491

0.5908
0.8024
0.9442
1.0014
0.9667
0.8443
0.6484
0.3969
0.1093
-0.1900
-0.4731
-0.7130
-0.8879
-0.9833
-0.9916
-0.9125
-0.7521
-0.5232
-0 .. 2457
0.0526
0.3426
0.6015
0.8163

<end of program>

Figure 16~11 illustrates the ANFIS system module; figure 16-12 the error me; and Figure 16-13 rhe perfor-
mance of training dam and output data. Thus ir can be noted from Figure 16-13, that an f is has adapted
the given inpur to sine wave form.

iI
I
i A

I ~ an! is

(sugeno) f(")

7 rules
c '-
I
input1 (7} o"tptrt (7)
I
Systemanfis: 1 inputs, 1 outputs, 7 rules

Figure 1611 ANFIS system module.

~

,;~

492 Hybrid Soft Computing Techniques

X 1o-" Error rate

3 r--'

-1

-2

-3

-4

-soL____-41o~--~2L0----~3~o-----=-----s=o~--~M~--~7o

Figure 1612 Error r:ue.

Training data vs Output data

1.5~~~,-------:,._r==l~=:=====;-!
Training data
ANFIS Output

!'@ .,"

"'
"'"'""'
(j)

.,"' ~
.,"' "'
0.5f
., ., .,

"'
"'
(j)
<!>
om

., ., .,
<J.5 f "' ., ., "'
"'
., ., .."'


(j)

-1
0 2 4
"' "' "'"'12 ., <!>

6 8 10 14 16 18 20
Fi_gure 1613 Performance of training data and output data.
16.7 Solved Problems using MATLAB
493 i~
l
2. Write a MATLAB program to recognize rhe given input ofalphabets to its respective Outpu~ using adaptive
neuro-fuz:z:y hybrid technique.
Source code
%program to recognize the given input of .alphabets to its respective l'
~
il
%outputs using adaptive neuro fuzzy hybrid technique. '!

'clc;
clear all;
close all;
~
%input data "
x=LO,l,O,O;l,O,l,l;l,l,l,2;1,0,1,3;1,0,1,4;
1,1,0,5;1,0,1,6;1,1,0,7;1.0,1,8;1,1,0,9;
0,1,1,10;1,0,0,11;1,0,0,12;1,0,0,13;0,1,1,14;
1,1,0,15;1,0,1,16;1,0,1,17;1,0,1,1B;1,1,0,19;
1' 1' 1' 20; 1' 0' 0, 21; L 1' 0' 22; 1' 0' 0 I 23; 1, 1 1 24; 1
J J

%target data
t:::[O;O;O;O;O;
1;1;1;1;1;
2;2;2;2;2; i.
'
~
3;3;3;3;3;
4;4-;4;4;4; 1

%training data
trndata= [x, t);
mfs::o3;
epochs=400; /
!
%creating fuzzy inference engine '
~

fis=genfis1(trndata,mfs); ~
plotmf ( fis, 'input' , 1) ;
r::oshowrule(fis);

%creating adaptive neuro fuzzy inference engine

nfis = anfis(trndata,fis,epochs);
surfview(nfis);
figure
rl=showru1e(nfis);

%evaluating anfis with given input

Y=eva1fis(x,nfis);
disp('The output data from anfis:'):
disp(y);

%calculating error rate

e=y-t;
plot (e);
title('Error rate');
figure
 494
Hybrid Soft Computing Techniques

%plating given training data and anfis output

plot(x,t, 'ro ,x,y, 'kx');
title('Training data vs Output data');
legend('Training data', 'ANFIS Output', 'location', 'North');
Output
X =
0 1 0 0
1 0 1 1
1 1 1 2
1 0 1 3
1 0 1 4
1 1 0 5
1 0 1 6
1 1 0 7
1 0 1 8
1 1 0 9
0 1 1 10
1 0 0 11
1 0 0 12
!!-,.
1 0 0 13
!: .
.......~ c 1 1 14
1 0 15
0 1 16
1 0 1 17
1 0 1 18
1 1 0 19
1 1 1 20
1 0 0 21
!" 1 1 0 22
1 0 0 23
'/. .'
1
~.r 1 1 24

t
0
0
0
0
0
1
1
1
1
1
2
2
2
2
2
3
~ 16.7 Solved P~blms "'ing MATlAB 495

3
3
3
3
4
4
4
4
4

ANFIS info:
Number of nodes: 193
Number of linear parameters: 405
Number of nonlinear parameters: 36
Total number of parameters: 441
Number of training data pairs: 25
Number of checking data pairs: 0
Number of fuzzy rules: 81

Start training ANFIS

1 0.08918
2 0. 0889038
3 0. 0886229
4 0.0883371
5 0.0880464

Step size increases to 0.011000 after epoch 5.

6 0.0877506
7 0. 0874193

I 398
399
0.00102161
0.00102102
400 0.0010191

Step size increases to 0.003347 after epoch 400.

Designated epoch number reached--> ANFIS training completed at epoch 400.

I
I
The output data from anfis:
-0.0000
0.0009
I 0.0000
-0.0031
I 0.0024
1. 0000

l
0.9997
 7
~:

496
Hybrid Soft Computing Techniques

1. 0000
1.0002
1.0001
2.0000
2.0001
1. 9998
2.0001
2.0000
2.9999
2.9982
3.0022
2.9994
3.0001
4.0000
4.0000
3.9999
4.0000
4.0000

<end of program>

Figure 16-14 shows rhe degree of membership. Figure 16-IS illusrmes the surface view of the given
system; Figure 16-16 the error rare; and Figure 16-17 the performance of training clara with output
data.

in1)mf1 in1mf2
1 inl:lmf3

0.8

"
~

.11 0.6
E
E
~ 0.4

~
0.2

or-----------------------J
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
i[ipUll
I
Figure 1614 Degree of membership.
I
I
_.&_
 16.7 Solved Problems using MATIAS 497

i
~I
t~
\,,
',,
I
Figure 1615 Surface view of rhe given system.

x 10-:1 Error rate

3~------------~----------~

-1

-2

-3

~~~0------~5~-----710~----~15~----~2~0------~25'

Figure 16-16 Error rate.

_
 1-

498 Hybrid Soft Computing Techniques

Training data vs Output data

4.5 .
0 Training data
4 0 ANFIS Output @

3.5

3T

2.5

T
1.5

0.5

or_:_:
000

-0.5
0 5 10 15 20 25
,., Figure 1617 Performance of training clara with ourput clara.

3. Write a MATI.AB program m train the given trmh table using adaptive neuro-fuuy hybrid technique.
Source code
%Program to train the given truth table using adaptive neuro fuzzy
%hybrid technique.

clc;
clear all;
close all;

%input data
X:;: [ 0, 01 0 j 0, 0, 1 j 0, 1, 0 i 0, 11 1 j 1 1 0, 0 j 1, 0 1 1 j 1, 1 1 0 i 1 1 1 1 1 j 1

%ta"rget data
t=(O;O; 0;1; 0;1; 1;1;]

%training data
trndata= [x, t];
mfs=3;
mfType = 'gbellmf';
epochs=49;

%creating fuzzy inference engine

fis=genfisl(trndata,mfs,mfType);
plotfis(fis);
title('The created fuzzy logic');
-
499
16.7 Solved Problems using MATlAB

figure
plotmf(fis, 'input',l);
title('The membership function of the fuzzy');
surfview ( fis) ;
figure
ruleview ( fis) ;
r=showrule(fis);

%creating adaptive neuro fuzzy inference engine

nfis = anfis(trndata,fis,epochs);
plotfis (nfis);
title('The created anfis');
figure
plotmf(nfis,'input',l);
title('The membership function of the anfis');
surfview(nfis);
figure
ruleview(nfis);
rl=showrule(nfis);

%evaluating anfis with given input

y=evalfis(x,nfis);
disp('The output data from anfis:');
disp(y);

%calculating error rate

e=y-t;
plot(e);
title('Error rate');
figure

%plating given training data and anfis output

plot (x, t, 'o' ,x,y, '*' l;
title('Training data vs Output data');
legend('Training data','ANFIS Output');

Output
X =
0 0 0
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1

t
0

L
0
 :'"7
500
Hybrid Soft Computing Techniques

0
1
0
1
1
1

ANFIS info:
Number of nodes: 78
Number of linear parameters: 108
Number of nonlinear parameters: 27
Total number of parameters: 135
Number of training data pairs: 8
Number of checking data pairs: 0
Number of fuzzy rules: 27
Start training ANFIS
1 3.13863e-007
2 3.0492e-007
3 2.9784le-007
4 2. 90245e-007
5 2.84305e-007

Step size increases to 0.011000 after epoch 5

6 2.78077e-007

47 2.22756e-007
48 2.22468e-007
49 2.22431e-007

Step size increases to 0.015627 after epoch 49.

Designated epoch number reached--> ANFIS training completed at epoch 49.

The output data from anfis:
-0.0000
0.0000
0.0000
1. 0000
0.0000
1.0000
.0000
1. 0000

<end of program>

Figure 16-18 shows the ANFIS module for the given system wirh specified inputs. Figure 16-19 illusuaces
che rule viewer for rhe ANFI_S module. Figure 16-20 gives rhe error rate. Figure 16-21 shows the performance
of Training d:na and ourpur data.

_l.
 501

n
16.7 Solved Problems using MATLAB
~-

-" :\
The created fuzzy logiC

n_,,,:;
anf1s

(sugeno) l(u)

'llrules
output (27)

n input3 (3)
System anfis: 3 inputs, 1 outputs, 27 rules

Figure 1618 ANFIS module for rhe given system with specified inputs.

Figure 16-19 Rule viewer forthe ANFIS module.

 502 Hybrid Soft Computing Techniques

X 10-7 Error rate

0.5

-0.5

-1

-1.5

-2

-2.5

-3

~-~~ -3.5
.' "..
~-.._;
1 2 3 4 5 6 7 8
Figure 1620 Error rate.

Training data vs output data

1.2,~~~-;:..__ __:,.._----o;:~==~
!
/
~---~-'-''
:-I
0.8

0.6

0.4

0.2

-0.2 !:--7:~;;';;----;;';;--;;";---;;';---;';::---c:C':;----;C;;-;;-;;------:
0 ~1 M M M M M ~ M M
FIQure 1621 Performance of training darn and ourpm data.
 16.7 Solved Problems using MATI.AB
503

4. Write a MATLAB program to optimize the neural network parameters for the given truth table using
genetic algorithm.

Source code
%Program to optimize the neural network parameters from given truth table
%using genetic algorithm

clc;
clear all;
close all;

%input data
p = [0011;0101];

%target data
t=[-11-11];
%creating a feedforeward neural network
net=newff(minrnax(p), [2,1]);
%creating two layer net with two neurons in hidden(1) layer
net.inputs{l}.size = 2;
net.numLayers = 2;

%initializing network
net= init(net);
net.initFcn = 'initlay';

%initializing weights and bias

net.layers{1}.initFcn = 'initwb';
net.layers{2).initFcn = 'initwb';
%Assigning weights and bias from function 'gawbinit'
net.inputWeights{1,1).initFcn = 'gawbinit';
net.layerWeights{2,1).initFcn = 'gawbinit';
net.biases{l}.initFcn='gawbinit';
net.biases{2).initFcn='gawbinit';

%configuring training parameters

net.trainParam.lr = 0.05; %learning rate
net.trainParam.min_grad=Oe-10; %min. gradient
net.trainParam.epochs = 60; %No. of iterations
%Training neural net
net=train(net,p,t);

%simulating the net with given input

y = sim(net,p);
disp('The output of the net is : ');
disp(y};
~plating given training data and anfis output
plot(p,t,'o',p,y, '*');
title('Training data vs Output data');

I~
504
Hybrid Soft Computing Techniques

%calculating error rate

e= gsubtract(t,y); % e=t-y
disp('The error(t-y) of the net is :');
disp(e);

%program to calculate weights and bias of the net

function outl = gawbinit(inl,in2,in3,in4,in5,-)

%%=========================================================================
%Implement~ng genetic algorithm

%configuring ga arguments
A= []; b = []; %:linear constraints
Aeq = [); beq = []; %linear inequalities
lb = {-2 -2 -2 -2 -2 -2]; %lower bound
ub = [2 2 2 2 2 2]; %upper bound
%plating ga parameters
options= gaoptimset('PlotFcns',{@gaplotscorediversity,@gaplotbestf});

%creating a multi objective genetic algorithm

%number of variables , for 2 layer 1 output 5 neuron net there are

%6 weights and 3 biases(6+3=9)
nvars=9;
[X,fval,exitFlag,Output]=gamultiobj{@fitnesfun,nvars,A,b,Aeq,beq,lb,
ub,options);
figure

%displaying the ga output parameters

disp(XJ;
fprintf('The number of generations was : %d\n', Output.generations);
fprintf('The number of function evaluations was : %d\n', OUtput.funccount);
fprintf('The best function value found was %g\n', fval);

%%=========================================================================

%Assigning the values of weights and bias respectively

%getting information of the net
persistent INFO;
if isempty(INFO), INFO= nnfcnWeightinit(mfilename,'Random Symmetric',
7.0, ... true,true,true, true,true,true,true, true); end
if ischar(inl)
switch lower(inl)
case 'info', outl =INFO;

%configuring function
case 'configure'
outl = struct;

......;:
16.7 Solved Problems using MATLAB 505

case 'initialize'

%selecting input weights , layerweights and bias separately

switch(upper(in3))
case {'IW') %for input weights
if INFO.initinputWeight
if in2.inputConnect(in4,in5)
x=X; %Assigning ga output 'X' to input weights

%Taking first 4 ga outputs to cFeate input weight matrix 'wi'

wi(l,l)=x(l,l);
wi{1,2)=x{1,2);
wi(2,l)=x(l,3);
wi(2,2)=x(1,4);
disp(wil;
outl = wi;%Returning input layer matrix
else
outl = [];
end
else
nerr.throw([upper(mfilename) ' does not initialize input weights.']);
end
case {'LW'} %for layer weights
if INFO.initLayerWeight
if i~2.layerConnect{in4,in5)
x=X; %Assigning ga output 'X' to layer weights

%Taking 7th and 8th ga outputs to create layer weight matrix 'wl'
wl(l,l)=x{l, 7);
wl{1,2)=x(l,Bl;
disp (wl);
outl = wl;%Returning layer 11eight matrix
else
outl [];
end
else
nnerr.throw([upper(mfilename) ' does not initialize input weights.']);
end
case {'B'} %for bias
if INFO.initBias
if in2.biasConnect{in4)
x=X; %Assigning ga output 'X' to bias

%Taking 5th, 6th and 9th ga outputs to create bias matrix 'bl'
bl[l)=x{l,5);
bl[2)=x[1,6);
bl [3) =x(l, 9);
disp(bl);
outl = bl;
else
 506 Hybrid Soft Computing Techniques

outl [];%Returning bias matrix

end
else
nnerr.throw([upper(mfilename) ' does not initialize biases.']);
end
otherwise,
nnerr.throw('Unrecognized value type.');
end
end
end
end
%Creating fitness function for genetic algorithm
function z = fitnesfun(e)
%The error(t-y) for all 4 i/o pairs are summed to get overall error
%For 4 input target pairs the overall error is divided by 4 to get average
%error value (1/4=0.25)
z=0.25*surn(abs(e));
end
Output
' Optimization terminated: average change in the spread of Pareto solutions
less than options.TolFun.
Columns 1 through 7
0.0280 0.0041 0.0112 0.0069 0.0050 0.0062 0.0075
Columns 8 through 9
0.0018 0.0003
I
The number of generations was : 102
The number of function evaluations was : 13906
The best function value found was : 0.0177734
Optimization terminated: average change in the spread of Pareto solutions
less than options.TolFun.
Columns 1 through 7
0.0012 0.0020 0.0096 0.0014 0.0018 0.0044 0.0084
Columns a through 9
0.0084 0.0025
The number of generations was 102
The number of function evaluations was : 13906
The best function value found was 0.00988699
The output of the net is :
-1.0000 1.0000 -1.0000 1.0000
The error(t-y) of the net is :
1.0e-011
-0.3097 0.2645 -0.2735 0.3006

Figure 16~22 shows the plot of the generations versus fitness value and histogram. Figure 16-23 illustrates the
Neural Nenvork Training Tool for the given input and output pairs. Figure 16-24 shows the neural network
training p.erf'Oimance. Neural necwork training state is shown in Figure 16-25. Figure 16-26 displays the
performance of uaining data versus output data.
"l
16.7 Solved Problems using MATl.AB 507

I '* 40
Score histogram

"~" 30
".50 20
;;;
"E
z"
10 10.02
3
Score (range) X 10-

Best: 0.009887 mean: 0.0099592

1.5
m

""
>
m
m
m
~ 0.5

0
o 200 400 soo aoo 1ooo 1200 1400 1soo 1aoo
~[Pause[ Generation

Figure 1622 Plor of the generations versus fimess value and hiswgram.

f Ne1.1r"l Network-

Algorithm.
Tr.,ifling:- Lev~b,..-g-M..-q_l.lardl (H~nh.>)
Poert'o<meno:;o:: M..onSqUD.SErrot (.,--,,~1
P<orlv;otlve: Od'~l.llt (detnu!tdN"'l

i>rogrcu
Epoch! 0 1- '~ ; ,_ __ .tiO.Itcratio(l's_>~,-:;_,;i>>~<:O;' .j 60
Time<
. .
0;011;01
---
.
""'
P<!rfonnano:e: 1~7

l' - I '""
Gndlotnt: 2.32
M... 0.00100 1.00~20 l.OOe .. lO i
'llidation Chlu< 0 ! .

Piouo- -

i' 1-.-Pttdorma1c:!y-l {plotp~iorrnl

; I: ....,.TDI)!I.~~-..;1 (plotlnolnn.o<<l)
~ h-~rtRecrn:DIP!h1.-,-.l (ploUt:::!jlltc.,.,onJ

i Plotln~. ~~=---- --~ 1 epochs-

. , Mol"""""" ..,....,h ro.d..od.

4:!1' ':-'<' 1 looon"'~- .J (.a C"'><:
Figure 1623 Neural Network Tr:Uning Tool for rhe given input and ourp-ur pairs.

i.'~'
508 Hybrid Soft Computing Techniques

Figure 1624 Neural network training performance.

Figure 1625 Neural network training state.

\.
A
16.9 Exercise Problems 509

Training data vs output data

0.8

0.6

0.4

0.2

-0.2

-0.4

-0.6

-0.8

-1 c:~
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Figure 1626 Performance of training clara versus output data.

I 16.8 Review Questions

1. State the limitations of neural nerworks and fuzzy 6. How are generic algorithms utilized for optimiz-
systems when operated individually. ing the weights in neural nerwork archirecmre?
2. List the various cypes of hybrid systems. 7. Explain i:1 derail the concepts of fu1.zy generic
3. MelHion the characteristics and properties of hybrid systems.
neuro-fuzzy hybrid systems. 8. Differentiate: ARTMAP and Fuzzy ARTMAP,
4. What are the classifications of neuro-fuz:z.y Fuzzy ARTMAP and back-propagation neural
hybrid sysrems? Explain in derail any one of the. nerworks.
neuro-fuzzy hybrid systems. 9. Write nares on the supervised fuzzy ARTMAPs.
5. Give derails on the various applications of ncuro- 10. Give description on the operation of ANFIS
fuzzy hybrid systems. Editor in MATI.AB.

116.9 Exercise Problems

1. Write a MATLAB program m train NAND gate 2. Consider some alphabeS of your own and recog-
wirh binary inputs and targeS (rwo input-one nize rhe assumed characters using ANFIS Editor
Output) using adaptive neuro-fuzzy hybrid tech- module in MATLAB.
nique.
 \\'I

510 Hybrid Soft Computing Techniques

3. Perform Problem 2 for any assumed numeral an OR gate wiili 2 bipolar inputs and 1 bipolar
charaaers. . targets.
4. Design a generic algoriilim to optimize the 5. Write a MATLAB Mfile program for the working
weights of a neural network model while training of washing machine using fuzzy genetic hybrids.
If

Applications of Soft Computing 17

Learning Objectives - - - - - - - - - - - - - - - - - - ,
Deals with the various applications of soft Provides knowledge to develop hybrid fuzzy
computing in derail. comrollers using soft computing techniques
Discuss how SAR image are solved using Derails how pocket engine comrol is acheived
neural ne[Work approach. using various soft computing methods.
Gives a methology for solving traveling sales
man problem and Imernetbased search using
genetic algorithms.

111.1 Introduction
In this chapter we are going to discuss few applications of neural nernrorks, fuzzy logic, generic algorithm and
hybrid systems. As we already know soft computing has a wide range of applications. Here a few topics of irs
applications are being covered. We believe that the chapter would give the reader a brief idea of how the soft:
computing can be applied to any practical problem.

17.2 A Fusion Approach of Multispectral Images with SAR (Synthetic

Aperture Radar) Image for Flood Area Analysis
There have been several efforts to monitor and assess the area destroyed by floods, especially, the monsoon
regions that were suddenly inundated by slash flood caused by the storm and other natural hazards, such as
El Nino, LA Nina I. etc. Floods cause much damage to the environment, people's live and properties. Several
techniques have been applied to estimate the flood area, important ones being the NDVI (normalized differ-
encevegetation index) derived from multispectral data and 3-second grid DEMs (digital elevation models) to
investigate and identify the damaged area depending on elevation intervals. However, the SAR images have
been known to efficiently detect floods, because of the object absorption property depending on the moisture
of rhe back.scattering wave in radar image. As the multispectral images provide necessary information for land
cover interpretation, the fusion of multisensor images achieves the complememary narure of these different
data types. Therefore, the fusion techniques have been adopted to perform the flood area classification. To
assess the flood areas, JERS-1 SAR data acquired on June 3, 1997 and August 30, 1997 (Figures 17-1 and
17-2) were taken before and during the flood hazard from the tropical storm Zira in Surat Thani province.
The cloud penetration capability is shown in Figures 17-1 and 17-2, respectively. Fusion of these images with
JERS-1 OPS data acquired on March 14, 1997 (Figure 17-3) was performed ro distinguish flood area.

i
I:
512 Applications of Soft Computing

Figure 17~1 SAR image acquired on June 3, 1997, Surat Thani Province.

Figure 17-2 JERS-l SAR image acquired on August 30, 1997.

IL.

Figure 173 JERS-1 OPS data acquired on March 14, 1997.

I

1 .

.
 ..
17.2 A Fusion Approach of Multispectral Images 513

To study the flood assessmem, the image classification is performed using a revolurionary computing
methodology known as the artificial neural network (ANN) computing. This method presents how the
neuron in the human brain processes the data to identifY ~e complex and noisy patterns of information. An
error back-propagation ANN structure is used in this secri_9n.

I 17.2.1 Image Fusion

Image fusion integrates both spatial and spectral data ro hold the superior characteristics of rr:ultisensor
images and improve rhe knowledge of scene. Therefore, the: fused images could improve: rhe accuracy image:
classification and help the: fe:amre extraction and recognition. The: image: fusion can be: divided into rwo I'
classes: spatial domain method and spe:crral domain method. The second method is used in mos~ applications
including color space transformation. In this se:ccion, the: Ime:nsiry-Hue:-Sarurarion (IHS) mo4el will be: used
"!
l!
'li
as a color space: and rhe image fusion is done as follows. ;
~:

l. The: RGB color space ofOPS images is transformed to the IHS model:

i=R+G+B
i
(17.1)
3
3
S= 1-
R+G+B
[min(R,G,B)] (17.2)
.,~\
1
H<R- G)+ (R- B)]
H =cos
-I {
(R- G) 2 +(R- B)(G- B)'
)
(17.3)
'\
2. The different gray value of pixel in the black-white of two SAR images {gl and g2) is added into OPS
images intensity: );
!' = I+ (g1 - g2) (17.4) 'l
.,
.,
The last term of the above equation is rhe difference of the images before and during flood. The flood area
will be emphasized and non flood area will be depressed. Adding this term to imensiry component in IHS
mode means transferring of flood area data to OPS image.
3. The IHS model is inversely transformed to the RGB space and is ready for furrher classification using
neural networks.

117.2.2 Neural Network Classification

In this section, the multilayer perceptron (MLP) neural nerwork based on back propagation (BP) algorirhm
is used as a classifier, which consists of set of nodes arranged in multiple layers with connection only between
nodes in adjacent layer by weights. The input information is presenred at input layer as the input vecror,
and the output vector is the processed information that was retrieved at the outpur layer. A schematic of a
three-layer MLP model is shown in Figure 17-4.
The input and output of the node I in hidden layer ofMLP neural network, according to BP algorithm, are:

Input: X;= L WxOj+ bi (17.5)

j
Outpm: 0; = j(X;)

1
 514 Applications of Soft Col!lputing

Upper
layer

Figure 174 The rhree-layer MLP model of neural nerwork.

where Wij is theweighr of connection from node ito node}; B; the numerical value called bias;frhe activation
function. In this work, rhe nonlinear function- sigmoid function given in Eq. (17.6)- is used m determine
the omput stare:
'
~
f(x;) = I
I
+ ,-,,; (17.6)
\.
The BP learning algorithm is designed to reduce an error beWeen acrual output and desired ourpm in a
gr:~.diemdescem manner. The summed squared error (SSE) is defined as:

SSE=~2 L L. (Op;- tp;l' (17.7)

p '
I
where Opi and fpi are rhe acwal and desired outputs of node i when applying the input vector p imo rhe
;: ner;work.

I 17.2.3 Me1hodology and Results

17.2.3. 1 Method
1. The SAR data obtain 16 bits and then are reduced to 8 bits by using linear scaling in order ro obrain
256 values of imensicy. From wavelet decomposition, the low wavelet coefficient of SAR images will be
used for twO reasons- ro remove the speckle noise and ro conrinue the proper data for applying to neural
network training algorithm.
2. The 12.5 m x 12.5 m resolution ofSAR clara was reduced to 25m x 25m in the same order ofOPS
resolution. All images should be registered and geometrically corrected.
3. Data fusion technique as mentioned in Section 17.2.1, is used and is shown Figure 175.
4. After preprocessing, satellite image is prepared and then applied to neural network classification. Moreover,
all clara will be classified without fusing and the results will be compared against the fusion data.

17.2.3.2 Results
The results of flood assessment by neural network classification with clara fusion and without data fusion are
givenin.Tables 171 and 172, respectively.
-;;r

17.3 Optimization of Traveling Salesman Problem 515

Figure 17-5 Fused image for flood area assessment.

Table 171 The resulr of flood assessment by neural network classification wirh clara fusion
Urban Vegetation Bare soil
Classification Water Cloud Flood Nonflood Flood Nonflood Flood Nonflood
Result/testing (pixel) 447/500 47/50 41/50 42/50 44/50 901!00 41/50 87/100
Correction (%) 89.4 94.0 82.0 84.0 88.00 90.0 82.0 87.0

Table 172 The result of flood assessment by neural nerwork classification without clara fusion
Urban Vegetation Bare soil
Classification Water Cloud Flood Nonflood Flood Nonflood Flood Nonflood
Resulthesting (pixel) 457/500 45/50 38/50 40/50 4!150 92/100 37/50 89/100
Correction(%) 9l.4 90.0 76.0 80.4 82.0 92.0 78.0 89.0

The srudy resuhs show thar mulritemporal SAR clara are very useful for flood assessment and moniroring.
On the other hand, the OPS dara provide rhe necessary informarion for land cover imerpretation. The fusion
of these data is very helpful for flood assessment classification, because it enhances the flood area and gives a
highly reliable result.

17.3 Optimization of Traveling Salesman Problem using Genetic

Algorithm Approach
i The traveling salesman problem (TSP) is conceptually simple. The problem is ro design a tour of a set of
I n cities in which the traveler visits each city exacdy once and then rerurns to the starting poinr. One has
to minimize ilie disrance traveled. Solving this combinatorial problem is NP (nondeterministic polynomial
time) difficult as ilie search space is n facrorial. Examining all possible Hamiltonian circuits of n vertices by
calculating edge weights is certain to reveal the optimum solution, but cannot guarantee to do so in a rractable

I rime for all n.

Perhaps first considered by Euler as the knighrs' rour problem in 1759, and popularized by the RAND
Corporation in the 1950s, TSP has many applications that involve large numbers of venices. These include
VLSI - for which size can exceed one million - circuit board drilling, Xray crystallography and many
l.

l
516 Applications ol Soft Computing

1.2

0.8

0.6

0.4

0.2

0
0 0.2 0.4 0.6 0.8

Figure 176 Poin(S placed in a unit square represenring a landscape of 14 cities.

scheduling problems. Therefore it is imponanr to find algorithms that lower the time coSts by providing
reasonably good solutions.
This section explores application of GAs to TSP by examining combinations of different algorithms for the
binary and unary operators used to generate better solutions and minimize the search space. Three binary and
two unary operations were rested. These were compared to a base-line developed from a brute force algorithm
for a tractable I4~city problem and to random tour generation, which provides mean and standard deviation
statistics very close to brute force methods as the distribution of solutions is normal. For a 14-ciry problem,
Figure 17-6 shows the points placed in a unit square representing a landscape of 14 cities.
The binary methods examined include uniform order~based crossover (OCX), heuristic order-based
crossover (HCX) and edge recombination (ER). Unary operators were reciprocal exchange and inversion.

117.3.1 Genetic Algorithms

Genetic algorithms are modeled on biological processes in which parents pass character traits to their offspring.
The next generation contains data inherited from irs predecessors and in each generation the finest members
have the greatest potential to survive and send genetic material to the progeny of their population. As children
are developed from the best parents, they are likely m introduce an improvement in fitness of the group.
Genetic algorithms mimic this survival of the finest by randomly generating a population of solutions and
then selecting members, with greater possibility of seleaion given to the finest, from which tO build the next
generation. This section used populations of 100 subjects and evolved each trial for 1000 generations. The
objective function used in rhis section was the length of the paths. Shaner circuits were given best fitness
consideration by inverting their tour lengths cluough subuaction from the ceiling of the population's longest
tour. The roulette wheel method was employed to choose parent solutions.

,.
Successors were developed by binary operations called crossovers that create child solurions using informa~

..
tion inherent in the two chosen parents. The cype of information passed is problem dependent and affects the

-""
fitness of the resultant population. These experiments rested three crossover operators and in all cases allowed

'"'"' _, "'"""' ,. ~".

.
I
 ~I'
,.
1:~i'
:r
517
17.3 Optimization of Traveling Salesman Problem
~

I
Unary operators mutate individual solutions and are applied at a low rate. In this case two mutation
.
methods were employed at a rare ofO.Ol. Their purpose is to allow solutions to exceed local maximums bur
.
they are usually destructive to the mutated offspring.
;i
I 17.3.2 Schemata

The solutions created by genetic algorithms are instances of schema. They belong to a set of other solutions
i,,,,
rhat share common traits. A solution [0 1 0 0] is an instance of the schema [0 0], where the asterisks may ]
represent either bit value. The solution [0 1 I 0] would also be an instance of this schema of order two, which i~
could be thought of as a regular expression representing all srrings of length four over the alphabet {0, 1) ~:
t
beginning and ending with zeros. The fitness of a schema is the average of the fitness's of its instances. '
The probability that a schema S found in one generation will occur in the next is given by,

Pf' = I - Pcx[L,I(L- l)]

where 1 - Pcx is the rare at which cloning occurs and Ld is the defining length of the schema, or the
number of bits between the outermost set bits, and L is the totallengrh of the solution. For our purposes
PfX =' I - 0.70(,113}.
The probabiliry of remaining in the population after the mutation cycle is given by

P~ = (I - PM)''
.,
~\

!
where I -PM is the probability that a bit will not undergo mutation and n is the order of the schema, or the
number of ser bits. In the..~e experiments Pt:
= 0.99". '\
The success of generic algorithms lies in rhe propagation of the finesr schemata.

I 17.3.3 Problem Representation )

!
All reproduction and mutation operators rested employed a path represenrarion of rhe problem. This implies
that a list of cities (l 2 3 4) would represent the circuit I-2-3~4-1. This representation was chosen over others :~
such as adjacency, ordinal or matrix, because ir is a very natural representation and accommodates many
crossover functions that result in valid offspring. Hence it eliminates rhe need for repair algorithms or penalry
functions.
Two of the binary operators rested augment rhis representation with additional information. The heuristic
order-based crossover includes a list of the distances between each leg of rbe tour. The edge recombination
crossover method utilizes edge lim for each city that are complied from rhe parent information.

I 17.3.4 Reproductive Algorithms

The first binary operator tested was the uniform-order-based crossover, which is useful when order is significant
to a problem, and preserves rhe legality of solutions. This method first creates a random binary string which
is of the same length as the parent tours. The child receives infOrmation from parenr one in all positions
corresponding to I in the binary string. On average 50% of the data then comes from parent one and reflects
the positions of cities in this path. The rest of the tour is filled in from parent two using those cities not already
in the child and having the order found in parent two. For example, if parent one is (1 2 3 4), parent rwo is
(4 3 2 1} and the binary string is (I 0 1 0), then the child becomes (1 4 3 2).
The next crossover method employed was modeled on heuristic crossovers developed for adjacency rep-
tesemarions, which favor edges with more desirable fitness values. This operator employed a list of distances


518 Applications of Soft Computing

berween edges of the tours such that a path (I 2 3) with a distance map of (0.2, 0.8, 0.3) would correspond
ro edge (1 2) having disumce 0.2, edge (2 3) having distance 0.8 and edge (3 1) having diSa.nce 0.3. The
offspring created receive edge and not position information for an average of SO% of their data. A roulette
wheel function gives greater probabiliry for selection to more fir edges. The rest of the information is filled by
using the cities not already passed to the child in the order found in the remaining parent. Here shoner legs of
a tour are tre;ued like dominant genetic traits. They are fearures that are more likely to be passed to the child.
The final binary operator tested focused on passing as much edge informacion as possible to the child. Edge
recombination (ER) uses no heuristic rules or fitness information, but insures iliat each offspring receives
95% of their edge information from ilie parents. ER uses edge maps such iliat if parenr one is (1 2 3 4 5) and
parent two is (5 2 3 4 I), then mapping is performed as given below:

1:(254)
2: (1 3 5)
3: (24)
4: (3 5 1)
5: (4 1 2)

An initial city is chosen randomly between ilie first cities of the parents and placed in the offspring as the
current city. The next city chosen is taken &om the edge list of ilie current city giving priority to cities with
shorter edge lists, ilie largest list possible being of lengili four. Ties are broken randomly and used cities are
removed &om choice available. In the event of edge failure, defined as a currem city of edge list length zero,
the next point is chosen randomly from the cities nor yet visited.

I 17.3.5 Mutation Methods

The first unary operator tested was reciprocal exchange, which simply swaps nvo randomly chosen elements
in the solution. Inversion was the other meiliod applied. Here all values between cwo elements in a lisr are
reversed. A solution (1 2 I 3 4 5 6 I 7), where the bars represent random break points with a window size of
four, would produce the mmam (1 2 6 5 4 3 7).

I 17.3.6 Results

Examining measures of center for the data distributions, OCX was the overall best binaryoperarion performer
with inversion mutation method. HCX also performed well with the inversion operator. All genetic method
distributions were skewed toward the shorter tour lengths with OCX showing ilie lowest modes.
The ER crossover posted the lowest standard deviations of the GAs coming very close to baseline.
All methods produced excellent shonest path means with many trials finding the optimum solution. OCX
and ER, both using reciprocal exchange, found the optimum tour length with greatest frequency.
All generic operators produced good shortest path means in less than 37 s. The generational time to bcsr
solution was 186 generations using the ER binary operator with reciprocal exchange mmation. Since all
meiliods find good solutions quickly, the smallest number of generations necessary to produce near optimum
results is an important criterion. ER was consistently the best number of generatiOns to soludon performer.
While ER descended to best tour rimes the fastest, OCX performed better finding the optimum solution
and produced beuer results for all generations. HCX, which favored shoner edge lengths in child construction,
did not perform as well as expected. Table 17-3 shows ilie scores obtained for various algorithms on an average
of 20 trials.
~

17.4 Genetic A!gorilhmBased Internet Search Technique 519

Table 173 Averages of 20 trials where Z scores represent the number of standard deviadom from the
exhaustive search baseline
Standard
Algorithms Mean Mode deviation Shortest path Longest path Z score lime (ms)
HCX/Inversion 4.94 4.38 !.07 3.78 10.29 -4.09 8389
HCX/Reciprocal exchange 5.05 4.50 LOB 3.83 10.58 -3.95 7954
OCX/Inversion 4.78 3.88 !.04 3.87 10.36 -4.29 6796
OCXJReciprocal exchange 5.02 3.88 !.07 3.77 10.61 -3.99 6821
ERllnversion 5.00 4.62 0.81 3.81 10.21 -4.01 36280
ER/Reciprocal exchange 5.11 4.12 0.87 3.77 10.26 -3.88 36310
Random tours 8.21 8.38 0.81 4.23 10.99 0 6440
E.xhausrive search' 8.21 8.38 0.80 3.76 1!.08 0 366
'Only one trial.

An examination of the schemata of some final populations reveals that HCX produced the most homoge-
nous solutions. Total population schemata of order seven, 50% of the total tour lengrh of 14, were not
uncommon. HCX schemata had much in common with greedy algorithm solutions. The effect of rhis oper-
ator was to favor a more narrow set of fit schemata rather than to produce shorter solmions more quickly.
ER produced the most diverse final populations, with most final population schemata at order zero. While
edge information is significant in TSP solutions, the relative positions of points within the solution also play
a strong role in finding optimum values. The success of the OCX crossover, which passes relative position
information to the offspring, is an evidence of rhis.
The inversion mutation operaror performed well in center for data distribution, but the reciprocal exchange
method performed well in optimum solutions. This operator acrually introduces more variety to a population
than inversion in TSP because the direction of a path does nm change its length. Only the values at rhe nvo
edges disrupted are changed when inversion is applied. Reciprocal exchange changes at minimum rwo and
maximum four edge values.

117.4 Genetic Algorithm-Based Internet Search Technique

Among the huge number of documents and servers on Internet, it is hard to quickly locate documents that
contain pmentially useful information. Therefore, rhe key facwr in sofnvare developmenr nowadays should be
the design of applications that efficiently locate and retrieve Internet documents that best meet user's requests.
The accent is on intelligent content examination and selection of documenrs that are .most similar to rhose
submitted by the user as input.
One approach to this problem is indexing all accessible web pages and storing this information into
the database. When the application is starred, it extracts keywords from the user-supplied documems and
consults the database to find documents in which given keywords appear with the greatest frequency. This
I approach, besides the need to maintain a huge database, suffers from the poor performance- it gives numerous
documents totally unconnected to the user's topic ofinreresr.
The second approach is to follow links from a number of documents submitted by the user and to find
I the most similar ones, performing a generic search on the Internet. Namely, application starts from a ser of
input documents, and by following their links, it finds documents that are most similar ro them. This search
I and evaluation is performed using GAs as a heuristic search method. If only links from input documents
are followed, it is the Best First Search or generic search without mmation. If, besides rhe links of t.he input

l
documents, some other links are also examined, it is generic search with mutation.
520 Applications or Soh Computing

The second approach was realized and tested at the University ofHong Kong, and the results were presented
at the 30th Annual Hawaii Imernarional Conference of System Sciences. The Best First Search was compared
tO genetic search where mutation was performed by picking a URL from a subset of URI...s covering the
selected topic. That subset is obtained from a compile-rime generated database. It was shown rha,t search
using GAs gives better results than the Best First Search for a small set of input documents, because it is able
lO step outside the local domain and examine a larger search space.
There is an ongoing research at the Universicy of Belgrade concerning mutation exploiting spatial and
temporaJ localities. The idea of spatial locality exploitation is co examine documents in the neighborhood of
the best-rankeddocuments so far, i.e., the same server or local network. Temporal locality concerns maintaining
information about previous search results and performing mutation by picking .URLs from that set.
If either of above described methods is performed, a lot of time is spent for transferring documents from
the Internet onto Ute local disk, because content examination and evaluation must be performed off~line.
Thus, a huge amount of data is uansferred through the network in vain, because only a small percent of
uansferred documents will turn our to be useful. The logical improvement is construction of mobile agents
that would browse through the network and perform the search locally, on the remote servers, transferring
only the needed documents and data.
The first step ofa generic algorithm is to define a search space and describe a complete solution ofa problem
in the form of a data structure thai: can be processed by a computer. Strings and trees are generally used, but
any other representation couJd be equally eligible, provided that the following steps can be accomplished, too.
This solution is referred to as genome or individUd.l.
The second step is to define a convenient evaluation function (fitness Junction) whose task is to determine
what solutions are better than others. One approach, when meeting diverse requests, is to add a certain value
for every request met and subsrract another value for every ruJe violated. For instance, in a class~scheduling
problem, generic algorithm can add 5 points for every solution that has Mr. ]ones lecturing only in rhe
afternoon and subs tract I 0 for any one rhar has cwo lecturers teaching in the same classroom at rhe same time.
Of course, many problems require a specific definicion of the fitness function which works best in that case.
The third step in the creation of a GA is to define reproduction, crossover and mutation operators that
should transform the current generation into the next one. Reproduction can be generalized, namely, for every
problem one can pick out individuals for mating randoinly or according to their fitness function (only few of
the best are allowed to mare). The harder pan is to define crossover and mutation operarors.
These operators depend strongly on the problem representation and require thorough investigation, plus
a lor of experimenting to become truly efficient. Crossover generates a new offspring by combining generic
material from rwo parents. It incarnates the assumption rhat rhe solution which has a high fimess value owes
it to a combination of its genes. Combining good generic material from rwo individuals, better solutions can
be obtained. Mmation introduces some randomness into population. Using only a crossover operator is a
highly unwise approach, because it might lead to a siruation when one individual (in most cases only slighcly
better than others) dominates the popu1ation and the algorithm "gets stuck" with an averagely good solution,
and no way to improve ir by examining ocher alternatives. Mutation randomly changes some genes in an
individual, introducing diversity into population and exploring a larger search space. Nevenhless, a high rare
of mutation can bring oscillations in the search, causing the algorithm to drift: away from good solutions and
ro examine worse ones, thus converging more slowly and unpredictably.

'
The fourth srep is to define stopping criteria. Algoridtm can either stop after it has produced a definite
number of generations or when the improvement in average fitness over twO generations is below a threshold.
The second approach is becrer, yet the goal might be hard to reach, so the firsc one is more: resonable.
Having done all cl1is, one c;u:t write a code for a program performing the search (which is fairly simple at this
point).

1
'
'
 17.4 Genetic Algorithm-Based Internet Search Technique 521

The GA strength comes from the implicicly parallel search of rhe solution space that it performs via a
population of candidate solutions and this popuJation is manipu1ared in the simulation.
A fitness function is used to evaluate individuals, and reproductive success varies with fitness. An effective
GA representation (i.e., converting a problem domain into genes) and meaningful fitness evaluation are the
keys of the success in GA applications.
Although this mechanism seems "roo good to be true" ir gives excellent results, when compared to oilier
approaches, regarding rhe rime spent in search and quality of the solutions found. lI,,
I 17.4.1 Genetic Algorithms and Internet !

Basic idea in customizing Internet search is construction of an intelligent agent- a program char accepts a '
number of useHupplied documents and finds documents most similar to them on Internet. GA imposes
itself as "a right roo! for rhe job" since it can process many documents in parallel, evaluate them according to
their similarity to the supplied ones, and generate a result in rhe form of a group of documents found.
Inrelligem agent for the Internet search performs ~he following steps:

I. Processes a set ofUR.l..s given to it by a user and extracts keywords, if necessary, for evaluation.
2. Selects all links from rhe input set and fetches the corresponding www presentations; ilie resulting set
represents the first generation.
3. Evaluates the fitness functions for all elements of the set.
.,
:~

~
4. Repeatedly performs reproduction, crossover and mutation, and rhus transforms rhe current generation
into rhe next one. '\
There are several issues of importance cl1at have to be considered when designing a generic algorithm for
inrelligenr Internet search. These are:
)
l. represenrarion of genomes;
2. definition of the crossover operator;
3. selection of the degree of crossover;
4. definition of rhe mutation operator;
5. definicion of the fitness function;
6. generation of the output ser.

Each ofthe issues given above is described next, in the form of a classification of possible ways to implement
rhe issue. For each of the issues, two pictures are presented: a classification of possible approaches regarding
rhar issue and the most frequently used implementation.

17.4.2 First Issue: Representation of Genomes

First issue to be discussed is how one can encode possible solutions. In this case, one solution is URL, rhe address
of an Internet document. The aim is to create a result in the form of a list or an array of rhose documents, and
thar average fitness of chis set be the highest possible. Figure 17 ~ 7 gives the possible representation approaches.

17.4.2.1 String Representation

String represemation seems to be a narural choice since URL is already srring~encoded. However, in chis case,
there is only one gene in a genome, and classical crossover and mutation cannot be performed. Therefort.:, a new

1
'
 522 Applications of Soft Computing

Figure 177 A taxonomy of representation of genomes.

definition of the crossover and mutation operators is needed that would be applicable in this environment.
Nevertheless, redefinition of genetic operators seems to be the only reasonable thing m do, since classical
crossover and mutation can transform GA's search into a "random walk" through the search space.
17.4.2.2 Array of String Representation
Each URL c,ontains several fields that have different meanings, so it is convenient to represent it as an array or
list of these fields that are string-encoded .. Figure 17-8 shows the suing representation of a URL terminated
by a End-of-Suing character.
r First, there is a name of rhe Internet promcol either http or ftp. Only the URLs starring wiili "http" are
\., of interest to us. Therefore, the agent should take into consideration only rhe documems that these URLs
point to.
Second, there is a server address consiscing also of several fields: net name (www, ustnet, etc.), server name,
and additional information concerning the type of organization that the server belongs to (com for commercial
organizations, org for noncommercial ones, edu for universities and schools, and so on). In some cases, this
field can comain specification of the city and country_ the server is located in.
/ The third is the string that gives the path from the server root to a panicular document.
All these fields must be variable in length so that the solution can be represemed in the form of an array
of variable-length strings. Mutation and crossover operarors can be implememed more easily in this ClSe than
it was possible with the string represemation, because there are several genes in a genome that can be crossed
or mutated. Either classical or user-defined crossover and mutation can be performed.

17.4.2.3 Numerical Representation

Since every Imernet address up to the document path is number-encoded, genecic algorithm can use this
representacion in order to perform classical crossover and muration. However, this is flO[ a promising approach
since documents similar to one another seldom reside on adresses that have much in common. Therefore
genetic algorithm would actually perform a random search. Moreover, many of the addresses generated may
not exist at all, which places more overhead than can be-wlerated. Numerical representation can be eith(!r l~(t:
integer-encotkd or transformed into a bit-string representation. :~

J http:t/\www.altavis~:com J EOS J

J http://galeb.-etl~bg.ac.yul-vrnltutorial.html . J EOS J

Figure 178 The most frequent approach regarding the representation of-genomes: String representation . .URl
is represented as a suing, terminated by an End-Of-String character.
 17.4 Genetic AlgorilhmBased Internet Search Technique
523

Figure 179 A taxonomy of the crossover operator.

17.4.3 Second Issue: Definition of the Crossover Operator

Crossover operator is used to produce a new offspring by combining generic material from rwo parents, each
one characterized with a high fitness function. The idea is to force our the domination of good genes in the
furure populations. Figure 17-9 gives the possible definitions of the crossover operator.

17.4.3.1 Classical Crossover

Classical crossover can be performed only if URLs are encoded as array of strings or numerically, since it
requires that individual contains more than one gene.
It is performed by combining different fields of an address. This approach would, in most cases, produce
URls that do not exist on Internet. For example, combining www.msu.edu with www.novagenuica.com could
result in www.msu.com or in www.novagenetim.edu. Neither of rhesc is a valid Inrerner address. Though this
technique is slightly better than a random search, but is not a wise choice.

17.4.3.2 Parent Crossover

Parent crossover is performed by picking our parents from rhe mating pool and choosing a constant number
of their links as offspring, without any evaluation of those links. This approach is easy w conduct; however, it
could result in many nonrelevant documents being picked our for the nexr generation, because one document
can contain links m many sites that are not related to the user's subject of interest.

17.4.3.3 Link Crossover

Link crossover can, if carefully performed, produce more meaningful results rhan classical crossover. The idea
is to examine links from the documents in the mating pool and pick those that are the best for the nexr
generation. This evaluation of the links could be done in two ways: ouerlappiug links and link pre-evaluation.

17.4.3.4 Overlapping Links

I
1
The links of the parent documents are being compared to the links of the input documents, and only those
that they have in common are selected as offspring. This technique might not be the best one (it is always
I possible that the fittest document contains links that have nothing to do with the links of input documents),
bur can be easily conducted. Moreover, it is a common practice on lmerner rhar documents conrain links tO

i
related sites, so this approach could score high in most cases .
.

. .
 524
Applications of Soft Computing

~a.
Parents

PotenUal
offspring
0,1 0,3 I 0,9 I 0,93 0,2 I 0,8 I 0,9
I I I I

I
I '
I
I ' I
I
I
I '
Selected
offspring
00 00
Figure 1710 The most frequent approach regarding the crossover operawr: Link pre-evaluation. Numbers
next m the nodes represent normalized values of lheir firness functions. Offspring wirh rhe
greatest values are selected for the next generation.
17.4.3.5 Link Pre-Evaluation
Computation of the fitness function can be performed on rhe documems that parem links refer ro, and rhe
best ones can be picked out for the next generation. Since rhis compmarion must be done sooner or later, it
places a small overhead on clte program (because of rhe evaluation of the documems thar will nor be picked
out for the next generation), but it gives good resuhs. However, this approach could be rime-consuming
in the case when ilie documents in the mating pool comain many links, since genetic algorithm must wait
for all documents that those links refer to, w be fetched and evaluated in order to proceed with irs work.
Figure 17-l 0 shows ilie approach of link pre-evaluation operator.

17.4.4 Third Issue: Selection of the Degree of Crossover

There are two different approaches regarding crossover and insertion of offspring imo rhe nexr generation.
Figure 17-11 gives possible approaches concerning rhe degree of crossover.

17.4.4.1 Limited Crossover

Only a fixed number of offsprings can be produced from each couple. This could result in rejection of
documents thar have higher fimess values than offsprings of orher nodes bur are ranked less rhan second
among offspring documents of their parenr node.
i

17.4.4.2 Unlimited Crossover

Genctic-afgorirhm can rank the documents from the mating pool and all documents rhar parent links refer
mgeilier according to ilie values of their fitness function. Then, it can pick from this set those individuals that

I
can be forwarded to the next generation. Overall fitness would be better and there is no risk of losing some
good solutions.

I
I
Figure 1711 A taxonomy of the degree of crossover.

. I
17.4 Genetic Algorithm-Based Internet Search Technique 525

i
.if. .J).
Parents

Potential L
,
!

oo
oHspring
0,1 0,3 0,4 0,5 0, 7 0,8 0,9
Ranked
solutions
0,1 0,2 0,3 0,4 0,5 ,'0,6 /0,7/0,8 ,'0,9
I I I I
I I I I
I I I I

~~~~
Selected
oHspring

Figure 1712 Unlimited crossover.

The most frequent approach regarding the selection of the degree of crossover is the Unlimited crossover.
In this case parents and all offspring are ranked according to their fitness function values and the best among
them are selected. Thus, instead of picking out nodes with rhe fitness function values of0,4, 0,5, 0,8, 0,9, as
would be done using limited crossover, only the best solutions are chosen for the next generation. Figure 17-12
shows the operation of unlimited crossover.

17.4.5 Fourth Issue: Definition of the Mutation Operator

Mutation is used to introduce so,ne randomness in the population and rhus slow down the convergence and
cover more of the search space. Figure 17-13 gives rhe possible definitions of the mutation operator.

Figure 1713 A taxonomy of the mutation operamr.

 :;;:.

526 ApplicationS of S~_ft Com~ting

1Z4.5.1 Generational Mutation

Generational mutation is performed by generating a URL randomly. It is easily conducted, bur has no
significance since high percent of UR.I.l; generated in this way would nor exist ar all. So, conclusion is chat
URLs for mutation must be picked out from some set of erisrant addresses, such as database.

1Z4.5.2 Selective Mutation

Selective mutation is performed by selecting URLs from a set of existing URis. It can be either DB-based or
semantic.
DB-based: DB-based mutation is based on the existence ofa database rhar contains URLs that are somehoW
soned. Few of them are picked our and inserted into rhe population. The URis can be the following.
1. Unsorted - generic algorithm picks any one of them. This approach usually does nor promise good
performance.
2. Topic sorted- there is a field rhat says ro which topic URL belongs, i.e., enrenainment, politics, business.
GA chooses only from the set ofURLs dm belong to rhe same topic as the input URLs. This approach
is a bit limited since one document can cover several topics but should produce reasonably good scores.
Figure 17-14 depicts ropic-sorred DB-based mutation.
~~~ 3. Indexed- there is a database rhat contains all words that appear in documents with a certain frequency
and also links to documents in which they appear. GA writes a query tO a database with keywords from
input documents and picks out URLs for mutation from the resulting ser. This requires some effort for
' implementation and updation of the database bur promises good scoring. All one has w worry about is
finding a compromise between database size and quality of search results.

Semantic: Semantic techniques use some logical reasoning in order to produce URLs for mutation.

I . 1. Spatial kcality mutation: If GA finds a documem of a high fimess value on a particular sire, there is a
strong possibility that it can find similar documents somewhere on the same server or on the same local
network. This is because many people that have accounts on the same server or network usually have
similar interests (which is most likely for academic networks). This approach is a bit hard ro conduct since
GA has to either examine all sites on a server/net (which is time-consuming) or randomly pick a subset of
them.
2. Temporal IDeality mutation: A database is maintained of a huge number of documents rhat were in the
result set, for every search made. GA keeps scoring them on how frequemly they appear in chat set. Those
with high frequency promise to give good performance in the future too, so GA insens them in tht:

Offspring

Figure 1714 The most frequent approach regarding the definition of the mutation operator: Topic-sorted
.~ DB-based mutation. Offspring are selected from the set of the documents in a database that
are related to a certain topic.
 527
17.4 Genetic Algorithm-Based Interne! Search Technique

population, thus performing the mutation. This will yield good results for usual queries (from the field
that many users are interested in) but will do poor for less popular ones.
3. Type Wcttlity mutation. This mutation is based on it ryp~ of rhe sire the input documents are located on.
If it is, say, an edu sire then there is a suong probability ~at some other sites with same suflX have similar
documents. A database is maintained containing types of sites and a set of URLs referencing those sires
and GA chooses the candidates for mutation from this set.
Although last cwo rypes of mutations deal with databases, they involve logical reasoning and semantics
consideration in picking our URLs for mutation, and therefore are not classified as DB-based.

I
-- 17.4.6 Fifth Issue: Definition of the Fitness Function

To evaluate fimess of a document, GA must go through it and examine its contents. Figure 17-15 gives several
possible defmitions of the fitness function.

17.4.6.1 Simple Keyword Evaluation

Once the occurences of keywords (selected at the beginning from input files) in the document are counted,
the GA can simply add those numbers and add a certain value when cwo or more keywords appear in the
document (so it ranks it higher than those documents that contain only one keyvvord) .It can also add a value
for occurences of keywords in a tide or hyperlinks. This method, though rough, can produce fairly good
results with minimum time spem for evaluation.

17.4.6.2 Jaccard's Score

Jaccard's score is computed based on links and indexing of pages as follows:

Jaccard's score from links: Given two homepages x andy and their links, X = Xt ,x1, ... ,xn, and
Y = Yl ,y2, ... ,y , the Jaccard's score between x andy based on links is computed as follows:
11

#(Xn YJ
)SHok> = #(XU Y)
Jaccard's score from indexing: Given a set of homepages, the terms in these homepages are identified
(ke)"Nords). The term frequency and the homepages' frequencies are then computed. Term frequency, ifxJ

\I represents the number of occurences of term j in homepagex. Homepage frequency, d[j, represents the number

II
Simple
keyword
I evaluation

'J.. .
'
Figure 1715 A raxonomy of rhe fimess function .
 ~T
528
Applications of Soft Computing

of home pages in a collection of N homepages in which term j occurs. The combined weight of term j in
homepage x, ~-,is computed as follows:

d.,= if, 'log(~* Wj)

where Wj represents the number of words in term j and N represents the mral number of homepages. The
Jaccard's score beMeen homepages x andy based on indexing is then computed as follows:

L
'd,gdyj
j=l
JSindexing(x,y) = L L L

'd.,'+ '<4i' + 'd,g<4i

j=l j=l j=l

where Lis the total number of terms. Fimess function for homepage h; is then computed as foltows:

l N
]Slinks(h;) = NL)slinks0npur1, h;)
j=J

I N
JSindes.ing(h;) = N LJSindexing(inpu~, h;)
j=l

Fitness function is defined as

I
]S(h;) = 2(JS,;,~(h;) +]S;,<b;,,(h;)] [
Although computation of rhis fitness function can be time-consuming for a big population, it gives
excellent results concerning quality of homepages reuieved. Figure 17-16 shows the use of Jaccard's score as I
an evlauation function.
I

- lnpul documents
l.
!
S Potential offsprings
0.4 0.3
e Output documents

01 Jaccard's score
0.6 Iteration number
I_

Figure 1716 The most frequent approach regarding the fitness funccion: JaccardS score. The figure illustrates
I
1
Best First Search performed using Jaccard's score as the evaluacion function.

1
.

'.
 II
17.5 Soft Computing Based Hybrid Fuzzy Controllers 529
~~
1. Population 0 0 e ____ The output
0,1 0,2 0,4 ----- -
jl
2.Population 0 0,300,5
0 ------- -,..o,4
3.Poputalion 0
0,8
0,7 ---- &
------
0,85 0,9
7
-._o,
------0,9
j
!i

Figure 17~17 The most frequent approach regarding rhe generation of the output set: Interactive generation.
Besr individuals from each generation are selected for the .omput ser.

17.4.6.3 Link Evaluation

Documents can be evaluated according to the number of links that belong to the ser of links of input
documents. This narrows rhe search space but can produce good results in most cases and is easily implemented.

17.4. 7 Sixth Issue: Generation of the Output Set

Resulting output set can be either intemctively gmerated or postgenerated.

~
>
17.4. 7.1 Interactive Generation
From each generation of individuals one or few of them that have highest fitness values are picked out and
'
insetted into rhe result set. Thus, solutions from earlier generations that are inserted in the output set disqualify
later ones that are just below the line for inserrion. An advantage is that the user is not required to wait for
\
the end of the search, but can view docurilents found so far, while rhe search is performed. Sometimes it is
possible even to modify some parameters during the search, i.e., add new input documents or new keywords. I
Figure 17-17 shows the process of an interactive genemion.

17.4.7.2 Post-Generation
The final population is rhe one that represents the last generation and is declared ro be the result ser. The
quality of the documenrs found is definitely better, and overall f1mess is higher dtan for interactive generation,
but user cannor view documents and make modifications until the end of the search.
Because of the fast growth of the quantity and variety of Internet sires, finding the information needed
as quickly and thoroughly as possible becomes an important issue for research. There are two approaches to
Internet search: indexed search and design of intelligent agents. GA is a search method that can be used in the
design of intelligent agents. However, incorporating the knowledge about spatial and temporallocaliries, and
making these agents mobile, can improve the performance of the application and reduce rhe ncl:\vork traffic.

117.5 Soft Computing Based Hybrid Fuzzy Controllers

Traditional methods, which address robotics control issues, rely upon strong mathematical modeling and
analysis. The various approaches proposed rill dare are suitable for control ofindusrrial robots and automatic
guided vehicles, which operate in various environments and perform simple repetitive tasks chat require end
effectors positioning or motion along fixed paths. However, operations in unstructured environments require
robots ro perform more complex tasks for which analytical models for connol is very difficult to determine.
ln cases where models are available, it is questionable whether or not uncertainry and imprecision are

1
sufficiently accounted for. Under such conditions fuu.y logic control is an amactive alternative that can bt:
 530 Applicalions of Soft Computing

successfully implemented on real-time complex systems. Fuzzy controllers and their hybridization with orher
paradigms are robust in the presence of penurbarions, easy to design and implement, and efficient for systems
chat deal wich continuous variables. The control schemes described in this section are examples of approaches
that augment fuzzy logic with other soft computing teclmiques to achieve the level of intelligence required of
complex robotic systems. '
Three soft computing hybrid fuzzy paradigms for automated learning in robotic systems are briefly
described. The first scheme concentrates on a methodology that uses neural networks (NNs) to adapt a
fuzzy logic controller (FLC) in manipulator control tasks. The second paradigm develops a two-level hierar-
chical fuzzy control Structure for flexible manipulators. It incorporates GAS in a learning scheme to adapt to
various environmental conditions. The third paradigm employs GP to evolve rules for fuzzy behaviors to be
used in mobile robot control.

I 17.5.1 Neuro-Fuzzy System

Neural networks exhibit the ability to learn patterns of static or dynamical systems. In the following neuro-
fuzzy approach, the learning and pattern recognition of NN are exploited in two stages: first, to learn static
response curves of a given ~tern and second, to learn the real-time dynamical changes in a sysrem to serve as
a reference model. The neuro-fuzzy control architecture uses rwo neural networks to modify the parameters of
an adaptive FLC. The adaptive capability of che fuzzy conrroller is manifested in a rule generation mechanism
and automatic adjustment of scaling facrors or shapes of membership functions. The NN functions as a
,, classifier of the system's temporal responses.
A multilayer perceprron NN is used to classify the temporal response of che system into different pat-
terns. Depending on the type of pattern such as" response with overshom," "damped response," "oscillating
response," ere. che scaling factor of the input and output me_mbership funCtions is adjusted to make the system
respond in a desired manner. The rule generation mechanism also utilizes the temporal response of che system
I , to evaluate new fuzzy rules. The nomedundant rules are appended ro the existing rule base during che tuning
cycles. This controller architecmre is used in real-rime to comrol a dirccr drive motor.

17.5.2 Real-Time Adaptive Control of a Direct Drive Motor

In order ro perform real-time control, it is necessary for the controller to stand alone with the sole task of
calculating rhe output needed to control the object system. This means the task of communicating data for
storing as well as acquiring controller parameters (if the controller is adaptive) should be performed by external
processors. In chis way a real-rime control can be achieved with required sampling rate for high bandwidth
operation.
The FLC algorithm requites processing of several functionaliries such as fuzzificarion, inferencing and
defuzzificarion.
This means the computation time taken by che FLC itself does not leave any room for an adaptive algorichm
such as rule generation, calculating fie scale factor of fie membership ftmction, or NN algorithms. In order
ro implemem all iliese funaionalities, a multiprocessing architecture is needed. This can be ad1ieved by
combining a sufficiently fast processor specifically designed for real-time processing. such as a TMS320C30
digital signal processor (DSP) combined widt a PC Intel processor (Pentium or 486).

I 17.5.3 GA-Fuzzy Systems for Control of Flexible Robots

In chis section, GAs are applied to fuzzy control ofa single link flexible arm. GAs are guided probabilistic search
routines modeled after che mechanics of Darwinian ilieory of natural evolution. GAs have demonstrated the
 17.5 Soft Computing Based Hybrid Fuzzy Controllers 531

cocling ability to represent parameters of fuzz.y knowledge domains such as fuzzy rure .sclS and membership
functions in a genetic srrucrure, and hence are applio.b\e to optimization of fuzzy rule sets.
Several issues should be addressed when designing a GA for optimizing fuzzy controllers: fie design of
a transformation (interpretation) funaion, che mechod of -incorporating initial expert knowledge and the
choice of an appropriate fitness function. Each of the above issues significantly influences the success of GA
in finding improved solutions. These issues are briefly discussed below as chey apply to design of a GA-fuzzy
controller for a flexible link.
17.5.3.1 Application to Flexible Robot Control
The application ofGA-fuzz.y systems applied m flexible robot is discussed here. The GA-learning hierarchical
fuzzy control architecture is shown in Figure 17-18. Within the hierarchical control architecture, the higher-
level module serves as a fuzzy classifier by determining spatial features of che arm such as straight, oscillatory,
curved. This information is supplied to the lower level of hierarchy where it is processed among ocher sensory
information such as errors in position and velociry for che purpose of determining a desirable control input
(torque). In this, control system is simulated using only a priori expert knowledge. In the given structure, a
GA fine-tunes parameters of membership functions.
The following fitness function was used ro evaluate individuals within a population of potential solutions:

Fitness = J~ . 1
dt

where e represents the error in angular position and y represents overshoot. Consequendy, a fitter individual
is an individual with a lower overshoot and a lower overall error (shorrer rise time) in its rime response. Here,
I
I
results from previous simulations of the architecture are applied experimentally. The method of grtd-parmting
was used ro create the initial population. '

I Fuzzy
feature
extractor

I ~
lnilial '" ~~~
<0'
a ciil iilc
~

l.I c
~
knowledge
ReTrence input
Initial knowledge
0 ~ ~
;;;
~

~
~
s
~I~~
0

II ~r-

~
o'\cn
3
!"
~

Q 0 -

I_ ~
c
Control
effort
~
-
~
<
0.

..1"...
1!:
I.." A distributed parameter system
~

'D
Figure 1718 GA-based learn"mghierarchid control architwure .
""'. --::,-
. . '-
532 Applications: of Soft Computing

- (A) (B)
II (C)

Figure 17 a19 GA simulation: (A) Comparison of simulation responses; (B) plot of average firness;
(C) inilial experimental results.

Members of the initial population are made up of mutation of the knowledgeable grandparent (sb). A!; a
result, a higher fir initial population results in a faster rare of convergence as is exhibited in Figure 17~19(A).
Figure 17~19(A) shows the time response of the GA~optimized controller when compared ro previously
obtained results through the non~GA fuzzy controller.

17.5.4 G P-Fuzzy Hierarchical Behavior Control

The robot comrol benefits to be gained from soft computing-based hybrid FLCs is not limited to rigid
and Aexible manipulators. Similar benefits can be gained in applications ro conuol of mobile robot behavior.
Autonomous navigation behavior in mobile robots can be decomposed into a finite number of special~purpose
task-achieving behaviors. An effective arrangement of behaviors as a hierarchical nerwork of distributed fuzzy
rule bases was recently proposed for autonomous navigation in unsuuctured environments. The proposed
approach represents a hybrid control scheme incorporating fuzzy logic theory imo the framework ofbehavior~
based control.
A behavior hierarchy rhat encompasses some necessary capabilities for autonomous navigation in indoor
environments is shown in Figure 17~20. It implies rhar goal~directed navigation can be decomposed as a
behavioral function of goal~seeking and roure~following. These behaviors can be furrher decomposed into rhe
lower~level behaviors shown, with dependencies indicated by the adjoining lines. Each block in Figure 17~20
is a ser of fuzzy logic rules.
The circles in rhe figure represent dynamically adjustable weights in the unit interval, which specify the
degree to which low~level behaviors can influence control of the robot's actuators. Higher~level behaviors

i)

Primitive level

~- Doorway I
Figure 1720 Hierarchical decomposition of mobile robot behavior. ..
1
.
Uj
~
 ~

533 I!
~
17.5 Soft Computing Based Hybrid Fuzzy Controllers

consist of fuzzy decision rules, which specify these weights according to goal and sensory information. Each
low~level behavior consists of fuzzy control rules, which prescribe motor control inputs lhat serve to achieve ~
rhe behavior's designated task.
The functionality of this hierarchical fuzzy~behavior control approach depends on a combined effect of !:
rhe behavioral functionality of each low~level behavior and the competence of the higher~level behaviors char
coordinate them. Perhaps the most difficult aspect of applying the approach is the formulation of fuzzy rules
for the higher~level behaviors. This is nor entirely intuitive, and expert knowledge on concurrent coordination
of fuzzy behaviors is nor readily available. This issue is addressed using GP to compuracionallyevolve rules for
composite behaviors. The forthcoming section describes the genetic programming approach to fuzzy rule~base
learning.

-
117.5.5 GP-Fuzzy Approach
The GP paradigm computationally simulates the Darwinian evolution process by applying fimess~based
selection and genetic operators to a population of individuals. Each individual represents a compurer program
of a given programming language and is a candidate solution to a particular problem. The programs are
strucrured as hierarchical compositions of functions (in a set F) and terminals (function arguments in a set
T). The population of programs evolves over time in response to selective pressure induced by the relative
fitness's of the programs for solving the problem.
For the purpose of evolving fuzzy rule bases, the search space is contained in the set of all possible rule~
bases rhar can be composed recursively from F and T. The set F consists of components of the generic ifthen
rule and common fuzzy logic connectives, i.e., functions for amecedems, consequents, fuzzy intersection,
rule inference and fuzzy union. The set Tis made up of the input and output linguistic variables and the
corresponding membership. functions associated with the problem. A rule base chat could potentially evolve
from F and T can be expressed as a tree data structure with symbolic elements ofF occupying internal nodes
and symbolic elements ofT as leaf nodes of the tree. This rree structure of symbolic elements is the main
feature, which distinguishes GP from GAs, which use the numerical string representation.
All rule bases in rhe initial population are randomly created, bur descendant populations are created primar~
i\y by reproduction and crossover operations on rulc~base tree structures. For the reproduction operation several
rule~bases selected on the basis of superior fitness are copied from the current population into the next, i.e., the
new generation. The crossover operation startS with rwo parental rule bases and produces two offsprings chat are
added to the new generation. The operation begins by independently selecting one random node (using uni~
form probability distribution) from each parenras rhe respective crossover point. Thesubrreessubtendingfrom
crossover nodes are then swapped between the parents to produce rhe cwo offsprings. GP cycles through rhe
current population perform fi.rness evaluation and apply genetic operators to create a new population. The cycle
repeats on a generation~by~generarion basis until satisfaction of termination criteria (e.g. lack ofimprovement,
maximum generation reached, etc.). The GP result is rhe best~flr rule base rhat appeared in any generation.
In the GP approach to evolution of fuzzy rule bases, the same fuzzy linguistic terms and operators that
comprise the genes and chromosome persist in the phenotype. Thus, rhe use ofGP allows direct manipulation
of the actual linguistic rule representation of fuzzy rule~based systems. Furthermore, rhe dynamic variability
of the representation allows for rule bases of various sizes and different numbers of rules. This enhances
population diversity, which is important for the success of rhe GP system, and any evolutionary algorithm
for that matter. The dynamic variability also increases the potential for discovering rule bases of smaller sizes
than necessary for completeness, bur sufFICient for realizing desired behavior.
In this section, rhe softcompuringapproaches in handling complex models and unstructured environments
are smdied. Neuro~fuzzy, GA-fm.zy and GP-Fuzzy hybrid paradigms can he successfully implemented to solve

j,
~
 j,

534 Applications ofSoft Computing

the prominent robot control issues, namely, control of direct drive robot morors, control of flexible links and
intelligent navigation of mobile robots. This in near future allows us ro combine soft computing paradigms
for more intelligent and robust control.

117.6 Soft Computing Based Rocket Engine Control

Many of the rocket engine programs initiated by NASA's Marshall Space Flight Center (MSFC) in Hunrsville,
Alabama, have been successful as evident by success of the Space Shuttle Main Engine, ground resting of the
former X-33 engine and Fa.mac X-34 engine for the reusable launch vehicle program. As a result, a database of
test cases and lessons learned has been created from which improvements to engine control for future engine
programs can be made. Such cases include premature engine shutdowns, propellant leaks, and numerous cases
of anomalouS sensors and data. Such cases are not only costly to the American taxpayer, but also present a
risk in social acceptance of current and future space programs.
The Space Transportation Directorate at MSFC has continually expressed an interest in improving engine
conuol and many efforrs in various areas for conuol and anomaly detection and mitigation have been
undertaken. Some successful anempts have included nozzle plume analysis and engine vibration analysis.
Other efforrs, although successful in theory and simulation, have been partially successful in actual engine test
firings. It is the harsh engine environment of cryogenics, vibrations, real~rime control demands and different
engine configurations from test ro test that continually encourage researchers to determine alternative solutions
or improvements to approaches for engine control and anomaly d.etecrion and mitigacion.
Current control technologies depend on proven, sometimes archaic, hardware and logical programming
techniques which are costly to implement and maintain, and do not account for unforeseen conditions leading
to the kinds of problems referenced earlier. The principle goal is to provide another avenue to address MSFC's
Space Transportation Directorate's interest in improving overall engine control. An approach for investigating
and demonstrating how the application of soft computing technologies can further address presented control
issues in rocket engine control is presenred in this section as a case study. The testbed engine is shown in
Figure 17~21.

FiQUre 1721 Turbine technologies SR~30 turbojet engine.

 535
17.6 Soft Computing Based Rocket Engine Control

In this particular work, amomation and control of a smallscale turbojet engine is described and some
preliminary data obtained using a PID controller has been provided. Turbine technologies turbojet engine
is equipped with instrumentation for moniwring the operating conditions of the engine. Some preliminary
data obtained to demonstrate the safecy of the engine under expected hazardous operating conditions and
ro demonstrate the applicability of one~dimensional propulsion equations to calculate the thrust induced by
the engine are shown. Additional data obtained to determine the system transfer function to design a PID
controller are also shown. The PID control algorithm design has been outlined.
The instrumentation includes several thermocouples and pressure transducers and a load cell to measure
the rhrusr generated by the engine. A valve controls the fuel~flow rate. In the present work rwo separate control
approaches were used. First the difference berween the desired thrust from rhe engine and the thrust measured
using the load cell was used as the feedback signal ro control the fuel~flow me to the engine. In the second
approach the temperature and pressure sensor data were used to calculate the thrust produced by the engine
using the aero-thermodynamic equations applying to turbojet engine operations, and the difference between
the calculated thrust and the desired thrust was used as the feedback signal. In the present approach, turbojet
engine's operation will be auwmated and several control logics will be uimmed to show their capabilities. In this
hardware-in~rhe-loop control demonmation effort, firsra simple PID conuol algoriffim is demonstrated. The
testbed development and some preliminary results obmined are presented using rhe experimental apparatus.
Simply stated, the term "soft computing" here refers to computational mechanisms that can determine
suitable relationships (in a system data set) to assess and determine a quantitative opinion(s) based on furure
conditions. Wirhin MSFC, such computational mechanisms are viewed as a collection of algorithms that
can achieve optimal or near-optimal results in the presence of imprecise data, uncertainty, unknown physics
and probabiliscic outcomes. Such algorithms include automated reasoning, nondeterministic or probabilistic
methods. Examples of the laner include Bayesian nerworks, statistical resampling techniques, chaos theory and
parrs of learning theory. Other well-known soft computing technologies include fuzzy logic, neural ne[\vorks
and generic algorithms. The rerm soft compuring is used metaphorically ro commr with hard computing.
Hard computing systems are based on those traditional approaches used commonly in most evem~driven
systems. Such approaches are often viewed as crisp or binary. For example, in a propulsion system for engine
scan preparations, if liquid oxygen rank temperature A< xand liqt1id oxygen bottom rank pressure A> y, then
open liquid oxygen engine supply valve can be opened. For this example, soft computing would accommodate
a region of acceptable temperature and pressure valves as well as observe mher conditions such as liquid level
and so on. A mechanism (e.g. NNs) for determining when ro open the liquid oxygen engine supply valve
would be used.
The approach would differ in that it would be tolerant of any imprecision and uncertain[)' In essence,
i one could view soft computing as being similar to the way the human brain works. Humans rend to use
t heuristic (objective) and subjective knowledge before making decisions based on current states of events. The
key features in soft computing stems from addressing any inherent imprecision, uncerrainry, partial truths
and overall system knowledge. The central goal in soft computing is to ana in more robust response. For this
efforr, rhe primary technologies to be used are Bayesian belief networks and fi.12.zy logic. For clte engine starr-up
sequence, rhe Bayesian belief nerworks will be used to ascertain the stare of rhe engine prior w proceeding
to main stage control. For main stage engine control approach, fuzzy logic will be employed and it is largely
dependent on the complexity of the engine control requirements and functions.

lt7.6.1 Bayesian Belief Networks

For the engine starr phase, rhe primary soft computing technology to be utilized is Bayesian belief nerworks
(BBNs). The sole inrenrofthe BBN is to qualify each of rhestares during engine start-up prior to reaching main

~~~
 536
Applicalions of Soft Computing
l
r

;p<y(~(~)t
/:;;y(~ ,!,(~

K ~
Figure 1722 Bayesian belief network.

srage. This will funher assure cerraincy in the health of the engine and proceeding imo main srage in addition
w providing added assurance inro preventing any premarure engine shut.downs. BBNs have been proven
to be good predictive and diagnostic mechanisms for reasoning about the srare of evencs in environments
where uncerraimy is universal. Suppressing rhe derails, the genealogy of this SCT is srrongly rooted in classic
srariscical Bayesian inference theory where a subjectivist viewpoint is taken. Figure 1722 shows a Bayesian
belief nerwork.
In short, Bayesian inference uses a different imerprerarion of probabiliry where one's degree of belief in
some event is pan of rhe reasoning. BBNs are compmarional archicecmres chat permit declarative (prior
conditional probabilisric values) and subjective opinions (posterior probabilistic values) about world (factual)
knowledge to be parr of rhe reasoning and assessment through a visual network representation and a unique
syntactic message-passing feature.

1. Beliefupdllting: \XIhen node X is activated to update irs parameters for belief updating, it first inspects all
messages transmitted ro it by its parents (;rr) and its children nodes (A). Then using all input, it updates
ics belief.
2. Bottom-up propagation: Using messages rransmined by Y and Z, compute message to transmit co parent
node U.
3. Top-down propagation: Node X then computes new messages to be sent ro irs children nodes Y and Z.

117.6.2 Fuzzy Logic Control

For main stage control, the plan is to use fuzzy logic. The use of fuzzy logic is suitable in that it accommodates
the uncertainties associated wich control during power. Fuzzy logic is a branch of mathematics thar deals with
approximate reasoning.
Zadeh of the University of California at Berkeley combines the copies of mulrivalued logic, probability
cheory and artificial inrelligen_ce for simulation of human cltought by using computer software as a medium.
The technology of fuzzy logic enables a computer w make decision based on vagueness or imprecision intrinsic

it
~
 17.6 Soft Computing Based Rocket Engine Control 537

in most physical systems. Fuzzy logic comprises sees and subsets where a set representS an input linguistic
variable and its subsets represent the linguistic values.
A cook-book process is followed using fuzzification and defuz.zificarion to determine a suitable response
m conditions on any given system co be controlled. The centroid method (or cencer of gravity method) will
be used for fuzzification. The intent in employing fuzzy logk wirh the SR-30 engine is to utilize all engine
test data and conventional PID data to design and develop the fuzzy logic based controller.
Once the soft computing techniques ofBBNs and fuzzy logic have been developed and tested and verified
(off-line), both will be integrated imo the SR-30 engine tested control environment for final integration and
verification testing.
j]

i
17.6.3 Software Engineering in Marshall's Flight Software Group

In every software development organization a set of processes and standards for their base product line are
typically adhered roo. Such processes and standards generally adhere to a type of software development life ~
cycle. For the Flight Software Group the popular Waterfall Model is used. Furthermore, the Flight Software
Group's process is ISO 9001 certified. And more importantly, has recently been certified as a CMM Level3
l{i
~
organization, a first for any NASA organization. CMM is the Capability Maturity Model for software that was
developed by Carnegie Melon's Software Engineering Institute and has become an internationally recognized '"'
standard for evaluating software development processes where a level 5 is the highest certification a software ~~'
development organization can achieve. The principle function of the Flight Software Group is to develop flight ~
;J.,
critical software for embedded systems, hence requiring all software development processes to be stringent

)
with software quality assurance functions underlying all activities of software development.
The Flight Software Group traditionally views software engineering as rhe establishmenr and uses sound
~
;
engineering processes to develop reliable software, based on human processes and thinking, that works on real
i
machines. Furthermore, software engineering is also viewed as the design and implemenrarion of a set of user
requirements into sofrware using sound engineering processes. The emphasis here is that the Flight Software ''
Group uses sound sofrware development processes based on empirically proven and sound practices.
~1
17.6.4 Experimental Apparatus and Facility Turbine Technologies SR-30 Engine

Soft compming technology hardware-in the-loop experiments were conducted using Turbine-Technologies
model SR-30 turbojet engine shown in Figure 17-2!. The demonstration engine consists of rhe turbojet engine
manufactured by Turbine Technologies Ltd. in irs custom enclosure. The enclosure includes a control panel
for engine operation and monitoring and a PC-based data acquisition unit for measuring the engine operating
conditions.
The SR-30 engine has a single-stage radial flow compressor with a maximum pressure ratio of PR = 3.4,
single-srage axial-flow turbine, and reverse-flow annular combustion chamber and it operates obeying the
Brayton thermodynamic cycle in rhe same fashion as rhe large turbojet engines. The engine as produced by
the Turbine Technologies includes many pressure and temperature sensors, a load-cell for rhrusr measure-
menrs, a cusmm motor winding for reading the engine rpm and a fuel flow-rare measurement system to
monitor/measure the operating parameters of the engine. The engine generates 20 lbs of thrust at 90,000
rpm while ingesting m = I. lib s- 1 of air. The engine has a length of 10.75 in. and rhe exit exhaust diameter
ofDexit = 2.25 in.
The engine available is instrumented with pressure transducers in the compressor inlet and exit, in the
combustor, in the mrbine exir and in the thrust nozzle exit, and Krype thermocouples in the compressor inlet
and exit, in the turbine inlet and exit and in the rhrust nozzle exit. The engine availabl~ was also equipped

~
 538 Applications 'of Soft Computing

with a National Instruments (NI) PCI4351, AID board with 24 bit resolution for 16 analog inputs with a 60
samples s- 1 capabilicy,.and aNI Virrual Bench Logger data acquisition program for monitoring the measured
parameters on a PC.
Starting the engine requires an external source ofhighpressure air at minimum 100 psi to spin up the
engine to appmximatdy 10,000 rpm. Subsequent fuel injection and ignition srnru the engine. The fuel.flow
rate is comrolled by the person operating the engine with the use of a lever, which basically comrols a valve
conscricting the fuel flow to the engine. Engine idles at approximately 50,000 rpm and the thrust generated
increases with the increased rpm. To obtain higher thrust values the engine operator steadily increases the
fuel-flow rate from the idle condirions. In order to stop the engine it is brought to the idle conditions and
run until the exhaust temperature drops under 100 C, to minimize engine damage.

I 17.6.5 SystemModifications

Turbine Technologies data acquisition system as purchased and used for classroom demonstrations is not
sufficiently fast enough for use with the hardware-in-the-loop control algorithms. Since one of the main
scopes of the present work is to implement and demonstrate different control algorithms in controlling a
turbojet engine thrust, a new data acquisition system and sofrware has been implemented into the existing
system tO increase the data acquisition speed and to increase the control capabiliry. Additionally, the available
system was d'esigned and used to collect and present data and it did not have provisions to send signals via
computer for closed-loop control applications. Changes implemented include the replacement of the data
,, acquisition board, connection panels for the sensors, addition of a low flow-me fuel-flow rate measurement
unit, a fast acting linear servo-controller.

17.6.6 FueiFiow Rate Measurement System

'.: The SR-30 engine as produced by the Turbine Technologies uses a pressure transducer together with a
calibration curve to determine the fuel-flow rare to the engine. The pressure values read on rhe fuel line are
plotted against the fuel flow spent and also against the rpm of the engine to generate pressure vs. the fuel rate
and the pressure vs. the rpm calibration curves for long term operations. However, for the present purposes,
since the fuel-flow rate is mainly the only control input to control the desired thrust of the engine a more
accurate and faster fuel-flow rare measurement device has robe implemented.

I 17.6. 7 Exit Conditions Monitoring

Turbojet propulsion equations used in calculation of the engine thrust requires the measurement of the
exit conditions, namely the exhaust toral pressure and the total temperamre. Although the existing system
available from Turbine Technologies incorporated a pressure transducer and a thermocouple, for this purpose,
the response time for the equipment was rather slow. In order to increase the tim'e resolution of the data
obtained at the exit conditions a new pressure transducer with a 0.2 ms response time and a 0.1 s response
time has been incorporated.
As a result of this effon, new insight has been gained into the behavior and application of soft computing
technologies in a rocket engine control environment. The methodology created here will provide a new
approach to the area of employing soft computing technologies in rapid response engine control systems
for future vision vehicles. )t will yield better insight into incorporating soft computing technologies with
proven and practical sofrware engineering methods. It is e..'<pecred that rhis efforr will demonsuate that by
 539
17.9 Exercise Problems

employing soft computing technologies, issues in quality and reliability of the overall scheme of engine
conuoller development can be further improved and thus safery be further insured.
Furthermore, the use of these soft computing technologies is expected ro supplement effons in improving
software management, software development rime, software maintenance, processor execution, fault tolerance
and mitigation and nonlinear control in power level transitions, all of which contribute to a better engine
control system. It is projected that the final product wiU yield a foundation for a path to further development
of an alternative low cost engine controller that would be capable of performing in unique vision spacecraft
vehicles requiring low cost and advanced avionics architectures for autonomous operations from engine
pre-start to engine shutdown.

I 17.7 Summary
In this chapter we have dealt with rhe applications of soft computing techniques. The application areas of
these soft computing techniques are growing day by day. Neural netvVorks and fuzzy logic are effectively used
in various control applications. Genetic algorithm plays a major role in providing solutions for optimizing a
problem. The combinations of all these techniques give an accurate solution to complex systems. There are
various researches going around the world in the field of soft computing.

111.8 Review Questions

1. State the various applications of neural networks. 7. Explain the application of fuzzy logic systems m
image processing applications.
2. Mention rhe application areas of fuzzy logic.
8. Describe in derail the application of generic
3. In what areas does generic algorithm gives a best
algorithm to Civil Engineering area.
optimized solution?
9. With suitable block diagram, explain the prin-
4. Lisr few applications of hybrid fuzzy GA sysrems ciple involved in a liquid level controller using
and neurofuzzy systems.
neurofuzzy technique.
5. Soft compming techniques gives best solution to 10. With a case study example, describe in derail the
complex problems. Justify.
application of soft computing.
6. Wirh suitable case study, explain how neural
network best performs its control action.

117.9 Exercise Problems

1. Write a program for implementing generic 4. Implement a waterflow management system and
algorithm based Internet search technique. water neatment system using soft computing
approaches.
2. Write a program for processing an image of size
16 x 16 using neural networks and fuzzy logic. 5. Construct a neural network training controller
for controlling rhe motion of a sateHire.
3. Build a 3D game using the design issue~ in
generic algorithm.
}~
1.:~

t;;
 ::.~.:'
..
540 Applications of Soft Computing

6. Write a program for analyzing the landing of an 9. Implement with any example, the concept
aircraft: using fuzzy logic methodology. involved in parallel genetic algorithm.
7. lmplernem robot motion control using neuro- 10. Write a program for conuolling the motion of
fu:z.zy controller. an inverted pendulum using neural networks and
8. Wrire a program using genetic algorithm to solve fuzzy logic.
a traveling saleman problem.

1..:.":;.

-~
 I
Soft Computing Techniques
Using Cand C++ 18 !
I!
Learning Dbjeclives !
Gives rhe source codes for soft compuring The Cartesian producr.s of two given fuzzy ser.s, 'il'
techniques inC and C++. max min composition for fuzzy relations are ''
Neural network implementation is performed also implememed in C and C++ w enhance
for perceptron network, Madaline net, BPN, the reading of fuzzy logic concept.
CPN, ART and Kohonen self-organizing fea- Few problems of maximizing and minimizing
ture maps. a function, traveling salesman problem, pris
In fuzzy logic, the implementation is carried onner's dilemma, quadratic equation solving
our for primitive operations of classical sets
are implememed in the universal language ro ~ol

and fu:u.y sets. depict the genetic algorithm operation. '}

l

lta.t Introduction
)l:
This chapter gives the source codes for implementation of Soft Computing Techniques using the languages
C and C+-t. Cis a generalpurpose strucmred programming language that is powerful, efficiem and compact.
It combines the features of a high-level language with the elements of the assembler and rhus is close to ;~
man and machine. Programs written in C are very efficiem and fast. C++ on the other hand is an object
oriented language that a C programmer can appreciate, especially who is an early age assembly language
programmer. C++ orients toward execution performance and then roward fb:ibiliry. The name C++ signifies
the evolutionary nature of the changes from C. Thus Soft Computing being an approach based on evolutionary
strategies and evolutionary programming can be implemented using rhe structured programming and objecr
programming languages. This chapter discusses few problems solved using CIC++.

lt8.2 Neural Network Implementation

The various neural networks discussed through Chapters 2-6 are implemented using C and C++ languages
in this section. The source code for each network for a specific application is given below.

I 18.2.1 Perceptron Network

The program for perceprron nerwork is as follows:

/'"'PERCEPTRON* I
#include<stdio.h>
#include<conio.h>
main()
{

~
 542 Soft Computing Techniques Using C and C++

signed int xl4J [2], tar[4];

float w[2],wc[2],out=O;
int i,j,k=O,h=O;
fl~at s=O,b=O,bc=O,alpha=O;
float theta;
clrscr{);
printf[Enter the value of theta & alpha");
scan ( ~%f%f", &theta, &alpha);
for(i=O;i<=3;i++)
(
printf{"Enter the value of %d Inputrow & Target",i};
for(j=O;j<=l;j++l
(
scan ( "%d" ,&x[i] [j]);
}
scanf("%d~,&tar[i]);
w[i]=O;
wc[i]=O;

printf("\Net\t Target\tWeight changes\tNew weights\t Bias changes\tBias\n");

printf("-----------------------------------------------------\n");
mew:
\ print ("ITERATION %d\n", h);
print( ---------\n"l;
for(i=O;i<=3;i++)

for(j=O;j<=l;j++)
(
s~=(float)x[i] [j]*w[j);

s+=b;
printf("%.2f\t",s);
if (s>thetal
out=l;
else if(s<-theta)
out=-1;
else

out=O;
}
printf ( "%d\t", tar[i]);
s=O;
if (out==tar [ i] )
I
for(j=O;j<=l;j++)
(
we[j]=O;
be=O;
print ( "%. 2f\t" ,we (j]);
)
 18.2 Neural Network.lmplementalion 543

for(j=O;j<=l;j++}
print("%. _2f\t", w [j]);
k+=l;
b+=be;
print ( "%. 2f\t\t" ,be);
print ( "%. 2f\t", b);

else
(
for ( j=O; j<=l; j++)
I
we[j]=x[i] [j]*tar[i]*alpha;
w[j]+=we[j);
printf(b%.2f\t" ,we[j]);
we[j]=O;
}
for(j=O;j<=l;j++l
print ( "% .2f\t" ,w[j ));
be=tar[i]*alpha;
b+=be;
print ( "%. 2f\t\t", be);
printf ( "%. 2f\t", b);

print ( "\n");
}
i f (k===4)
I
printf ( "\nFinal weights\n" l;
for(j=O;j<=l;j++l
(
print ( "w[%d] =%.2f\t", j ,w[j]);

print (~Bias b=%. 2f", b);

}
else

k=O;
h=h+l;
geteh();
goto mew;
}
geteh();
}

118.2.2 Adaline Network

The program for ada.line network is as follows:
I*ADALINE*/
'. #include<stdio.h>

I #include<conio.h>

a
544 Soft Computing Techniques Using C and C+t

main()
[
signed int x[4] [4],tar[4];
float wc[4),w[4],e=O,er=O,yin=O,alp=0.5,b=O,bc=O,t=0;
int i,j,k,q=l;
clrscr();
for(i=O;i<=3;i++l
[
printf ("\nEuter the %d row and target\t", i) ;
for(j=O;j<=3;j++l
[
scanf ( ''%d", &x[i] [j));

scanf ( "%d", &tar[i]);

printf ( "%d", tar[i));
w[i]=O.O;
wc[i)=O.O;

mew:
er=O;e=O;
yin=O;
print ("\n ITERATION%d" ,q);
printf{"\n------------------"1;
for(i=O;i<=3;i++)
[
t=tar[i);
for(j=O;j<=3;j++)
[
yin=oyin+x[i] [j]*w[j];
)
b=b+bc;
yin=yin+b;
bc=O.O;
print ( "\nNet=%f\t", yin) ;
e=(float)tar[i]-yin;
yin=O.O;
printf ( "Erroro::%\t", e);
print ( "Target=%d\t\n", tar[iJ);
er=er+e*e;
for(k=O;k<=3;k++)
[
wc[k]=x[i] [k]*e"alp;
w[k) +=we [k);
wc[k)=O.O;
)
printf ("Weights \t");
for(k=O;k<=3;k+i)
[ \-
printf ( "%f\t", w[k]);

1
)
 ~

~
~
18.2 Neural Network Implementation 545

bc=e*alp;

I
printf..(-"'b=%, 2f\t", b);
get.ch();
printf("\n Error Square=%f",er);
if(er<=l.OOOJ
[
printf("\n"); 4
for(k=O;k<=l;k++l j
print ( "%f\t", w[k] l; 1
~
getch(); J

else '
e=O;
er=O;
yin-=0;
q=q+l;
goto mew;

getch(); ~:,
)
)

I'
I 18.2.3 Madaline Network for XOR Function
,II
The program is as follows:
//XOR function using madaline
~include<stdio.h>
'
#include<conio.h>
void main ()
[
signed int x[4)[2),tar[4);
float w[2] [2] ,a,o[2];
float we (2] [2], zin [2], z1.=0, z2=0 ,yin=O, b(2], er=O, b3=0, vl=O, v2=0. 5;
int i,j,c=O,in,d;
float be [2];
float alp=O. 5;
clrscr();
for(i=O;i<=3;i++)
[
printf("Enter the %d row & target:");
for(j=O;j<=l:j++)
scanf("%d",&x[i) [j));
scanf("%d",&tar[i]);

getch ();
print ("Enter Weights: J;
for(i=O;i<=l;i++l

1
[
546 Soft Computing Techniques Using C and C++

for (j=O; j<=l; j ++)

{
scanf{"%f",&a);
w[i] [j]=a;
wc[i] [j]=O;

printf("bias~);
scanf("%f",&b[i]);
zin(i] =0;

mew:
printf("Iteration\n");
print ( "--------------\n");
for(i=O;i<=3;i++)
{
for(in=O;in<=l;in++l
{
for (j=O; j<=l; j ++)
{
zin[in]+=x[i] (j]*w[j] [c);

zin[in]+=b[in];
printf("zin%d= %.3\t",in,zin[in]);
c+=l;

c=O;
d=l;
if(zin[c)>=O & zin[d]>=O)
zl=z2=1;
else if(zin[c)>=O & zin[d]<=O)
{
zl=l;
z2=-l;

else if{zin[c)<=O & zin[d]>=O)

{
zl=-1;
z2=1;

else

zl=z2=-1;

yin=zl*vl+z2*v2+b3;
print ("NET %. 3\t", yin);
for (in=O; in<=l; in+-+)
{
o[in]=tar[i]-zin[in);
er+=o[in]*o[in];
zin[inl=O;
 ,..
I~

18.2 Neural Network Implementation 547

f~
if{yin==tar[i]J
I
{
for{in=O;in<=l;in++J
(
1'j!
l
for{j=O;j<=l;j++) ;
{
we[inl [j]=O;
w[in] [j]+=we[in] [j); 1'
be [in] =0;
}
yin=O;

else

for(in=O;in<=l;in++)
{
for(j=O;j<=l;j++J
{ ~~
we[in] [j]=alp*o[j]*x[i) [in];
printf( "wc%d%d=%. 3f\t ,in, j ,we [in] {j]};
w[in) {j]+=wc(in) [j];
printf ( "w=%. 3f\t", w[in) (j]);
we[in] [j]=O;
)I
for(in=O;in<=l;in++)

bc[in)=alp*o[inl;
b[in] +=bc[in);
printf ( "\nb%d=%. 3f", in, b[in]);

for(in=O;i<=l;in++)

bc[in)=O;

yin=O;

printf ( "\n);

if(er<=l}
{
for(i=O;i<=l;i++)
{
for(j=O;j<=l;j++)
printf ( "%. 3f", w[i] (j]);

else

~
548
Soft Computing Techniques Using C and Ct+

yin=O;
for(in=O;in<=l;in++)
{
bc[in]=O;
zin[in) =0;

er=O;
getch ();
goto mew;
)
getch();

18.2.4 Back Propagation Network for XOR Function using Bipolar Inputs and
Binary Targets

In rhis case the assumption is made for the necessary parameters. The initial weighrs and bias are assumed to
be of small random values. The program is as follows:
/*BACK PROPAGATION NETWORK*/
#include<stdio.h>
#include<conio.h>
#include<math.h>
#include<stdlib h>
void main ()
{
float v(2] (4] ,w[4) [1] ,vc[2] (4] ,wc[4) [1] ,de,de1[4] ,bl,bia,bc(4] ,e=O;
float x[4] [2], t [4], zin[4], delin[4], yin=O, y, dy, dz [4) ,b[4], z [4],
es,alp=O.G2;
int i,j,k=O,itr=O;
v[O) 101=0.1970;
viOl [11=0.3191;
v[O] [2]=-0.1448;
v[O) [3]=0.3594;
V[l] (0]=0.3099;
v[11 111=0.1904;
v[1) [2)=-0.0347;
v[11 [31=-0.4861;
w[OI [0)=0.4919;
w[1) [01=-0.2913;
w[2) (0]=-0.3979;
w[31 [0)=0.3581;
b[0)=-0.3378;
b[11=0.2771;
bl.2)=0.2859;
b[3)=-0.3329;
bl=-0.141;
x[O) [0)=-1; I

-~
 a
18.2 Neural Network Implementation

x[O) 111=-1;
x[11 [01=-1;
549
I
~~
li
x[1) 111=1;
x[21 101=1;
x[21 [11 =-1;
x[31 [01=1;
x[31 [1)=1;
t[O)=O;
'j
t 11 I =1, {
t [21=1;
t 13 I =0;
clrscr();
for(itr=O;itr<=387;itr++)
{
e=O;
es=O;
for(i=O;i<=3;i++)

do
{
for(j=O;j<=l;j++) :~
{
zin [k]~=x(i] [j] *v[j J [k);

zin(k] +=b [k);

k+=l; )I
]while (k<=4);
for(j=O;j<=3;j++)
{
z[j]=(l-exp(-zin[j] J) I (l+exp(-zin(j]l);
dz[j]=( {l+z[j) }* (1-z(j))) *0. 5;

for{j=O;j<=3;j++l
{
yin+=z[j)*w[j) [0];

yin+=bl;
y=(l-exp{-yin)}/(l+exp(-yin));
dy= ( (l+y},. (1-y}) *0. 5;
de= (t [iJ -y) *dy;
e=t[i)-y;
es+=O. 5"' (ee);
for(j=O;j<=J;j++)
I
wc(j) [O]=alp*de*z(j);
delin[j]=de*w[j] [0];
del[j)=delin(j]*dz(j);

bia=alp*de;
for(k=O;k<=l;k++l
 550 Soft Computing Techniques Using C and Ct+

for(j=O;j~=3;j++)
[
vc [k] [j] =alp*del [j] *x[i] [k];
v[k] [j]+=vc[k] [j];
}

for(j=O;j<=3;j++l
[
bc[j]=alp*del[j];
w[j] (O]+=wc[j] (O];
b[j)+=bc[j);

bl+=bia;
for(j=O;j<=3;j++)
[
zin[j]=O;
z [j] =0;
dz(j)=O;
~ delin[j] =0;
~
del[j]=O;
bc[j]=O;

\ k=O;yin=O;y=O;
dy=O;bia=O;de=O;
}
/' printf(~\nEpoch %d:\n",itr);
for(k=O;k<=l;k++)
[
for(j=O;j<=3;j++l
[
print ( "%f\t", v[k] [j]);

printf("\n");

printf("\n");
for(k=O;k<=3;k++)
[
print ( "%f\t" ,w[k] [0]);

print ( "\n%", bl);

print ( "\t");
for(k=O;k<=3;k++)
[
print ( "%f\t" . blkl);

getch();
)
getch();
 18.2 Neural Network Implementation 551

18.2.5 Kohonen SelfOrganizing Feature Map

The user can enter the inputs and inirialize the weights of his wish.

//A KSOM to cluster four input vectors

#include<stdio.h>
#include<conio.h>
void main()
{
signed int x[4] [2];
float w[4][2] ,dl,d2,o,m=O;
int i.j,k,J;
float alp=O. 6;
clrscr () ;
printf("Enter the input:");
for(i=O;i<=3;i++)
[
for{j=O;j<=3;j++)
{
scan ( "%d", &x [ i l [ j l ) ;

printf("Enter the Weight matrix:");

for(i=O;i<=3;i++)
[
for(j=O;j<=l;j++l
[
scanf("%fp,&o);
w[i) [j]=O;

mew:
for(i=O;i<=3;i++J
[
for{k=O;k<=l;k++l
[
for{j=O;j<=3;j++)
{
i f (k==O)
dl+= (w(j) [k] -x[i] [j]) * (~ [j] {k)-x[i] [j]);
else
d2+= (w[j] [k)-x[i) (j]) * (w[j] [k]-x[i] [j]);

if(dl>d2)
J=l;
else
J=O;
dl=d2=0;
I for ( j =0; j <=3; j ++)
i
I
I
~
552 Soft Computing Techniques Using C and C++

w[j] (J]+=alp(x[i] [j)-w[j) [J));

getch();
)
alp=alp/1. 014;
if{m>=lOOJ
[
for(i=O;i<=3;i++l
[
for(j=O;j<=l;j++)
[
printf ( "\n%f\t" ,w[i] {j]);

getch();

e]se

m=m+l;
for(i=O;i<=J;i++)
[
for{j=O;j<=l;j++J
[
print ( "\n%f\t' ,w[i) [j) l;
I
printf ( "\n");
)
getch();
gate mew;

getch();

18.2.6 ART 1 Network with Nine Input Units and Two Cluster Units

The program is as follows:

; AN ARTl NET WITH 9 INPUT UNITS AND 2 CLUSTER UNIT */
#include<stdio h>
#include<conio h>
main()
[
float ro;
float b[9] [3], t [2) [ 9 J, s [9], x[9] , sin=O ,y [2) ,xin=O;
int i,j,k=O,J,c=O;
y[Ol=O;
y[1]=0;
biOI [0]=0.33;b[.1] [O]=O.O;b[2] [0]=0.33;
b[3] [0]=0.0;b[4)[0]=0.33;b[5][0]=0.0;
 :r,
.. !

'1,I
18.2 Neural Network Implementation 553 !
:
b[6] [0]=0.33;b[7] [O]=O.O;b[B] [0]=0.33; li
i:
b[O] [1]=0.1;b[1] [1]=0.1;b[2] [1]=0.1; :f:
b[3] [1]=0.1;b[4] [1]=0.1;b[5] [1]=0.1;
b[6] [1]=0.1;b[7] [1]=0.1;b[BJ [1]=0.1;
t[O] [O]=l.O;t[O] [1]=0;t[O] [2]=l.O;t(O] [3]=0;
t[O] [4]=1.0;t[O] [5]=0;t[O] [6]=l.O;t[O] [7]=0;
t[O] [8]=1.0;

t[1] [0]=1;t[1] [1]=1;t[1] [2]=1;t[1] [3]=1;

t[1] [4]=1;t[1] [5]=1;t[1] [6]=1;t[1] [7)=1;
t[1) 18]=1;

clrscr();
mew:
printf("Enter the value of ro\n");
scanf("%f",&ro);
printf("Enter the input value\n");
for(i=O;i<=8;i++)
{
~
scan ( "%" , &s [i] ) ;
x[i]=s[i];
'
sin=s[O)+s[l]+s[2]+s[3]+s[4]+s[5]+s[6]+si7]+s[8];
for(i=O;i<=l;i++)
[
II
do
[
y[i]+=S[k)*b[k) [i);
k+=l;
}while(k<=B);
k=O;

for(i=O;i<=l;i++)
print ( "\tyin=%", y [i] l:
if(y[O]>=y[l])
J=O;
else
J=l;
print ( "J=%d" ,J);
me:
for(i=O;i<=B;i++)
[
x[i] =s [i] *t[J] [i];

xin=x[O]+x[l]+x[2]+x[3]+x[4]+x[5)+x[6]+x{7]+x[8];
if((xin/sinl>= ro)
[
for(i=O;i<=B;i++)
 -

554 Soft Comp,t;og Tooho;q,es u,;,g C aod C++

b[i] [J]=(2*x[i])/{l+xin);
t.[J) [i)=x[i);

else

y[J]=-1;
xin=O;
goto me;

print ( "\nBottom up weights\n");

for(i=O;i<=S;i++)
{
for(j=O;j<=l;j++)
{
printf ( "%f\t", b[i) (j)) ,

print ( "\n");

:-~. printf("\nTop down Weights\n");

..... for(i=O;i<=l;i++)
" {

\ 'i for(j=O;j<=S;j++l

print ( "%f\tft, t[i) [j]);

,,.I
/ print ( "\n");

. .' getch();
~ I y[O]=O;
y[l]=O;
y[2]=0;
sin=xin=O;
c+o::l;
k=O;
if(c<=2l
goto mew;
getch();

18.2. 7 ART 1 Network to Cluster Four Vectors

The program is as follows:

/* ARTl NETWORK TO CLUSTER FOUR VECTORS*/
~include<stdio.h>
#include<conio.h>
main()
{
float n=4.0,m=3.0,o=0.4,1=2.0;
'!l\'111"
-~-

I 18.2. Neural Ne!Work Implementation 555

float b[4] (3] ,t[3] [4] ,s[4] ,x{4] ,sin=O,y[3] ,xin=O;

int i,j,k=O,J,c=O;
y[0]=0,y[l]=O,y[2]=0;
clrscr{);
for(i=O;i<=3;i++l
{
for(j=O;j<=2;j++)
{
b[i] {j]=0.2;

for(i=O;i<=2;i++)
{
for(j=O;j<=3;j++)
{
t[i] [j]=l.O;

mew:
printf{"Enter the input value:\n");
for(i=O;i<=3;i++)
{
scanf("%f" ,&s(i]);
x[i]=s [i];
sin+=s[i];

for(i=O;i<=2;i++)
{
print ( "\nY");
do

y[i] +=s [k] *b[k] [i);

k+=l;
)while (k<=3);
if (y[O]>=y[l])
{
if(y[O]>=y[2))
J=O;
else
J=2;

else

if(y[ll>=y[2])
J=l;
else
J=2;

for{i=O;i<=3;i++l
i {

~
.
.
.
-::..
 ::...
:l_o;-
,<C,'

556 Soft Computing Techniques Using C and C+t

x[i]=s[i)*t[J] [i];
xin+=x [ i.] ;

if(xin/sin>=0.4)
{
for(i=O;i<=3;i++l

b[i] [J] = {2*x[i]) I ( l+xin);

t[J] [i]=x[i);

else

y[J]=-1;
}
printf("\n");
for(i=O;i<=3;i++}
{
for (j=O; j<=2; j+"+)
{
printf("%f\t",b[i) [j]);

print ( "\n");

for(i=O;i<=2;i++)
{
for(j=O;j<=3;j++)
{
print ( "%f\t", t[i] [j]);

print ( "\n");

getch();
y{O]=y{1]=y[2]=0;
sin=xin=O;
c+=l;
k=O;
i f (C<=3)
goto mew; I

getch();
} I
I 18.2.8 Full Counterpropagation Network

The program is as follows:

!* FULL COUNTER*/
~include<stdio.h>
#include<conio.h>

1
 ..
' .I
18.2 Neural Network Implementation 557 :;I
,,;
void main ()

float alp=O. 6, x=O .1, n [10], v[l] [10], d[lO] ,p,w[l) [10] ,y, bet=O. 6;
float u[lO] (1], t[lO] [1], a=O. 6,b=O. 6;
int i,j,J,k=O,m;
clrscr();
v[O] [0]=0.1;v{O] [1]=0.15;v[O] [2]=0.2;
v[O] [3]=0.3;v[O] [4]=0.5;v[O] [5]=1.5;
v{Oj [6]=3.0;v[O] [7]=5.0;v[O] [8]=7.0;
v[O] [9]=9.0;
w[O] [0)=9.0;w[O] [1]=7 .O;w[O) [2)=5.0;
,J
w[O} [3]=3.0;w[O]
w[O] (6]=0.2;w[Ol
[4]=1.5;w[O] [5]=0.5;
[7]=0.2;w[O] [8]=0.15;
I
w[O] [9]=0.1;
u[O] [0]=0.1;u[1] [0]=0.15;u[2] [0]=0.2; I
u[3] [0]=0.3;u[4]
u[6] [0]=3.0;u[7]
[0]=0.5;u[5] [0]=1.5;
[0]=5.0;u[8] [0]=7.0; I'
u[9] [0]=9.0; l
t[O] [0]=9.0;t[1} [0]=7.0;t[2] 10]=5.0;
t[3] [0]=3.0;t{4] [0}=1.5;t[5} [0]=0.5;
t[6] [0]=0.3;t[7] {0]=0.2;t[8] [0]=0.15; ~i
~~
t[9] [0]=0.1;
do ~~
I
{
y=l/x;
printf("\n");
1:,.
<
for(j=O;j<=9;j++) '
{
n[j]= (x-v[O] [j]) * (x-v[01 [j 1) + (y-w[O] [j 1) * (y-w[O] [j1);
d[j]=n[j]; 7
}
for(m=O;m<=9;m++)

for(j=m;j<=9;++j)
{
if{d[k]>d[j]}
{
p=d[j];
d{j]=d[k];
d[k]=p;

k+=l;

for(j=O;j<=9;j++l
{
if(d[O]==n[j1)
{
J=j;

1
 ~'
558
Soft Computing Techniques Using C and C++

v[O] [J]+=alp* (X.-v[O] [J]);

w[O) (J] +=bet* (y-w(O) [J]);
printf(~\ninput X=%f",xJ;
printf("\nUpdated weights: v");
for(j=O;j<=9;j++)
{
printf ( "%f\t" ,v[O) [j]);
n[j] =0;
d{j]=O;

print ( "\n Updated weights: w");

for (j=O , j<=9; j ++)
{
print ( "%f\t" ,w[O) [j]);

x=x+0.5;
alp=alp/1.014;
bet=bet/1.014;
J=O;
k=O;
getch();
}
while(x<=l0.50);
x=O.l;
do
{
for(j=O;j<=9;j++)
{
n[j)= (x-v[O] [j]) * (x-v[O] [j]) + (y-w[O] [j)) * (y-w[O] tj]);
d[j)=n[j];

for(m=O;m<=9;m++)
{
for(j=m;j<=9;j++)
{
if{d[k]>d(j])
{
p=d{j);
d{j}=d{k);
d{k]=p;

k+=l;

for(j=O;j<=9;j++)
{
if (d[O)==n[j 1 l
{
J=j;

.
'

18.3 Fuzzy Logic Implementation 559

)
u[J][O]+=a*(y-u[J] [0]);
t[J] [O]+=b* (x-t[J] [0] l;
printf("\n Input=%f~,x);
printf("\n Updated wights u: ");
for(j=O;j<=9;j++)
{
print ( q%f\t", u [j 1 [0));
n{j]=O;
d{j]=O;
}
printf("\nUpdated weights t:");
for (j=O;j <=9; j ++}
{
print ( "%f\t", t[j 1 [0] l;

k=O;
J=O;
a=a/1. 014 ,
b=b/1.014;
x=x+0.5;
y=1/x;
getch();
)while(x<=10.5);
getch();
}

118.3 Fuzzy Logic Implementation

The various concepts of fuzzy logic through chapters Chapters 7-14 are implemenred using C and C++
languages in this section. The source code for the same is a5 given below.

18.3.1 Implement the Various Primitive Operations of Classical Sets

The program is a5 follows:

#include<stdio.h>
#include<conio.h>
#include<string.h>
#include<alloc.h>
struct SET

char *elts;
int n;
};
typedef struct SET set;
set s;
void getval(set m,char x)
i
{
int i;
I
.fi
560 Soft Computing Techniques Using C and C++

printf("\n Enter the %c:\n",x);

for (i=O; i2 *s ._n)
{
printf(~\n Invalid values");
getch();
exit(O);

a.elts=(char *) malloc(a.n);
b.elts=(char *) malloc(b.n);
getval(s,'S');
getval (a, 'A');
getval (b, 'B');

while(l)
{

printf("\n Menu:\n l.AUB\n 2.A B\n 3.A-\n 4.8""' \n S.Print

S,A,B\n 6.Exita);
switch((ch=getch{)))
{
case '1':
ans=unionset(a,b);
printval(ans, 'U');
getch();
break;
case '2':
ans=interset(a,b);
printval(ans, '~');
getch();
break;
case '3':
ans=complement(a);
printval(ans, 'a');
getch();
break;
case '4':
ans=complement(b);
printval(ans, 'b');
getch();
break;
case '5':
printval(s,'S');
printval(a,'A');
printval (b, '8');
getch();
break;
case '6':
exit(O);

[,,,
.,
1.

i
~
 ~~~
I I
18.3 Fuzzy Logic Implementation 561 k
~:
Omput ~~
Enter the no of elts in sample space:S. t
Enter the no of elts in A:3 !f.
Enter the no of elts in B:2
i
Enter the S:
Element 1:1
Element 2:2
Element 3:3 ,I
Element 4:4 . j
Element 5:5

Enter the A:
Element 1:1
Element 2:2
Element 3:3

Enter the B:
Element 1:3
Element 2:4

Menu: ~
l.AUB }
2.AAB I
3.A"-
4 .B......
5.Print S,A,B '
6.Exit ''
;
U={1,2,3,4} >
={3}
a={4, 5)
b={l,2,5}
S::o{1,2, 3,4, 5)
A={1,2,3)
8={3,4}

18.3.2 To Verify Various Laws Associated with Classical Sets

The program is as follows:

~include<stdio.h>
~include<conio.h>
#include<string.h>
#include<stdlib.h>

struct SET
{
char *elts;
int n;
};

~
 562 Soft Computing Techniques Using C and C++

typedef stru?t SET set;

set s;

void getval(set m,char *x)

[
int i;
printf("\n Enter the %s:\na,x);
for(i=O;i3*s.n)
[
printf("\n Invalid values");
getch();
exit(O);

a.elts=(char *) malloc(a.n);
b.elts={char *) malloc(b.n);
c.elts=(char *) malloc(a.n);
getval(s,"S");
getval(a,"A");
getval (b, "B");
getval(c,"C");
clrscr();
printf("\n Menu: \n l.DeMorgan's Law\
\n 2.Associative Law\
\n 3.Distributive Law\
I
I'
\n 4.Comrnutative Law\
\n S.Exit");
while(!)
[
switch( {ch=getch()))
[
case '1':
clrscr();
print ( "\n DeMorgan' s Law: (A B) -=A-UB"'");
tl=intersect(a,b);
printval(tl,"A~B");
t2=complement(tl);
printval (t2, (A~B) ....... ) ;

tl=complement{a);
printval{tl, "A-");
t4=complement(a);
t2=complement{b);
printval(t2, "B-~);
ans=unionset(tl,t2);
printval(ans, "A-UB-n);

printf{"\n DeMorgan's Law: (AUB) "'=A""' ~ 8"'");

tl=unionset(a,b);
printval (tl, "AUBR);
ans=complement(tl);
 563
18.3 Fuzzy logic Implementation

printval(t2," (AUBJ-");

tl=complement (a);
printval(tl, "A-");
t2=complement{bl;
printval (t2, "8""'");
ans=intersect(tl,t2);
printval (ans, "A- ~ 8""'") ;
break;
case '2':
clrscr(); Alec) ),
printf("\n Associative Law: (A~B)"C
tl=intersect (a, b);
printval {tl, "A~B");
t2=intersect(tl,cl;
printval{t2,"(A"B)"C");

tl=intersect(b,c);
printval(tl, "B"C");
t2=intersect(tl,a);
printval(t2, "A" (B"C)");
AU(BUC)");
printf{"\n Associative Law:(AUB)UC
tl=unionset(a,b);
printval(tl,"AUB");
t2=unionset(tl,c);
printval(t2," (AUB)UC");

tl=unionset(b,c);
printval(tl,"BUC");
t2=unionset{tl,a);
printval(t2, "AU(BUC) ");
break;
case '3':
clrscr(); {A"B) U (A"C)");
printf ( "\n Distributive Law: (AUB) "C
I tl=unionset(a,b);
! printval (tl, "AUB");
t2=intersect(tl,c);
I printval(t2,"(AUB)"C");

tl=intersect{a,b);
printval{tl,RA"B");
t2=intersect{a,c);
printval{tl,"A"C");
ans=unionset(tl,t2);
printval(ans."(A"B)U(A"C)");

printf ( "\n Distributive Law: (A"B)U C= (AUB) ... (AUC) );

tl=intersect(a,b);
printval ( tl, A"B");

i
 ~I

~tz
564
Soft Computing Techniques Using C and C++

t2=unionset(tl,c);
printval (t2, ~ (AAB) UC");

tl=unionset(a,b);
printval(tl, "AUB");
t2=unionset(a,c);
printval(tl,"Auc~);
t2=intersect(tl,t2);
printval{t2,"(AUB)A(AUC)");
break;
case '4':
printf("\n Commutative Law: AUB=BUA");
tl=unionset(a,b);
print val (tl, "AUB");
tl=unionset(b,a);
printval(tl,"BUA");
printf("\n Commutative Law: AAB=BAA");
tl=intersect(a,b);
printval(tl,"AAB");
tl=intersect(b,a);
printval(tl, "B"A");
break;
case '5':
exit(O);
default:
putch( '\a');

putch( '\n');
printval (s, "S");
printval (a, "A");
printval (b, "8");
printval (c, "C");
getch(J;

Output

Enter the no of elts in sample space:3

Enter the no of elts in A:2
Enter the no of elts in 8:2
Enter the no of elts in C:2
Enter the S:
Element 1:1
Element 2:2
Element 3:3

Enter the A:
Element 1:1
Element 2:2

JL
 f:.

18.3 Fuzzy Logic Implementation 565 r,

i:
"
Enter the B:
Element 1:2
Element 2:3

Enter the C:
Element 1:1
Element 2:3

Menu:
l.DeMorgan's Law
2.Associative Law
).Distributive Law

Il
4.Cornmutative Law
S.Exit

DeMorgan' s Law: (A"B)""'=A""'UB.....

A'B =(2)
(A'B)- =(1,3)
A- =(3)
B- =(1)
A""'UB"' =(1}
DeMorgan's Law: (AUB)-=A..., A 8""'
~~
'1
AUB ={1,2,3} H
(AUB)- =(1) '.
A- =(3) .:
a..... ={1}
A"- h 8"' ={1} ''
Associative Law: (AAB)AC = A"(BAC)
'
AAB =(2)
(A-B)"C ={)
'
a-c ={3}
A" (B"C) ={}

Associative Law: (AUB)UC = AU(BUC)

AUB ={1,2,3}
(AUB)UC ={1,2,3}
BUC =(2,3,1}
AU(BUCI ={2,3,1}

Distributive Law: (AUB)-C = {AAB) U (A-C)

AUB ={1,2,3}
(AUB)-c ={1.3)
231 ={2}
A'C =(1)
(A'B)U(A'C) =(1)

Distributive Law: (A"B)U C=(AUB)-(AUC)

A'B =(2}
(A-B)UC ={2,1,3}
 566 Soft Computing TeChniques Using C and C++

AUB ={1,2,3}
AUC ={1,2,3}
IAUB)'IAUC) =[1,2,3)

Commutative Law: AUB=BUA

AUB =11,2,3)
BUA ={2,3,1}

Commutative Law: AAB=BAA

A'B =12)
B'A =[2)
s =[1,2,3)
A =[1, 2)
B =[2,3)
c =[1,3)

18.3.3 To Perform Various Primitive Operations on Fuzzy Sets with Dynamic Components

The program is as follows:

r #include<stdio.h>
#include<alloc.h>
tinclude<conio.h>
#include<stdlib.h>

struct SET
I
float nr[S];
float dr(S);
int n;
);

typedef struct SET fuzzy;

void getval(fuzzy *m,char *x)

I
int i;
float f;
clrscr ();
printf("\n Enter the %s:\n",x);
for(i=O;i<m->n;i++)

printf ( ~ Numerator Element %d : ", i+l) ;

scanf("%f",&f);
m->nr(i]=f;
fflush(stdin);
printf("Denominator Element %d:",i+l);
scanf("'U",&f);
m->dr[i]=f;
 567
18.3 Fuzzy logic Implementation

void printval{fuzzy *m,char *x)

[
int i;
printf{"\n %s=(,x);
for(i=O;i<m->n;i++)
[
printf("%6.2 I %6.2f~,m->nr(i},m->dr[i]);
if{i!=m->n-1) putch('+');
)
print (")");

fuzzy unionset{fuzzy a, fuzzy b)

fuzzy temp;
char ch;
int i;
temp.n=a.n;
for (i=O; i<a.n; i++)

if (a.dr[i] !=b.drlil)
I
printf ( "\n Denominators not equal");
getch();
exit (0);
)
if(a.nr[i)<b nr{i])
temp. nr I i l=b. nr[ i) ;
else
temp.ndi]=a.nr[i);
temp.dr[i)=a.dr(i];

return temp;

fuzzy intersect(fuzzy a, fuzzy b)

i
. iZ., I
fuzzy temp;
int i;
temp.n=a.n;
for(i=O;i<a n; i++)
I
if(a dr[il !=b.ddill
[
printf ( \n Denominators not equal"') ;
.,,_ getch();
~ exit(O);
I
-_.;.,' if(a.nr[i]>b.nr[i))
$;,
"''
ci
568
Soft Canputing Techniques Using C and C++
temp.nr[i]=b.nr[i];
else
temp.nr[i)=a.nr[i];
temp.dr[i]=a.dr[i];

return temp;

fuzzy complement(fuzzy a)
(
fuzzy temp;
int i;
temp.n=a.n;
for{i=O;i<a.n;i++)
(

temp.nr[i]=l-a.nr[i];
temp.dr[i]=a.dr[i];

return temp;

void main()
(

fuzzy a,b,ans;
char ch;
clrscr ();
printt(~\n Enter the no of componets:");
scanf("%d",&a.n);
b.n=a.n;
getval (&a, "A");
getval (&b, "B");
clrscr{);
printval (&a, "A");
printval {&b, "8");
getch();
l.,hile(l)
(
clrscr();
printf{"\n Menu:\n l.AUB\n 2.A~B\n 3.A"-'\n 4.8 ...... . 'j
S,A,B\n 6.Exit"); \n S.Print
switch((ch=getch()))
(
case '1':
ans=unionset(a,b);
printval(&ans,"AUB);
getch();
break;
case '2':
ans=intersect(a,b);
printval (&ans, "A"B) ,

.........
 il'

18.3 Fuzzy Logic lmplemante.tion 569 ,rr~

getch(l;
i
break;
case '3':
(
ans=complement(a);
printval(&ans, "A-);
getch();
break; I
case '4':
ans=complement(b); !l
printval(&ans,s-");
getch(); ."
t

break;
case '5': ~
printval{&a,"A"); 1
printval(&b,"B");
getch();
1
1
break;
case '6':
exit (0);

..,
h
~":
Output
Enter the no of componets:3

Enter the A:
Numerator Element l :0.4
Denominator Element 1:1
Numerator Element 2 :0.2
Denominator Element 2:2
Numerator Element 3 :0.7
Denominator Element 3:3

Enter the 8:
Numerator Element l :0.4
Denominator Element 1:1
Numerator Element 2 :0.8
Denominator Element 2:2
Numerator Element 3 :0.2
Denominator Element 3:3

A={ 0.40 I 1.00+ 0.20 I 2.00+ 0.70 I 3.00

B={ 0.40 I 1.00+ 0.80 I 2.00+ 0.20 I 3.00

Menu:
l.AUB
2.A~B

3 .A"'
 570 Soft Computing Techniques Using C and C++

4.8 .....
S.Print S,A,B
6 .Exit

AUB={ 0.40 I 1.00+ 0.80 I 2.00+ 0.70 I 3.00 }

AAB={ 0.40 I 1.00+ 0.20 I 2.00+ 0.20 I 3.00 }
A-={ 0.60 I 1.00+ 0.80 I 2.00+ 0.30 I 3.00 }
B-={ 0.60 I 1.00+ 0.20 I 2.00+ 0.80 I 3.00}

18.3.4 To Verify the Various Laws Associated with Fuzzy Set

The program is as follows:

#include<stdio.h>
flinclude<alloc.h>
#include<conio.h>
#include<stdlib.h>

struct SET

',.
.... float nr[S]:
float dr[S];
int n;
\. ) ;
I
' typedef struct SET fuzzy;
f
void printval(fuzzy *m,char *x)
{
int i;
print ( ~\n %s= {", x);
for(i=O;i<m->n;i++)

print(" %6.2 I %6.2f",m->nr[i],m->dr[i));

if(i!=m->n-1) putch('+');

printf(" }");

fuzzy unionset(fuzzy a, fuzzy b)

{
fuzzy temp;
char ch;
int i;
temp.n=a.n;
for(i=O;i<a.n;i++)
{
if (a.dr[i] !=b.dr[i) l
{
print ( "\n D.enominators not equal");
getch();
 571
18.3 Fuzzy logic Implementation

exit{Ol;
)
if{a.nr[i]<b.nr[i])
temp.nr[i]=b.nr[i];
else
ternp.nr[i]=a.nr[i);
temp.dr[i]=a.dr[i];

return temp;

fuzzy intersect{fuzzy a, fuzzy b)

{
fuzzy temp;
int i;
temp.n=a.n;
for{i=O;i<a.n;i++l

if{a.dr[i]!=b.dr[i]l
{
print ( "\n Denominators not equal");
getch,YI ;
exit(O);

if(a.nr[i]>b.nr[i))
ternp.nr[i]=b.nr[i];
else
ternp.nr[i)=a.nr[i);
ternp.dr[i]=a.dr[i];

return temp;

fuzzy cornplernent{fuzzy a)
{
fuz.zy temp;
int i;
temp.n=a.n;
for(i=O;i<a.n;i++)

temp.nr[i]=l-a.nr[i);
temp.dr[i]=a.dr[i];

return temp;

void main()
{
fuzzy a,b,templ,temp2,ans;
char ch;
 .0'::-'F-'
572
Soft Computing Techniques Using C and C++

clrscr{);
a.n=b.n=3;
a.nr[O]=O.l; a.dr(O]=l;
a.nr[1]=0.2; a.dr[1]=2;
a.nr[2]=0.3; a.dr[2]=3;
b.nr[0]=0.4; b.dr[O]=l;
b.nr[l]=0.3; b.dr[l]=2;
b.nr{2]=0.2; b.dr[2]=3;
printval(&a,~A");
printval(&b, "B");
getch();
printf("\n Menu:\n !.Difference A/B \n 2.Difference 8/A\n
3.DeMorgan's law -1\n 4.DeMorgan's law -2\n 5.Excluded Middle
laws\n 6.Print S,A,B\n ?.Exit");
while(!)
I
switch( (ch=getch()))
{
case '1':
templ=complement(b);
printval (&temp!," 1. B-");
printval(&a,"A");
ans=intersect{a,templ);
printval (&ans, "A/8 = A"B ...... );
break;
case '2':
templ=complement{a);
printval(&templ,"2.A-");
printval(&b, ~B");
ans=unionset(a,templ);
printval (&ans, "A/B = 8-A-");
break;
case '3':
ans=unionset(a,b);
ans=complement (ans) ,
printval(&ans, a3. (AUB)-");
templ=complement(a);
temp2=complement(b);
printval (&templ, "A-");
printval{&temp2, "B-");
ans=intersect(templ,temp2);
printval (&ans, "A"-"8-");
break;
case '4':
ans=intersect(a,b);
ans=complement(ans);
printval(&ans, "4. (A"B)-");
templ=complement(a);
temp2=complement(b);
printval (&templ, "A-...");

1
 18.3 Fuzzy Logic Implementation 573 I;
printval(&temp2,aB"'");
ans=unionset(templ,temp2);
printval (&ans, "A"' U 8"-'");
break;
case '5':
ans=complement(a);
ans=unionset(ans,a);
printval (&ans, "5 .A-"A");
ans=complement(b);
ans=unionset(ans,b);
printval(&ans,"B-B");
break;
case '6':
printval(&a,"A"l;
printval (&b, "B");
break;
case '7':
exit(O);

\
Output '
Menu:
!.Difference AlB
2.Difference BIA
3.DeMorgan's law -1
4.DeMorgan's law -2
5.Excluded Middle laws
6.Print S,A,B
7.Exit
l.B-={0.60 I 1.00+ 0.70 I 2.00+ 0.80 I 3.00}
A={O.lO I 1.00+ 0.20 I 2.00+ 0.30 I 3.00)
AlB = A"B-...={0.10 I 1.00+ 0.20 I 2.00+ 0.30 I 3.00}
2.A-=(0.90 I 1.00+ 0.80 I 2.00+ 0.70 I 3.00}
8={0.40 I 1.00+ 0.30 I 2.00+ 0.20 I 3.00}
AlB= B"A-={0.90 I 1.00+ 0.80 I 2.00+ 0.70 I 3.00}
3. (AUB)-={0.60 I 1.00+ 0.70 I 2.00+ 0.70 I 3.00}
A-={0.90 I 1.00+ 0.80 I 2.00+ 0.70 I 3.00}
B-={0.60 I 1.00+ 0.70 I 2.00+ 0.80 I 3.00}
A- "8-...={0.60 I 1.00+ 0.70 I 2.00+ 0.70 I 3.00]
4.{A"B)-={0.90 I 1.00+ 0.80 I 2.00+ 0.80 I 3.00}
A-={0.90 I 1.00+ 0.80 I 2.00+ 0.70 I 3.00}
B-={0.60 I 1.00+ 0.70 I 2.00+ 0.80 I 3.00}
A"- U B-={0.90 I 1.00+ 0.80 I 2.00+ 0.80 I 3.00}
5.A-..."A={0.90 I 1.00+ 0.80 I 2.00+ 0.70 I 3.00}
B- 8={0.60 I 1.00+ 0.70 I 2.00+ 0.80 I 3.00}
A={O.lO I 1.00+ 0.20 I 2.00+ 0.30 I 3.00}
8={0.40 I 1.00+ 0.30 I 2.007 0.20 I 3.00}

1
 574 Soft Computing Techniques Using C and C++

18.3.5 To Perform Cartesian Product Over Two Given Fuzzy Sets

~
tinclude<stdio.h> 1::~
#include<limits.h>
#include<alloc.h>
#include<conio.h>
#include<stdlib.h>
#define min(x,y} (x<y ? x : y)

struct SET
(
float nr[S];
float dr[S];
int n;
);
typedef struct SET fuzzy;

void printval(fuzzy *m,char *x)

(
int i;
(. print ( "\n %s={" ,x);
for(i=O;i<m->n;i++)
(
printf{" %5.2 /%5.2 ,m->nr[i] ,m->dr[i]);

l if(i!=m->n-1) putch('+'l;

printf( }");

void main()
I
fuzzy V,I;
int L j;
float P[6] [6];
clrscr();

V.n=I.n=S;
V.nr[0]=0.2; V.dr(0]=30;
V.nr[l]=O.B; V.dr[l]=45;
V.nr[2]=1; V.dr[2)=60;
V.nr[3]=0.9; V.dr[3)=75;
V.nr[4]=0.7; V.dr[4)=90;

I.nr[0]=0.4; I.dr[OJ=O.B;
I.nr[l}=0.7; I.dr[l)=0.9;
I.nr[2)=1; I.dr[2)=1;
I.nr[3]=0.8; I.dr[3)=1.1;
I.nr(4]=0.6; I.dr[4)=1.2;

printval(&V,gV");
 575
18.3 Fuzzy Logic Implementation

printval (&I, "I");

print ( \n");
for(i=O;i<=V.n;i++)
for(j=O;j<=I.n;j++)
(
if(i==O && j>O)
P[i) [j]=I.dr[j-1);
else if(j==O && i>O)
P[i] [j}=V .dr[i-1];
else if(i>O && j>O)
P[i} [j]=min(V.nr(i-1] ,I.nr[j-1]);

for(i=O;i<=V.n;i++)
(
for (j=O; j <=I .n; j++)
[
if (i==O && j==Ol
print(" );
else
print(" %6.2 ,P[i][j]);

printf(u\no);

getch();

Output
V={0.20/30.00 + 0.80/45.00 + 1.00/60.00 + 0.90/75.00 + 0.70/90.00)
+ 0.70/0.90 + 1.00/1.00 + 0.80/1.10 + 0.60/1.20}
I={0.40/0.80
Vxl=
1.20
0.80 0.90 1.00 1.10
0.20 0.20 0.20
30.00 0.20 0.20
0.40 0. 70 0.80 0.80 0.60
45.00
0.60
60.00 0.40 0.70 1.00 0.80
0.60
75.00 0.40 0.70 0.90 0.80
0.40 0.70 0.70 0. 70 0.60
90.00

18.3.6 To Perform Max-Min Composition of Two Matrices Obtained from

1- Cartesian Product

The program is as follows:

~include<stdio.h>
ninclude<limits.h>
#include<alloc.h>
#include<conio.h>
#include<stdlib.h>

idefine min(x,y) (x<y? x y)

!

b
576
Soft Computing Techniques Using C and C++

struct SET
{
float nr[S];
float dr[SJ;
int n;
};

typedef struct SET fuzzy;

void printval(fuzzy *m,char *x)
{
int i;
printf("\n %s={",x);
for(i=O;i<m->n;i++)
{

print(" %5.2 /%5.2 ",m->nr[i],m->dr[i));

if(i!=m->n-1) putch('+');

print("}");

void main( J

fuzzy V,I,C;
int i,j,k,prows,pcols,trows,tcols;
float P[6] (6] ,T[6] (4] ,E(6] [4],max;

clrscr();

V.n=I.n=S; C.n=3;
V.nr[0)=0.2; V.dr[0]=30;
V.nr[l]=O.B; V.dr[1]=45;
V.nr[2]=1; V.dr(2)=60;
V.nr[3]=0.9; V.dr(3]=75;
V.nr[4]=0.7; V.dr(4)=90;
I.nr[O]o::0.4; I.dr(O]=O.B;
I.nr[l]=0.7; I.dr[l]=0.9;
I.nr[2)=1; I.dr(2]=1;
I.nr[3]=0.8; I.dr(3J=l.l;
I.nr[4]=0.6; Ldr[4]=1.2;
C.nr[0]=0.4; C.dr[O)=O.S;
C.nr[l)=l; C.dr[l]=0.6;
C.nr[2]=0.5; C.dr[2]=0.7;

printval {&V, "V");

printval (&I," r l;
printval (&C, "C"):
printf("\n M=Vxi=");

for(i=O;i<=V.n;i++)

J
 18.3 Fuzzy Logic Implementation 577

for(j=O;j<=I.n;j++l
{
if (i==O && j>Ol
P[i] [j]=I.dr{j-1];
else if(j==O && i>O)
P[i] [j]=V.dr[i-1);
else if{i>O && j>O)
P [i] [j] =min{V .nr [i-1], I .nr [j -1]);

for(i=O;i<=I.n;i++l
for{j=O;j<=C.n;j++l
{
if (i::-=0 && j>O)
T(i] [j]=C.dr[j-1];
else if(j==O && i>O)
T[i] [j]=I.dr[i-1];
else if(i>O && j>O)
T[i] [j]=min(I.nr[i-1] ,C.nr[j-1]);
}
for(i=O;i<=V.n;i++)
{
for(j=O;j<=I.n;j++)
{
if (i==O && j==O)
printf(" );
else
print{" %6.2 ",P[i][j]);
I
printf("\n");
}
print ( "\n N=IxC=" l;
for(i=O;i<=I.n;i++)

for(j=O;j<=C n;j++)
{
i f (i==O && j==O)
print("~);
else
print(" %6.2 ",T[i][j]);

print { "\n");

prows=6,pcols=6;
trows=6,tcols=4;
for(i=O;i<prows;i++l

for(j=O;j<tcols;j++)
{

J
 578
Soft Computing Techniques Using-c and C++

if{i==O && j==O)

E[i] [j]=O;
else if{i==O && j>O)
E[i) [j)=T[i) [j);
else if(i>O && j==O)
E[i] [j)=P[i] [j] i
else
{
max=O;
for(k=l;k<pcols;k++)
{
if (i>O && j>O)
if(max <: min(P[i][k],T[k][j)))
max=min{P[i] [k] ,T[k] [j]);

if(i>O && j>O)

E[i] [j]=max;

~:
getch();

printf("\n M oN");
for(i=O;i<prows;i++)
{
for (j=O; j <teals; j ++)
I
if(i==O && j==O)
print("");
else
print(" %6.2f ",E[i}[j]);

printf("\n");

getch ();

Output

V={0.20/30.00 + 0.80/45.00 + 1.00/60.00 + 0.90/75.00 + 0.70/90.00}

1={0.40/0.80 + 0.70/0.90 + 1.00/1.00 + 0.80/1.10 + 0.60/1.20)
C={0.40/0.50 + 1.00/0.60 + 0.50/0.70}

M=Vxi=
0.80 0.90 1. 00 1.10 1.20
30.00 0.20 0.20 0.20 0.20 0.20
45.00 0.40 0.70 0.80 0.80 0.60
60.00 0.40 0.70 1. 00 0.80 0.60
75 .oo 0.40 0.70. 0.90 0.80 0.60
90.00 0.40 0.70 0.70 0.70 0.60
 18.3 Fuzzy logic Implementation 579

N=IxC=
0.50 0.60 0.70
0.80 0.40 0.40 0.40
0.90 0.40 0.70 0.50
1.00 0.40 1.00 0.50
1.10 0.40 0.80 0.50
1.20 0.40 0.60 0.50

Mo N
0.50 0.60 0.70
30.00 0.20 0.20 0.20
45.00 0.40 0.80 0.50
60.00 0.40 1.00 0.50
75.00 0.40 0.90 0.50
90.00 0.40 0.70 0.50

18.3. 7 To Perform Max-Product Composition of Two Matrices Obtained from

Cartesian Product

The program is as follows:

#include<stdio.h>
#include<limits.h>
#include<alloc.h>
#include<conio.h>
#include<stdlib.h>

#define product(x,y) ({x)~(y))

struct SET
I
float nr[5];
float dr[S];
int n;
);

typedef struct SET fuzzy;

void printval(fuzzy *m,char ~x)
I
int i;
printf ( "\n 'ts={" ,x);
for(i=O;i<m->n;i++)
I
printf(" %5.2f /%5.2f ",m->nr[i),m->dr[i));
if(i!=m->n-1) putch('+');

printf{~ }");

l
void main()
(
580
Soft Computing Techniques Using C and C++
.-;r:,.
fuzzy V,LC; ~:'

int i. j_, k,prowS,pcols, trows, teals; ~

float PIG) 161 ,TI6) 141 ,EI61 14) ,max; .

;~
clrscr(); ')_
ic>-

V.n=I.n=S; C.n=3;
V.nr[0J=0.2: V.dr0]=30;
V.nr[l]=O.B; V.dr[l)=45;
V.nr[2]=1: V.drl2)=60;
v.nr[3)=0.9; V.drl3)=75;
V.nr[4]=0.7; V.drl4)=90;

I.nr[0)=0.<1; I.dr[0)=0.8;
I.nr[l]=O. 7; I.dr[l)=0.9;
Lnr[2]=1: I.dr[2]=1:
I.nr[3]=0.8; I.dri3J=l.l;
I.nr[4]=0.6; I.drl41=1.2;

C.nr[0]=0.4; C.dr[O)=O.S;
C.nr[l]=l; C.dr[ll=0.6;
C.nr[2]=0.5; C.dr[2]=0. 7;

printval (&V, "V");

printval(&I, "1");
printval(&C,"C");
printf("\na);

for(i=O;i<=V.n;i++)
for(j=O:j<=I.n;j++)
I
i f (i==O && j>O)
P[i) [j]=I.dr[j-1];
else if(j==O && i>O)
P[i) [j)=V.dr[i-1];
else if(i>O && j>O)
P[i] [j]=min(V.nr[i-1] ,I.nr[j-1]);

for(i=O;i<=I.n;i++)
for (j=O;j<=C. n; j++)
I
if(i==O && j>O)
Tli) liJ=C.drlj-1);
else if(j==O && i>O)
T[i) [j]=I.dr[i-1};
else if(i>O && j>O)
T[i] [j] =min (I .nr[i-1), C. nr [j -1));
 581
t.
18.3 Fuzzy Logic Implementation
~
for(i=O;i<=V.n;i++l i
for(j=O;j<=I.n;j++)
{
t
i
if (i==O && j==Ol
print(" "); ~~
else .1
print(~ %6.2 ",P[i][j]);

I '
printf("\n");
I
print ( "\n");
for(i=O;i<=I.n;i++l

for{j=O;j<=C.n;j++)
{ ~
if(i==O && j==O)
print(" "); i
)
else
print{" %6.2 ",T[i] [j)); I5
~~'
print {"\n");

:\
prows=6,pcols=6;
trows=6,tcols=4; ,,-,
. ;:: ~

for ( i=O; i<prows; i ++) .. ,-

I '
for(j=O;j<tcols;j++) ~
{
if(i==O && j==O)
Eli) lii=O;
else if(i==O && j>O)
E[i] [j]=T[i] [j];
else if{i>O && j==O)
E[i][j]=P(i] [j];
else

max=O;
for{k=l;k<pcols;k++l
{
if {bO && j>O)
if{max < product(P(i][kl.T(k][j)Jl
max=product{P[i) [k] ,T[k] [j]);

if(i>O && j>O)

EliJ lil=max;
 582 g
Soft Computing Techniques Using C and C++

getch();

print(~\n");
for(i=O;i<prows;i++)
{
for(j=O;j<tcols;j++)
{
if(i==O && j==O)
print("~);
else
print( %6.2 ",E[i][j]);

printf("\n");

getch();

Output

V={0.20/30.00 + 0.80/45.00 + 1.00/60.00 + 0.90/75.00 + 0.70/90.00}

!={0.40/0.80 + 0.70/0.90 + 1.00/l.OO + 0.80/1.10 + 0.60/1.20}
C={0.4o;o.so + l.00/0.60 + o.so;o.?oJ

M=Vxr
o.ao 0.90 1.00 1.10 1. 20
30.00 0.20 0.20 0.20 0.20 0.20
45.00 0.40 0.70 0.80 0.80 0.60
60.00 0.40 0.70 1.00 0.80 0.60
75.00 0.40 0.70 0.90 0.80 0.60
90.00 0.40 0.70 0.70 0.70 0.60
N=Ixc
0.50 0.60 0.70
0.80 0.40 0.40 0.40
0.90 0.40 0 70 0.50
1. 00 0.40 1.00 0.50
1. 10 0.40 0.80 0.50
1. 20 0.40 0.60 0.50

Mo N
0.50 0.60 0.70
30. oo 0.08 0.20 0.10
45.00 0.32 0.80 0.40
60.00 0.40 1. oo 0.50
75.00 0.36 0.90 0.45
90.00 0.28 0.70 0.35

118.4 Genetic Algorithm Implementation

The generic algorithm concept discussed in Chapter 15 is brought for various applications using C and C++
in chis section. The source cOde for each application is given below.
g
18.4 Gene~c Algorithm Implementation 583

I 18.4.1 To Maximize F(x,, x2) ~ 4x, + 3x2

To find the solution of the function Max F(x1, X2) = 4:q + 3X2., given the constraints
2*xi+3*X2.:S6
-3*XJ +2*X2 ::;3
2*xi+xz:::4
O::;xs2

using genetic algorithm.

Steps involved
,--------------------- -1
I Step 1: Generate initial population by using random number generation.
Step 2: Use the tournamem sdection method to select any two parents.
Step 3: Generate the offsprings by using ilie following arithmetic crossover operator:

Xi =a*xl +(l-a)*X2
x2 =a*X2+ (1-a)*xl

Step 4: Calculate the maximum f1mess value by applying this operation separately for each iteration.
I Step S: To prim the output of the function. I
#include<stdio.h>
#include<conio.h>
#include<dos.h>
#include<stdlib.h>
#include<math.h>
float mutation(float ,int ) ;
//Main Program
void main ()
{
float xl[lO],x2[10],sum[l0],max_val=O.O,a,max_xl,max_x2;
int flag=O,j,i,k;
clrscr ();
randomize ();
//Initial Population Generation
printfi. ( "\tlni tial Population \n") ;
for(i=~;i<4;i++)
{
xl [i] =(float) (random (1000. 0)) /500.0;
x21il =(float) {random (1000. 0)) /500.0;
printf("\t%d\txl: %f\tx2: %f\n",i,xl[iLx2[i]);
)

for(i=O;i<lO;i++)
{
for ( j =0; j <4; j ++)

l
584
Soft Computing Techniques Using C and C++

-~~.-
flag=O;
f /Constraints
if ( ( (2*xl[jJ )+(3*x2 [j]) l <=6) //2xl+3x2<=6
{
if ( ( ( -3"'xl [j]) + (2*x2 (j 1 l) <=3) I /-3x1+2x2<=3
{
if ( ( (2*xl {j]) + (X2 [j])) <=4) I /2xl+X2<=4
{
flag=l;
sum[j)=(4*Xl [j] )+(3*x2 [j]);

if(flag==O)
sum(j]=O;
)

printf("\t After %d generation\n",i);

for(k=O;k<4;k++)
{
printf("\t%d\txl %f\tx2 %f\tsurn %\n", k, xl [k] ,x2 {k], surn[k]);

for(k=O;k<4;k++l
{
if(max_val<sum[k))
{
max_val=sum[k);
max_xl=xl[k];
max_x2=x2 [ k];

printf("\txl : %f\tx2 : %f\tMax: %f\n",max_xl,max_x2,max_val);

getch();
for(k=O;k<:3;k++)
{
//cross over operation
a=(float) (random(l000.0))/1000.0;
xl [k] =( (a*xl [k] J + (1-a) *x2 [k+l]);
x2 [k] =( (a*x2 [k]) + (1-a) *xl[k+l]);

a=(float) (random(l000.0))/1000.0;
x1 [41= ( (a*x1 [4] )+ (1-a) *x2 [1]);
x2 [4] =( (a*x2 [4)) + (1-a) *x1 [1]);

//Mutation Operation
for(k=O;k<4;k++)
{
if(sum[k]==O) I
{

l
18.4 Genetic Algorithm Implementation 585

xl[k]=(xl[O]+xl[l]+k1[2]+x1[3])/4.0;
x1[k]=(x2[0]+x2[1J+x2[2]+x2[3])/4.0;
) ''
if(sum[k]==sum[k+lJ)
{
xl [k] =(float) (random(lOOO. 0)) /500'. 0;
x2[k]=(float) (random(lOOO.O) )/500.0;
)

clrscr();
printf ( "\tTHE SOLUTION OF THE FOLLOWING PROBLEM IS\n");
printf ( "\n\tSUM: \tMAXIMIZE\tF(xl, x2) : 4xl+3x2);
printf ( "\n\n\t2 *x1+3 *x2<=6\n\t-3 *x1+2*x2<=3\n\t2 *xl+x2<=4 \n") ;
printf ( "\n\txl : %f\tx2 : %f\tMax : %f\n" ,max_xl ,max_x2,max_val);
getch();
)

void cross_over(float xl,float x2)

{
int a; ,,,
a={float) (random(1000J) /1000;
xl= ( (a*xl) + (1-a) *x2);
x2= ( (a*x2) + (1-a) *x1l;

float mutation(float x,int i)

{
)
float max(float sum)
{
int i;
for(i=O;i<3;i++)
{
if(surn[i]<sum[i+l))
max= sum [ i + 1] ;
else
max=sum(i];
return max;

Output
Initial Population
0 xl 0.432000 x2 1.366000
1 xl 1.452000 x2 0.468000
2 xl 0.928000 x2 0.854000
3 xl 0. 860000 x2 1.342000

After 0 generation
0 x1 : 0.432000 x2 1. 366000 sum 5.826000
586 Soft Computing Techniques Using C and C++

1 x1 1. 452000 x2 0.468000 sum 7.212000

2 x1 0. 928000 - x2 0.854000 sum 6.274000
3 x1 0.860000 x2 1.342000 sum 7.466000
x1 0.860000 x2 : 1. 342000 Max 7.466000

After 1 generation
0 x1 1.477638 x2 1.397484 sum 0.000000
1 x1 1.364000 x2 0.960000 sum 8.336000
2 x1 0.820292~ x2 : 0.933684 sum 6.082220
3 x1 0.524000 x2 0.990000 sum 5.066000
x1 1.364000 x2 0.960000 Max 8. 336000

After 2 generation
0 x1 0.979173 x2 1.364904 sum 0.000000
1 x1 1. 039542 x2 0.854660 sum 6.722147
2 x1 0.914141 x2 0.707129 sum 5. 777948
3 x1 0.524000 x2 0.990000 sum 5.066000
x1 1.364000 x2 0.960000 Max 8.336000

After 3 generation
0 xl 1. 342291 x2 1.076486 sum 8.598619
1 xl 0.922836 x2 1.088421 sum 6.956608
2 xl 0.784797 x2 0.836438 sum 5.648500
3 x1 0.782000 x2 0.690000 sum 5.198000
xl 1.342291 x2 1.076486 Max 8. 598619

Afi:er 4 generation
0 x1 1 180576 x2 0.978611 sum 7.658136
1 x1 0 858469 x2 0.862221 sum 6.020540
2 x1 0 774369 x2 0.830450 sum 5.588825
3 x1 0 782000 x2 0.690000 sum 5.198000
x1 1.342291 x2 1.076486 Max 8.598619

After 5 generation
0 xl 1.106081 x2 0.950498 sum 7.275816
1 x1 0.842442 x2 0. 811970 sum 5.805677
2 x1 0. 757664 x2 0.820857 sum 5.493226
3 x1 0.782000 x2 0.690000 sum 5.198000
x1 1.347.291 x2 1.076486 Max 8.598619

Mter 6 generation
0 x1 1. 071964 x2 0.937963 sum 7.101746
1 x1 0.839420 x2 0. 804367 sum 5.770781
2 x1 0.695954 x2 0.785419 sum 5.140076
3 x1 0 782000 x2 0.690000 sum 5.198000
x1 1.342291 x2 1. 076486 Max 8.598619
18.4 Genetic Algorithm Implementation 587

THE SOLUTION OF THE FOLLOWING PROBLEM IS

SUM' MAXIMIZE F(xl, x2) = 4xl + 3x2
2 *xi+ 3 * x2< = 6
-3>tc.xl+2*x2<=3
2*xl +x2<=4
xl' 1.342291 x2' 1.076486 Max' 8.598619

118.4.2 To Minimize a Function F(x) = x2

To write a program to minimize F(x) = x 2 using genetic algorithm.
Steps involved

I Step 1: Generate the random number as-.----- --,

Step 2: Inicialize i,j w nand m respectively.
Step 3: Max +-- 1000, x{1] +-- 0, sum +-- 0, m_max ~ 1000
Step 4, Compute x[i] ~ x[i]+(pp[i][j]'pow(2,p-1-j)) and

foll = xll * xll

sum = sum +fo[ rl

Step j, If (max>"fx[i])
Step 6: max+-- h[i];
Step 7: unril m_max>max
[ Step 8: Compme minimum value [

Program
#include<stdic.h>
#include<iostream h>
#include<conio.h>
#include<stdlib.h>
#include<math.h>
#include<time.h>

int pop[lO] [10) ,npop[10) [10] ,tpop[lO] [10] ,x[lO] ,fx[10) ,m_max=961,
ico=O,ico1,it=O;
void iter{int [10] [10) ,int,int);
int u_rand{int);
void tour_sel(int,int);
void cross_ov(int,int);
void mutat ( int, int) ;
void main()
[
int k,m,j,i,p[l0],n=O,a[10],nit;
clrscr();
588 Soft Computing Techniques Using C and C++

randomize ( l ;
cout<<"\t\tENTER THE NUMBER OF POPULATION IN EACH ITERATION
cin>>n;
cout<<"\n\t\tENTER THE NUMBER OF ITERATION : "
cin>>nit;
m=S;
for(i=O;i<n;i++)
[
for(j=m-l;j>=O;j--)
[
pop[i] (j]=u_rand(2l;

cout<<"\niTERATION "<<it<<" IS :\n";

iter(pop,n,m);
it++;
getch();
do
[
it++;
cout<< "\niTERATION "<<it<<~ IS : \n";
tour_sel (n, m);
iter(pop,n,m);
getch();
}while (it<nit);
cout<<"\n\nAFTER THE "<<icol<<" ITERATION, THE MINIMUM VALUE IS
"(int) sqrt(m_max);
getch();
)
void iter(int ppLlO) [lO],int o, int p)

int i,j,sum,avg,max=961;
for{i=O;i<o;i++J

x[i] =:0;
for(j=O;j<p;j++)
[
I
x[i)=x[i] + (pp[i] [j) "pow(2,p-l-j) l; I
)
fx[i]=x[i]*x[i];
sum:oosum+ fx [ i] ;
if (max>=fx[i] l
max=fx[i);

avg=surn/o;
cout "\n\nS .NO. \tPOPULATION\tX\tF (X) \n\n";
Eor(i=O;i<o;i++)
[
cout<<ico<<"\~";
ico++;

l
 ~
';~
589
18.4 Genetic Algorithm Implementation l
for(j=O;j<p;j++J I
coutpp[i) (j];
cout<<" \t\t" <<X [ i] <<" \t ~ <<fx [ i] << n\n ";
!
)
l
"<<max<<~\n";
cout<<n\n\t SUM "<<sum<<\tAVERAGE ~<<avg<< "\tMINIMUM
\
if (m_max>max)
[
m_max=lllax;
icol=it;
)

int u_rand(int x)

int y;
y=rand()%x;
return(y);
)
void tour_sel(int np,int mb)

int i,j,k,l,co=O,cc;
do
[
k=u_rand(np);
do
[
cc=O;
l=u_rand(np);
i f (k==l)
cc++;
}while(cc!=O);
if (fx[kl>fx[l])
[
for(j=O;j<mb;j++)
npop[co} [j)=pop[k) [j);
)
else if (fx[k)<fx[l])
[
for(j=O;j<mb;j++)
npop[co] [j]=pop[l] [j];

co++;
}while (co<np);
getch();
cross_ov(np,mb);
getch();
)
void cross_ov(int npl,int mbl)

int i,j,k,l,co,temp;
i=O;
590
Soft Computing Techniques Using C and Ct+

do
{
k=rand () %2;
do
{
co=O;
l=u_rand (mbl) ;
i f ( ( {k==O) &&
!1==011 II 1lk==11 && 11==mb1111
co++;
}while(co!=O);

i f ((k==OJ && (l!=OJ)

{
for(j=O;j<l;j++)
(
temp=npop [ i J [ j 1,
npop(i) [j]=npop[i+l] [j];
npop[i+l] [j]=temp;

else if ((k==l) && (l!=mbl))

for(j=l;j<mbl;j++)
(
temp=npop[i] [j];
npop[i) [j]=npop(i+l] [j];
npop[i+l] [j]=temp;

i=i+2;
}while(::<npl);

for (i=O; i<npl; i++)

(
for(j=O;j<mbl;j++)
(
tpop [ i] [ j] =npop [ i] [j] ;
}
}
mutat (npl,mbl);
}
void mutat(int np2,int mb2)
(
int i,j,r,temp,k,z;
i=O;
do
(
for(k=O;k<np2;k++)
(
r=O;
18.4 Genetic Algorithm Implementation 591

if (i!=k)
I
for(j=O;j<rnb2;j++)
(
if (tpop[i) [j]==tpop[k] [j])
r++;

if (r!=mb2-l)
(
z=u_rand (mb2) ;
if (tpop[i] [zl==OJ
tpop[i] [z]=l;
else
tpop[i] [z]=O;
if (npop[k] [u_rand(mb2)]==0)
npop[k] [u_rand(mb2)]=1;
else
npop[k] [u_rand(mb2)]=0;
mutat(k,mb2);

i++;
}while(i<np2);
for(i=O;i<np2;i++)
(
for(j=O;j<mb2;j++)
(
pop[i] [j)=tpop[i] [j];
}
}
}

Output
ENTER THE NUMBER OF POPULATION IN EACH ITERATION: 5
ENTER THE NUMBER OF ITERATION: 8
ITERATION 1 IS
S.NO. POPULATION X F(X}
0 00001 1 1
1 11011 27 729
2 llOll 27 729
3 01001 9 81
4 00111 7 49
SUM : 1589 AVERAGE 317 MINIMUM 1
ITERATION 2 IS :
S.NO. POPULATION X F(X)
5 10101 21 441
6 11011 27 729
7 01011 11 121
 ~'

592
Soft Computing Techniques Using C and Ct+

8 10010 18 324
9 11110 30 900
SUM : 2517 AvERAGE : 503 MINIMUM : 121
ITERATION 3 IS :
S.NO. POPULATION X FIX)
10 10010 18 324
11 10110 22 484
12 11011 27 729
13 01111 15 225
14 01011 11 121
SUM 1886 AVERAGE 377 MINIMUM 121
ITERATION 4 IS :
S.NO. POPULATION X FIX)
15 10111 23 529
16 10101 21 441
17 11001 25 625
18 10101 21 441
19 10100 20 400
SUM 2440 AVERAGE 488 MINIMUM : 4 0 0
ITERATION 5 IS :
S.NO. POPULATION X FIX}
20 10110 22 484
21 11111 31 961
22 01111 15 225
23 10111 23 529
24 10110 22 484
SUM 2685 AVERAGE 537 MINIMUM : 225
ITERATION 6 IS
S.NO. POPULATION X FIX)
25 01001 9 81
26 01111 15 225
27 01011 11 121
28 01111 15 225
29 11100 28 784
SUM 1436 AVERAGE 287 MINIMUM : 81
ITERATION 7 IS :
S.NO. POPULATION X FIX}
30 00000 0 0
31 00100 4 16
32 01111 15 225
33 11101 29 841
34 11010 26 676
SUM 1760 AVERAGE 352 MINIMUM 0
ITERATION .8 IS
S.NO. POPULATION X FIX)
35 10000 16 256
36 00000 0 0
37 01011 11 121

1
 1

18.4 Genetic Algorithm Implementation 593

i
38 01010 10 100 I
39 00001 1 1 !
SUM : 481 AVERAGE 96 MINIMUM .Q

AFTER THE 7 ITERATION, THE MINIMUM'V~UE IS 0

I 18.4.3 Traveling Salesman Problem (TSP)

In TSP, salesman travels n cities and returns to the starting cicy with the minimal cost; he is not allowed to cross
the city more than once. In this problem we are raking the assumption rhat all then cities are interconnected.
The cost indicates ilie distance between two cities.To solve iliis problem we make use of GA because the
cities are randomly selected. AJso the initial population for this problem is randomly selected cities. Fitness
function is nothing but the minimum cost. Initially the fitness function is set ro rhe maximum value and for
each travel, the cost is calculated and compared with the fitness function. The new fitness value is assigned ro
the minimum cost. Initial populacion is randomly chosen and taken as the parent. For ilie next generation,
the cyclic crossover is applied over the parent.

Cyclic Crossover
Let P l and P2 are two parents

PI : 2 B 0 I 3 4 5 7 9 6
P2:10546B9723
Select the first city Pl make it as ilie first city of offspringl(Ol)

01: 2--

To find the next ciry of offspring 01 search current city, which is selected from Pl in P2. Find the location
of city in P2 and select the ciry which is in the same location in PI. 01: 2 - - - - - - - 9 -

Continue the same procedure, we will get 01 as

01 : 2 B 0 I - 4 5 9

In the next step we will get the city 2 which is already present in 01 and then srop the procedure. Copy
the cities from parent P2 in the corresponding lomtions

01:2801645793

For the generation offspring 02 the initial selection is from the parent P2, and repeat the procedure with Pl

02: I 5 4 3 B 9 7 l 6
If ilie initial population contain N parents it will generate N(N- l)/2 offsprings. The next generation
the offsprings are considered as parent. The procedure is continued for N number of generation to find the
minimum cost.

Source Code
#include<stdio.h>
#include<conio.h>
int tsp[lO] [10)={ {999,10 13 21 5,6, 7, 215, 4},
I

{20, 999,3 0 5,10, 218,1115, 6} 0

594 Soft Computing Techniques Using C and C++ -::;;.,,
='-
q;:
{10,5,999,7,8,3,11,12,3,2}, ;i;
{3,4,5,999,6,4,10,6,1,8},
{1,2,3,4,999,5,10,20,11,2}, .,
{8,5,3,10,2,999,6,9,20,1},
{3,8,5,2,20,21,999,3,5,6},
{5,2,1,25,15,10,6,999,8,1},
{10,11,6,8,3,4,2,15,999,1),
{5,10,6,4,15,1,3,5,2,999}
) ;
int pa[l000][10]= {{0,1,2,3,4,5,6,7,8,9),
{9,8,6,3,2,1,0,4,5,7},
{2,3,5,0,1,4,9,8,6,7),
{4,8,9,0,1,3,2,5,6,7)
};
int i,j,k,l,m,y,loc,flag,row,col,it,x=3,y=3;
int count,row=O,res[l] [lO],rowl,coll,z;
int numof=4;
int offspring[lOOO] [10];
int mincost=9999,mc;
main()
{
int gen;
clrscr {);
print ("Number of Generation "};
scanf("%d",&gen);
offcall (pa);
offcal2 (pa);
print (" \n\t\t First Generation\n");
for{i=O;i<count;i++)
{
for(j=O;j<lO;j++)
printf{"%d ",offspring(i] [j]);
printf{"\n");

for(y=l;y<=gen-l;y++)
{
getch();
clrscr();
for(i=O;i<count;i++)
for(j=O;j<10;j++)
pa[i] [j]=offspring[i] [j];
numoff=count;
offcall(pa);
offcal2 (pa);
printf(n \n\t\t %d Generation\n",y+l);
for(i=O;i<count;i++)
{
for(j=O;j<lO;j++)
printf(n%d ",offspring[i][j]);
printf ( "\n" J ;
 595
18.4 Genetic Algorithm Implementation
-

getch();
clrscr ();

printf("\n\nMinimurn Cost Path\n");

for(z=O;z<10;z++)
printf("%d ",res[O][z));
printf { "\nMinimurn Cost %d \n" ,mincost);

/* finding the offspring using cyclic crossover */

off call (pa)
int pa[1000] [10];
{
count=O;
for(i=O;i<1000;i++}
for(j=O;j<lO;j++)
offspring[i] [j]=-1;

for(k=O;k<nurnoff;k++}
{
for {l=k+l; l<nurnoff; 1 ++)
{
offspring[row][O]=pa[k][O];
loc=pa[l] [0];
flag=l;
while(flag != 0)
{
for(j=O;j<lO;j++)
{
if(pa[k][j] ==lac
{
if (offspring[row] [j]==-1)
{
offspring [row) [j] =lac;
loc=pa[l) [j);

else
'''
I
flag=O;

}/* end while*/

for(m=O;m<lO;m++l
{
if(offspring[row) [m] == -1)
offspring[row][m]=pa[l][m];

for(z=O;z<lO;z++)
{
if(z<9)
{
 ~t
596
Soft Computing Techniques Using C and C++

rowl=offspring[row][z];
coll=offspring[raw)[z+l];
mc=mc+tsp[rowl] [call];

else

rowl=offspring[raw] [z];
call=offspring[row) [0];
mc=mc+tsp[rawl] [call];

if(mc < mincost)

{
for{z=O;z<lO;z++J
res[O] [z]=offspring[row] [z];
mincost=mc;

count++;
row++;
}/" end 1*/

ofcal2 {pa)
int pa[lOOO] [10};
{
for(k=O;k<numaff;k++)
{
for(l=k+l;l<numoff;l++)
{
offspring(row] [O]=pa(l]{O];
loc=pa(k] (OJ;
flag=l;
while(flag != 0)
{
for(j=O;j<lO;j++)
{
if(pa[l} [j] == lac
{
if (offspring(row) [j]==-1)
{
offspring[row][j]=loc;
loc=pa[k] [j];

else
flag=O;

)/* end while*/

for(m=O;m<lO;m++)
{

I
'
_..l
18.4 Genetic Algorithm tmplementa\ion 597

if(offspring[row][m] == -1)
offspring.[row] [m)=pa[k) [m];

for(z=O:z<lO;z++)

if(z<9)
{
rowl=offspring[row] [Z];
coll=offspring[row][Z+l];
rnc=mc+tsp(rowl] [call];

else

rowl=offspring[row] [z];
coll=offspring[row] [0];
mc=mc+tsp[rowl] [call];

row++;
if(mc < mincast)

for(z=O;z<lO;z++J
res[O] [z)=affspring(row] lz);
mincost=mc;

count++;
)/* end 1*/

Output
Humber of Generation 2
First Generation
0 8 2 3 4 1 6 7 5 9
0 1 2 3 4 5 9 8 6 7
0 1 2 3 4 5 6 7 8 9
9 8 5 3 2 1 0 4 6 7
9 8 6 0 1 3 2 4 5 7
2 3 5 0 1 4 9 8 6 7
9 1 6 3 2 5 0 4 8 7
2 3 5 0 1 4 6 7 8 9
4 8 9 0 1 3 2 5 6 7
2 3 6 0 1 4 9 8 5 7
4 8 9 3 2 1 0 5 6 7
4 8 9 0 1 3 2 5 6 7

2 Generation
0 1 2 3 4 5 9 8 6 7
0 1 2 3 4 5 6 7 8 9
0 8 2 3 4 1 6 7 5 9
598 Soft Computing Techniques Using C and C++

0 8 6 3 4 1 2 7 5 9
0 B 2 3 1 4 6 7 5 9
0 1 2 3 4 5 6 7 8 9
0 B 2 3 1 4 6 7 5 9
0 B 9 3 4 1 2 5 6 7
0 8 2 3 1 4 6 7 5 9
0 8 2 3 4 1 6 7 5 9
0 8 9 3 4 1 2 5 6 7
0 1 2 3 4 5 6 7 8 9
Minimum Cost Path
0 8 2 3 4 1 6 7 5 9
Minimum Cost 53

I 18.4.4 Prisoner's Dilemma

Cooperation is usually analyzed in game theory by means of a nonzero-sum game called the "Prisoner's
Dilemma." The two players in the game can choose between two moves, either "cooperate" or "defect."
The idea is clm each player gains when both coopeme, but if only one of them cooperates, the other one,
who defects, will gain more. Ifborh defect, both lose (or gain very little) but nor as much as the "cheated"
cooperaror whose cooperation is nor returned. The whole game situation and its different outcomes can be
summarized by the following table where hyporherical "poims" are given as an example ofhow rhe differences
in result might be q!Janrified.

Action of N Action B Cooperate Defect

Cooperate Fairly good [+5] Bad [-10]
Defect Good [+10] Mediocre [0]

The type of crossover chat is performed is a "single point crossover" where the point of crossover is randomly
selected. The mutation is expected to happen every 2000 generation. It is easy to change the mutation as it
is implemented as a separate function.
Source Code
#include<stdlib h>
#include<stdio.h>
#include<conio.h>

int calculate(int*);
int* select(int *);
void crossover(int*,intk):
void sort_select{void);
//THESE ARE SOME GLOBAL VARIABLE USED
int best_score[20];
int score[9];
int index[6];

void main ()
I
int a[lO] [70],select_string[5] [70];
int best_string[20] [70],max,ind=O;
 599
18.4 Genetic Algorithm Implementation

int p,counter=l;
int i,n,j,ternp[lO);
randomize();
clrscr ():
for(j=O;j<lO;j++)
for(i=O;i<70;i++)
a[j] [i]=random{2);
//THE NUMBER OF GENERATION TO BE SCANED IN
printf(" Enter the no of generation");
scanf("%d",&n);
for(i=O;i<lO;i++)
score[i)=calculate(&a[i][O]);
//function for sorting the score array and finding the index of
best score
sort_select();
for(i=O;i<7;i++)

p=index[i]; //THE ORDER OF BEST SCORE STORED IN INDEX.

for{j=O;j<70;j+~l
select_string[il [j]=a[p] [j]:
)
best_score[O]=score[O);
for(i=O;i<70;i~+)
best_string [ 0) ( i) =select_s tring [ 0] [ i l ;

while(counter < n)

for(i=O;i<7;i=i~2)
crossover(&a[i] [0] ,&a[i+l] [0]);
for(i=O;i<9;i++)
score[i)=O;
for(i=O;i<7;i++l
score[i]=calculate(&a[i) [0]);
//CALCULATE FUNCTION RETURNS SCORE OF EACH STRING
sort_select ();
best_score[counter]=score[O];
p=index[O);
for(j=O;j<70;j++)
best_string[counter] [j]=a(p] (j];
counter++;
)
//OUTPUT THE BEST SCORES.
for(p=O;p<n;p++)
I
printf ("The best score in the generation %d :~,p+l);
printf (" %d \n" , best_score [p]);

//OUTPUT THE BEST STRINGS.

for(i=O;i<n;i++)

~
600 Soft Computing Techniques Using C and C++ 18.4

printf(\n\nTHE BEST STRNG IN GENERATION %d :\n\n",i+l);

for ( j =0; j <70; j+_+)
{
if(j%2==0&&j!=O)
print(" );
if(best_string(i] [j] ==1)
print ( d);
//COVERTING l'S AND O'S TO d AND c
else
printf("c");

//CALCULATING THE BEST OF THE BEST

for(i=O;i<n;i++)
temp[i)=best_score[i];
max=temp[O];
for(i=l;i<n;i++l
{
if(max<temp[i])
{
max=temp[i];
ind=i;

//CALCULATING THE BEST FROM THE SELECTED.

printf("\n\n");
printf("\nTHE BEST STRING IN ALL GENERATION IS \n\n");
for(i=O;i<70;i++)
{
if(i%2==0&&i!=Ol
printf(" ");
if(best_string[ind] [i]==ll
print ( "d");
else
printf ( "c");

print ( "\n\nTHE CORRESPONDING BEST SCORE IS: %d ",best_score [ind]);

getch();
)
int
{
calculate(int~ ptr)
1
int "a;
!
int pl,p2,i;
a=ptr;
pl=O; p2=0;
for(i=O;i<70;i=i+2) //calculating the values according to truth
table.

if{a[i]==l && a[i+l]==ll

[

........
 601
4 Genetic Algorithm Implementation

pl=p1+3; p2=p2+3;

if(a[i]==l && a[i+ll==O)

{
pl=pl+S; p2=p2+0;

if{a[i]==O && a[i+l]==l)

{
pl=pl+O; p2=p2+5;
)
if{a[i]==O && a[i+l)==O) ~
{
pl=pl+l; p2=p2+1; ~
"~
return(pl+p2); //RETRUN THE TOTAL SCORE OF THE STRING. "~
)
void sort_select{) //ORDINARY SORTING PROCEDURE
{
int ternp(9),i,j,t;
for{i=O;i<lO;i++)
temp[i]=score(i];
~i
;I
for(i=O;i<lO;i++)
for(j=9;j>=i;j--) '
~ ;
{
if(temp[i)<ternp[j)) //USUSAL SWAPPING PROCEDURE- . '

{ ( '
r'
t=temp[j];
ternp[j]=ternp[i];
temp [ il =t;

for(i=O;i<7;i++)
for(j=O;j<lO;j++)
if(temp[i]==score[j])
index[i]=j;
score[O]=temp[O];

void crossover(int *ptrl,int *ptr2)

int temp,i,j;
int ind=random(60); //RANDOM POINT OF CROSSOVER
for(i=ind;i<70;i++l
{
temp=ptrl[i);
ptrl[i]=ptr2[i];
ptr2[i]=temp;
602 Soft Computing Techniques Using C and C++

Output
Enter the no of generation 5
The best score in the generation 1: 171
The best score in the generation 2: 160
The best score in the generation 3: 170
The best score in the generation 4: 166
The best score in the generation 5: 169
The best string in generation 1:
dd de cd de de cd cd dd cc de de de dd de dd cd de cd cd cc de cd cc cd dd cd
cd dd cd de cc cd de dd dd
The best string in generation 2:
cd cc cd cc cd cd dd de cd cc de cc dd cd dd dd cc cc de dd de cd cd dd de dd
dd cc cd dd de de cd de cc
The best string in generation 3:
cd cc cd cc cd cd de dd cd de dd cc cd cd cc dd cd dd de cd de de dd cd de de
de cd cd cd de de dd de dd
The best string in generation 4:
cd cc cd cc cd cd de dd cd de dd cc cd cd cc dd cd dd de cd de de dd dd de dd
dd cc cd dd de de cd de ee
The best string in generation 5:
Cd dd ee ed dd de ed ee dd ed dd dd de ed ed ee de ed ed de ee dd dd de de de
dd de de cd de cc de ed dd
The best string in all generation is
dd de cd de deeded dd cc de de de-dd de dd cd de cd cd cc de cd cc cd dd cd
cd dd ed de ce cd de dd dd
The corresponding best score is 171

I 18.4.5 Quadratic Equation Solving

To find the rOQ(S of the quadratic equ:uion using generic algorithm. To solve the above problem for the
* * +
quadratic equation x x + 5 x 6 using following procedure. It could be used for solving any quadratic
equation by changing fitness function /(x) and changing length of chromosome.

Steps involved

Step 1: Initial population size is I 0 and chromosome length is set to 5. Selecting initial population, i.e.
random approximate solurion to the problem, which are 10 different 5-bit binary strings. Here
initial population consists of 10 chromosomes. Chromosomes are generated by using rnndom
number generator.
Step 2: Converting the chromosome's genotypes to its phenotype (i.e. binary string into decimal value).
*
ln the binary string the most significant bit is sign bit. Its weight is -2 (n- l) and other bits
*
are magnitude bits their weights are 2 (n - 1).
+ *
Step 3: Evaluate the objective funcrion/(x) = x* x 5 x+ 6. For each chromosome:
Convert the value of the objective function into fitness. Here for this problem fimess is simply
equal to the value of rhe objective function.
Ifj(x) ===- 0 .for a particular chromosome, that chromosome is required accurate solution. Now
display the value of chromosome and stop. Otherwise perform next generation by continuing
fol!owing steps.
 603
18.4 Genetic Algorithm Implementation

Step 4, Implementation of selection operation. For this problem the tournamem selection is adopted.
The tournament selection is implemented as follows: Take any two chromosomes randomly and
select one with min. Fimess for next gen~rarion. This process has to be repeated till we get 10
chromosomes.
Step 5: Implementation of crossover operation on newpopularion. Take chromosome 1 and 2 randomly
fix the cut-point position and randomlY decide left or right crossover and interchange the bits
and the resulting chromosomes are used in the next generation. Repeat fie above process for
chromosome pair (3,4), pair (5,6), pair (7,8) and pair (9,10). This crossover operation generates
10 new chromosomes for rhe next generation.
Step 6: Jump to Step 2 (i.e. perform next generation).

Source Code
#include <stdio.h>
~include <conio.h>
~include <dos.h>
#include <math.h>
#include <stdlib.h>
#include <tirne.h>
int f (intl;

void main ()

struct c{
int chrornosorne[S];
int decirnal_val;
int fittness;
) '
struct e ipop[lO], newpop[lO];
int i,j,cut,gen,t,flag,num,sl,s2;
clrser ();
I* generating Initial population */
candornize();
:or(i=O;i<lO; ++il
for(j=O; j<S; ++j)
ipop[i] .chromosorne(j] = rand()%2;

I* start of the next generation */

gen=l;
while(l)
I
!* Converting a binary string into decimal value ;
for(i=O;i<lO; ++i)
I
nwn=O;
for(j=O;j<4;++jl
nurn = num+ (ipop[i].chromosorne(j] * pow(2,j));
num =
nwn-(ipop(i) .chromosorne(4]*pow(2,4));
ipop[i].decimal_val = num;

'1,
rl.
604
Soft Computing TechniqUes Using C and C++

/* Calculating fittness value */

for{i=O;i<lO;+*i)
ipop[i].fittness = f{ipop[i].decimal_val);
printf("Generation- %1d\n", gen);
printf("Initial population- output\n");
for(i=O;i<lO;++i)
{
for(j=4; j>=O; --j)
printf("%ld", ipop[i] .chromosome[j]);
print(" %d", ipop[i] .decimal_val);
print( %d", ipop[i].fittness);
printf("\n");

forti=O;i<lO; ++i)
{
if(ipop[i] .fittness ==OJ
{
printf(~stop generations\n");
printf("result = %d\n", ipop[i].decimal_val);
goto ll;

!* tournament selection */
printf("tournament selection\n ");
i=O;
while(i<=9)
{
sl = rand()%10;
s2 = rand()%10;
printf("%d %d %d %d\n", sl,s2,ipop(sl).fittness, ipop[s2].
fittness);
getche{);
if( ipop[sl] .fittness < ipop[s2].fittness)
{
for(j=O;j<S;++j)
neWpop [ i] . chromosO_me [ j] ipop[sl] .chromosome(j];
else

for (j=O; j<S; ++j J

newpop[i] .chromosome[j] ipop[s2].chromosome[j];
i++;

getche ();
print ("new population -output\n");
for(i=O;i<lO;++i)
{
for(j=4; j>=O; --j)
18.4 Genetic Algorithm Implementation 605

printf("%1d", newpop[i] .chromosome(j]);

print ( "\n");

getche();

/*crossover operation */
print ("crossover operation\n");
printf("left/right cut-point position\n"l;
for(i=O;i<=4;++i)

flag= rand()%2;
cut= rand()%5;
printf("%1d %1d\n", flag, cut);
if(flag==O) /* crossover to left of cutpoint position*/
for(j=O:j<=cut-l;++j)
{
t=newpop[2*i] .chromosome[j];
newpop[2*i).chromosome[j]= newpop[(2*i+l)].chromosome(j];
newpop[(2*i+l)] .chromosome[j]= t;

else /* crossover to the right of cutpoint position*/

for(j=cut+l;j<=4;++j)
{
t=newpop[2*i] .chromosome(j];
newpop[2*i] .chromosome[j]= newpop[(2*i+l)].chromosome[j];
newpop[(2*i+l)].chromosome[j]= t;

for(j=4; j>=O; --jl

print ( "%ld", newpop [2*i] . chromosome [j] J;
print ( "\n" l;
for(j=4; j>=O; --j)
print ( "%1d", newpop [2*i+l] . chromosome (j ]) ;
print ( "\n");

/* copy nev~opulation to initial population*/

for(i=O: i<lO; ++i)
{
for(j=O; j<S;++j)
ipop[i] .chromosome[j] newpop[i] .chromosome{j];

gen=gen+l;

11:
print ( "end\n");

int f(int x)

return ( x*x + S*x + 6);

606 Soft Computing Techniques Using C and C++

Output

At the end of fifth generation, the output is

Generation- 5
Chromosome decimalvalue Fittnessvalue
00001 1 12
11100 -4 2
11100 -4 2
00000 0 6
00001 1 12
00001 1 12
10001 -15 156
01100 12 210
11101 -3 0
00000 0 6
stop generations
result = -3

118.5 Summary

Thus in this chapter the imp\emenration ofsoft computing concept using CIC++ has been dealt. The concepts
of neural networks, fuzzy logic and genetic algorithm discussed in various chapters have been implemented
here. C being a universal language helps in evolving the soft computing techniques and since it is portable,
soft computing programs written inC for one compurer can be run on anmherwith little or no modification.
With the availabiliry of large number of functions, the programming task becomes simple. C++, an evolution
of C, has helped soft compuring ro run in an object oriemed programming environmem.

118.6 Exercise Problems

1. Implemem the AND function using perceptron 9. Maximize Rosenbrock's function using a C++
network using a C program. program.
2. Write a C++ program to apply back propagation 10. Minimize Rasrrigin's function using strucrure-
network for a pattern recognition problem. oriented programming language.
3. Implement OR function with bipolar inputs and 11. Given a polynomial equation of the form f(x) ::::::
targets with a MAD ALINE neural net. 4x4 + 3~ + b?- +x + 7. Find rhe roots of this
4. Write a program to create an ART 1 network polynomial using GA approach.
to dusrer seven input units and three cluster 12. Consider a hyperbolic tangent function. Max-
units. imize it within the range 0 < x < 22/7 using
5. Develop a Kohonen self-organizing feature map a C program. Apply two-point crossover and
for a image recognition problem using a C tournament selection process.
program. 13. Find the roars of ilie quadratic equation using
6. Write a program w implement various opera- GA. The quadratic equation is f (x) = 6:l- +
tions of fuzzy sers. 5x+3.
7, Implement the properties of fuzzy sets using a 14. Find the solution of the function j(x) =
C++ program. sin(7rr x) + 10 with the constraint -3 < x< 3
8. Develop a C program to perform compositional by using genetic algorithm.
operations in fuzzy relations. 15. Write a program to minimize "cosine" function.
 MATLAB Environment for
Soft Computing Techniques 19
Learning Objectives - - - - - - - - - - - - - - - - - - ,
Discusses how soft computing techniques are logic toolbox, Genetic algorithm toolbox- are
implemented using MATLAB software. included with commands for ready reference
Gives brief note on the development of of the user.
MATLAB software. The chapter apart from the command line
Derails how basic operations are carried our functions also discusses rhe implementation of
using MATLAB. soft computing techniques using SIMULINK
blocks and graphical user interface (GUI)
An introduction SIMULINK which is a toolbox.
branch in MATLAB package is discussed.
The chapter provides the reader several prob-
The various soft computing toolboxes in lems solved using MATLAB software for soft
MATLAB - Neur.tl necwork toolbox, Fuzzy computing techniques.

119.1 Introduction
MATLAB (Matrix Laboratory), a product of Mathworks, is a scientific sofrware package designed to provide
integrated nurperic computation and graphics visualization in high-level programming language. Cleve Moler,
Chief Scientist ac Math Works, Inc., originally wrote MATLAB to provide easy access to matrix sofrware
developed in the UNPACK and EISPACK projects. The very first version was written in the late l970s for
use in courses in matrix theory, linear algebra and numerical analysis. MATLAB is therefore built upon a
foundation of sophisticated matrix software, in which the basic data dement is a matrix char does not require
predimensioning.
MATLAB program consim of standard and specialized toolboxes allowing users to take advantage of the
matrix-algorithm-based projects. MATLAB offers inreractive features allowing the users a great flexibilicy in
the manipulation of data and in the form of matrix arrays for computation and visualization. lviATLAB
inputs can be entered at the "command line" or from "mfiles," which contains a programming-like sec of
instructions to be executed by MATLAB. In the aspect of programming, tviATLAB works differently from
FOTRAN, C, or Basic; for example, no dimensioning required for matrix arrays and no object code file
generated. MATLAB offers some standard toolboxes and many optional (at extra charges) toolboxes such as
Financial Toolbox and Statistics Toolbox. Users may create their own toolboxes consisting of"mfiles" written
for specific applications. MATLAB is a high-performance language for technical computing. It integrates
computation, visualization and programming in an easy-to-use environment where problems and solutions
are expressed in familiar mathematical notation. Typical use includes:

IL
,&
1. math and compmation;
2. algorithm development;
""
 ;(;"

608 MATLAB Environment for Soft Computing Techniques

3. madding, simula~on and prototyping;

4. data analysis, exploration and vis_ualizacion;
5. scientific and engineering graphics;
6. application development, including graphical-user interface building.
MATLAB's rwo- and three-dimensional graphics are object-oriented. MATLAB is thus both an environ-
ment and a mauix/vecror-oriented programming language, which enables the use to build own required tools.
The main fearures ofMATI..AB are:
1. Advance algorithms for higlf.performance numerical computations, especially in the field of matrix algebra.
2. A large collecrion of predefined mathematical functions and the ability to define one's own functions.
3. Two- and three-dimensional graphics for piercing and displaying data.
4. A complete help system online.
5. Powerful matrix/veccor-oriemed high-level programming language for individual applications.
6. Abilicy to cooperare with programs written in ocher languages and for importing and exporting formatted
data.
7. Toolboxes available for solving advanced problems in several application areas.

119.2 Getting Started with MATLAB

Clicking on the program icon on a Windows/Mac machine can starr MATLAB. The command window can
be used to interactively issue commands and evaluate expressions. The extensive help files are essenrial for
obtaining information abour the various commands and functions available. Typing help at the command
prompt generates a list of the various function categories and toolboxes with a briefdescription of each. Typing
help topic generates help on rhe specified topic, which is generally a MATLAB command or toolbox name.
This can be used m get rhe syntax for different commands. A more user-friendly graphical help system is also
available via the help menu.
MKfLAB uses conventional notation for real numbers. Sciemific notation is also accepted in the form of
a real number followed by the letter e and an integer exponent. Imaginary numbers are obtained by using
either i or j as a suffix. Examples of valid numbers: 5, 4.55, 1.945e-20, i, 10+ 15i.
The basic mathematical operators(+,-, * I,') can be used direcdy at the command prompt to perform
calculations, as can various elementary mathematical functions. help elfun gives a list of the elementary
mathematical functions. Remember that angles are specified in radians to functions like sin{) and cos{).
Examples of usage:

8 + 3t+5
am= 13.0000 + 3.0000<
sqn(-4)
ans = 0 + 1.4142~

I 19.2.1 Matrices and Vectors

Construction: The simplest way to construct a matrix in MATLAB is to enumerate irs elements row by row
within square brackets, the rows separated by semi-colons, the elements of each row separated by spaces.
I
Ib
~~
 609
19.2 Getting Started with MAn.AB

These dements can be real/complex numbers or other vectors and matrices, as long as the dimensions match
up. For example,

A= [1234; 5 67 8]
A=
1234
5678

Row vectors are simply matrices with one row of elements:

x=[4567]
x=
4567

Vectors with equally spaced elements can be constructed using the colon notation:

x = starting value: increment: maximum value

A default increment of 1 is used if the increment is omitted. If the vector has to have a specific last element,
we use the linspace command:

x = linspace(first_element, lnst_element, number_of_elements)

CoLumn vectors are constructed as matrices with several rows of one element each:

y=[l;2;3j
y=

2
3

Equally spaced column vectors are obtained by first generating a row vector using the colon notation or the
linspace command, and then applying the transpose operator. There are other built-in functions to generate
specific rypes of matrices:
I. eye{m, n) generates an m X n matrix with ones on the main diagonal and zeros elsewhere. If m = n, eye(n)
can be used instead.
2. zeros(m, n) is an m X n matrix whose elements are all zeros.
3. ones(m, n) is an m x n matrix whose elements are all ones.
4. diag(v) is a square diagonal matrix with vector von the main diagonal.
5. diag(A) is1a column vector formed from the main diagonal of A.

~
 610 MATLAB Environment for Soft Computing Techniques

Addressing elemmts. The elemenr in the ith row and jth column of a matrix A isA{i,;). The subscriprs i and
j cannot be negative or zero. In the case of a vector x, the irh elemem of the vector is addressed as x(l):

A= [I 2 3; 7 8 9]
A(2, 1) = 7
X= [4 56}
x(2) = 5

Matrix operations: Matrix addition, subtraction and multiplication are implemented using the traditional
operators +. - and * The inverse of a square matrix A is given by inv(A). There are two matrix division
operators,\ and/. IfA is a non~singular matrix, thenA\Band BIA correspond ro the left and right multiplication
of Bby inv(A). The transpose of a matrix A is given by A'. The expression A' produces the conjugate transpose
of A, that is it transposes A and replaces irs elemenrs by their complex conjugates. If theelemenrs of A are
real, ilien A' and A.' are equivalent.
ATTayoperations: It is also possible to work with the matrix as an array, that is to perform uniform elemenrwise
operations. The array operators(+, -, * .f, .A) are used forthis purpose. For example, if A and Bare of the
same dimensions, the expression A.* Bwould multiply each elemenr of A with the corresponding element of
B m produce a matrix of the same dimension as A and B.
'
'\ Scripts: Instead of executing individual commands at the prompt, it is possible to make a rexr file with a
\
sequence of commands, that is a tv!ATLAB script, and execute all ilie commands in sequence. The script must
be a plain ASCII file with a ".m"' extension. When this file is in the working directory, typing the name of
ilie file wiiliour the extension is sufficient to execute the script.
Plotting. The general form of the two-dimensional plot command is plot(x,y, S) where x andy are vectors
of the same type and dimension and Sis a string of characters wiiliin quotes which specifies plot attributes
like color, line sryle, ere. Use help plot ro find our options and related commands.
;1
119.3 Introduction to Simulink
Simulink (Simulation and Link) is an extension ofMATLAB by Mathworks.lt works with tv!ATLAB to offer
modeling, simulating and analyzing of dynamical systems under a graphical user interface (GUI) environ-
ment. The construction of a model is simplified with click-and-drag mouse operations. Simulink includes
a comprehensive block library of toolboxes for both linear and nonlinear analyses. Models are hierarchical,
which allow using both top-down and bonom-up approaches. As Simulink is an integral part of .MATLAB;
it is easy to switch back and forth during the analysis process, and rhus, the user may take full advantage
of features offered in both environments. MATLAB is an interactive package for numerical analysis, matrix
computation, control system design and linear system analysis and design available on most CAEN (Com-
puter Aided Engineering Network) platforms (Macintosh, PC, Sun and Hewlett-Packard). In addition to che
standard functions provided by lv1ATLAB, there exist large set of toolboxes, or colleaions of functions and
procedures, available as part of the MATLAB package. The toolboxes are:
1. ControL system: It provides several features for advanced control system design and analysis.
2. Communications: It provides functions to model the components of a communication system's physical
layer.
3. SignaL processing. It contains functions to design analog and digital filters and apply these filters to data
and analyze the results.
=T

19.3 Introduction to Si~~nk

611

4..- SyStem identification: It provides features to build mathematical models of dynamical systems based on
observed system data.
5. Robust controL: It allows users to create robust multivariable feedback control system designs based on the
concept of the singular-value Bode plot.
6. Simuiink: It allows you to model dynamic systems graphically.
7. Ntural network: It allows you to simulate neural networks.
8. Fuzzy Logic: It allows for manipulation of fuzzy systems and membership functions.
9. [mage processin~ It provides access to a wide variery of funaions for reading, writing, and filtering images
of various kinds in different ways.
10. Analysis: It includes a wide variery of system analysis tools for varying matrices.
11. Optimization: It contains basic cools for use in consrrained and unconstrained optimization problems.
12. SpLine: It can be used to find approximate functional representations of data se.ts.
13. Symbolic: It allows for symbolic (rather than purely numeric) manipulation of functions.
14. User interface utilities: Ir includes tools for creating dialog boxes, menu utilities and other user interaction
for script files.
In MATI.AB command window, enter: > > simulink and press ENTER ro invoke Simulink. A Simulink
library browser window would appear as shown in Figure 19-l.

I
I -t:Ontinuoui:- ~Ot.tnwus

I
l
Conttol Syttern lCICIIlol!:

I DSPBioct.set
Deveiopel'f Kilol T1 OSP
Oi.!ht.G~Bkot:bel
F'Md-Poill ~ltxbel
fUZI)I~IToollox
MPCB!cd4
Motclcile DSP Blotbel
NCDB~
NeuaNctwcKk Bloc:ktel
Powef Sj~Ur~ Blocksel.
Re4-Tillo \r(rdoM llllget
Rca-Tine\'>(~

l Figure 191 Simulink library browser.

 ~
"
.~;

612 MATlAB Environment for Soft Computing Techniques 1

lt9.4 MATLAB N_eural Network Toolbox n

The NIATLAB neural network roolbox: provides a complete set of functions and a GUI for the design, n
implememation, visualization and simulation of neural neMorks. Ir supports the most commonly used
I
supervised and unsupervised network architectureS and a comprehensive set oftraining and learning functions.
n
The neural network toolbox extends the MATLAB computing environment to provide tools for the design,
implementation, visualization and simulation of neural networks. Neural networks are uniquely powerful
tools rhar are used in applications where formal analysis would be difficult or impossible, such as pattern
recognition and nonlinear o/-)tem identification and control.
Salient Ftatures:
1. GUI for creating, training and simulation of neural networks.
2. Set of training and learning functions.
3. Automatic generation of Simulink models from neural nerwork objects.
4. Pre- and post-processing functions for improving network training and assessing nerwork performance.
5. Routines for improving generalization.
6. Visualization functions for viewing network performance.

I 19.4.1 Creating a Custom Neural Network

The command NETWORK creates a custom neural nerwork.

Synopsis:
net = nerwork
ner = nerwork(numlnpurs,numLayers,biasConnect,
inpurConnect, layerConnect,outputConnect,targetConnect)
Description:
NET\Xr'ORK creates new cuswm networks. It is used to create networks that are then customized by
functions such as NEWP, NEWLIN, NEWFF. etc.
NETWORK rakes rhe following optional arguments (shown with default values):
numlnputs: Number of inputs, 0.
numl..ayers: Number of layers, 0.
biasConnect: numl..ayers-by-1 Boolean vector, zeros.
inpurConnect: numl.ayers-by-numlnpurs Boolean matrix, zeros.
layerConnect: numLayers-by-numLayers Boolean matrix, zeros.
ourpurConnect: 1-by-numl..ayers Boolean vector, zeros.
targerConnecr: 1-by-numlayers Boolean vector, zeros.
and rerurns,
NET: New nerwork with the given property values.
Propenies
Architecture properties
ner.numlnputs: 0 or a positive imeger.

j
Number of inputs.
.

.. .
 ~I

19.4 MATLAB Neural Network Toolbox

613 il
;I
m

I
ner.numl..ayers: 0 or a positive imeger.
Number of layers.
net.biasConnecr: numLayer-by-1 Boolean vector.
If net.biasConnect{i) is 1 then the layer i has a bias and
netbiases{il is a structure describing that bias.
net.inputConnect: numl.ayer-by-numlnputs Boolean vector. ~~
([;
If net.inputConnect(i,j) is 1 then layer i has a weight coming from
inputj and net.inpurWeights{i,j} is a suucture describing that weight.
"
~~

nedayerConnecr: numLayer-by-numLayers Boolean vector. "'-,

If ner.layerConnect(i,j) is I then layer i has a weight coming from
layer j and ner.layerWeighrs{i,j) is a structure describing that weight.
net.outpmConnect: 1-by-numLayers Boolean vector.
If net.ourputConnect(i) is 1 then the nerwork has an output from layer i and net.outputs{i) is a structure
describing that output.
net.targetConnect: 1-by-numl..ayers Boolean vector.
If net.outputConnect(i) is 1 then the nern:ork has a target from layer i and net.targets{i} ts a structure
describing that target.
ner.numOurputs: 0 or a positive int!!ger. Read only.
Number of network outpurs according to net.outpurConnect.
net.numTargers: 0 or a positive inreger. Read only.
Number of targets according to ner.rargetConnecr.
net.numlnpmDelays: 0 or a positive integer. Read only.
Maximum input delay according to all
net.inputWeight{i,j ).delays.
ner.numLayerDelays: 0 or a positive number. Read only.
Maximum layer delay according wall Ner.layerWeight{i,j).delays.
Suhobject structure properties
net. inputs: numlnputs-by-1 cdl array.
nct.inputs[i) is a snucrure defining input i:
ner.layers: numl..ayers-by-1 cell array.
net.layers{i) is a structure defining layer i:
net. biases: numLayers-by-1 cell array.
ifner.biasConnecr(i) is l, then ner.biases{i) is a strucmre defining the bias for layer i.
net.inputWeights: numLa.yers-by-numlnputs cell array.
if net.inputConnect(i,j) is I, then ner.inputWeights{i,j} is a structure defining the weight to layer i
from input j.
net.layerWeighrs: num Layers-by-numl.ayers cell array.
if nedayerConnect(i,j) is l, rhen net.layerWeights{i,jJ is a structure defining the weight ro layer i
from layer j .
 614 MATI.AB Environment lor Soft Computing Techniques

ner.ourputs: lby-numl.ayers cell array.

if ner.outputConnecr(;) is 1, then net.outputs{i} is a srrucrure defining the network output from
layer i.
ner.rargets: 1-by-numLayers cell array.
if net.targerConnecr(i) is_ I, rhen net.targets{i} is a mucrure defining the network target to layer i.
Function properties:
net.adaptFcn: name of a network adaption function
net.initFcn: name of a network initialization function
net.performFcn: name of a network perfOrmance function
net.trainFcn: name of a nerwork training ~merion or
Parameter properties:
net.adaprParam: network adaption parameters.
net.initParam: network initialization parameters.
ner.performParam: ne[Work performance parameters.
net.uainParam: network training parameters.
~ Weight aud bias value properties:
ner.IW: numl.ayers-by-numlnpurs cell array of input weight values.
net.LW: numLayers-by-numLayers cell array of layer weight values.
net.b: numl.ayers-by-1 cell array of bias values.
Orhrr properties:
ner.userdaw.: structure you can use to store useful values.

19.4.2 Commands in Neural Network Toolbox

The various commands used in the neural network toolbox are as follows:
Graphical user inteJfoce fimctions:
nntool: Neural network toolbox graphical user interface.
Anarysis fimctions:
errsurf Error surface of single input neuron.
maxlinlr: Maximum learning me for a linear layer.
Distance fimctions:
boxdist: Box distance function.
disr: Euclidean distance weight function.
mandisr: Manhattan distance weight function.
linkdist: Link disrance function.
lAyer initiali:union Junctions:
initnw: Nguyen-Widrow layer initialization function.
inirwb: By-weight-and-bias layer initialization function.
~

19.4 MATLAB Neural Network Toolbox 615

Learningfonctions:
learncon: Conscience bias learning function.
learngd: Gradient descem weight/bias learning function.
learngdm: Gradient descent wlmomenrum weight/bias learning function.
learnh: Hebb weight learning function.
learnhd: Hebb with decay weight learning function.
learnis: lnstar weight learning function.
learnk: Kohonen weight learning function.
learnlvl: LVQl weight learning function.
learnlv2: LVQ2 weight learning function.
learnos: Outstar weight learning function.
learnp: Perceptron weight/bias learning function.
learnpn: Normalized perceptron weight/bias learning function.
learnsom: Self-organizing map weight learning function.
learnwh: Widrow-Hoffweightfbias learning rule.
Line search functions:
srchbac: Backtracking search.
srchbre: Brent's combination golden section/quadratic interpolation.
srchcha: Charalambous' cubic interpolation.
srchgol: Golden section search.
srchhyb: Hybrid bisection/cubic search.
New networks:
network: Creare a custom neural network.
newc: Create a competitive layer.
newcf: Create a cascade-forward backpropagation network.
newclm: Create an Elman backpropagation network.
newff: Create a feed-forward backpropagation network.
newfftd: Create a feed-forward input-delay backprop nenvork.
newgrnn: Design a generalized regression neural nenvork.
newhop: Create a Hopfield recurrent ncrwork.
newlin: Create a linear layer.
newlind: Design a linear layer.
newlvq: Create a learning vector quantization network.
newp: Create a perceptron.
newpnn: Design a probabilistic neural network.
newrb: Design a radial basis nerwork.
newrbe: Design an exact radial basis network.
newsom: Create a self-organizing map.
Net input fonctions:
netprod: Product net input function.
netsum: Sum net input function.
Net input derivative fimctions:
dnetprod: Product net input derivative function.
dnersum: Sum net input derivative function.

'
616 MATLAB Environment for Soft Computing Techniques
~ 19

Network initialization fonction.s:

initlay: layer-by-layer 'network initiaJiz.arion function.
Pafimnance fonctiom:
mae: Mean absolute error performance function.
mse: Mean squared error performance function.
msereg: Mean squared error with regularization performance function,
sse: Sum squared error performance function.
Pnj'omutnce derivative JUnctions:
dmae: Mean absolute error performance derivatives function.
dmse: Mean squared error performance derivatives function.
dmsereg: Mean squared error w/reg performance derivative function. T
dsse: Sum squared error performance derivative function.
Plottingfimctiom:
himonw: Hinton graph of weight matrix.
himonwb: Hinton graph of weight matrix and bias vector.
plotbr: Plot network performance for Bayesian regularization training.
places: Plot an error surface of a single inpm neuron.
plotpc: Plot classification line on perceprron vector plot.
plorpv: Plot perceprron inpur/rarget vectors.
plorep: Plot a weight-bias posicion on an error surface.
plotperf Plot ne('Nork performance.
plotsom: Plot self-organizing map.
plorv: Plot vecrors as lines from the origin.
plorvec: Plot vectors with different colors.
Pre- and post-processing:
prestd: Normalize data for unity standard deviation and zero mean.
postsrd: Unnormalize data which has been normalized by PRESTD.
trasrd: Transform data with precalculated mean and standard deviation.
premnmx: Normalize data for maximum of 1 and minimum of -1.
postmnmx: Unnormalize data which has been normalized by PREMNMX..
uamnmx: Transform data with precalculated minimum and maximum.
prepca: Principal component analysis on input data.
uapca: Transform data with PCA matrix computed by PREPCA.
posrreg: PoH-rraining regression analysis.
Simulink mpp,.,.t:
gcnsim: Generate a Simulink block to simulate a neural network.
Topology JUnctions:
grid top: Grid layer topology fUnction.
hextop: Hexagonal layer topology function.
rand top: Random layer topology function.
Transfer fonctio1u:
com pet: Competitive transfer function.
hardlim: Hard limit transfer function.
 Ul
m
"I

~i)
9.4 MATlAB Neural Network Toolbox 617

hard.lims: Symmetric hard limit transfer function.

logsig: Log sigmoid transfer function.
poslin: Positive linear transfer function.
purdin: Linear transfer function. n,
4i
radbas: Radial basis transfer function. .r
4:
satlin: Saturating linear transfer function. I
satlins: Symmetric saturating linear transfer function.
sofi:max: Soft max transfer function.
tansig: Hyperbolic rangent sigmoid transfer function.
cribas: Triangular basis transfer function.

Trainingfonctions:
trainb: Batch training with weight and bias learning rules.
trainbfg: BFGS quasi-Newton backpropagation.
rrainbr: Bayesian regularization.
trainc: Cyclical order incremental training w/learning functions.
uaincgb: Powell-Beale conjugate gradient backpropagation.
traincgF. Fletcher-Powell conjugate gradient backpropagarion.
rraincgp: Polak~Ribiere conjugate gradient backpropagation.
rraingd: Gradient descent backpropagation.
traingdrn: Gradient descent with momentum backpropagation.
rraingda: Gradient descent with adaptive lr backpropagation.
traingdx: Gradient descent w/momentum and adaptive lr backpropagation.
rrainlm: Levenberg-Marquardt backpropagarion.
trainoss: One step secant backpropagation.
trainr: Random order incremental training wllearning functions.
trainrp Resilient backpropagarion (Rprop).
trains: Sequential order incremental training w/lcarning functions.
trainscg: Scaled conjugate gradient backpropagarion.

Trnnsfir derivative .fi1nctions:

dhardlim: Hard limit transfer derivative function.
dhardlms: Symmetric hard limit transfer derivative funcrion.
dlogsig: Log sigmoid transfer derivative function.
dposlin: Positive linear transfer derivative function.
dpurelin: Hard limit transfer derivative function.
dradbas: Radial basis transfer derivative function.
dsadin: Saturating linear transfer derivative function.
dsadins: Symmetric saturating linear transfer derivative function.
dtansig: Hyperboiic tangem sigmoid transfer derivative function.
drribas: Triangular basis transfer derivative function.

Update networks .from previous versions:

nm2c: Update NNT 2.0 competitive layer.
nnr2elm: Update NNT 2.0 Elman backpropagarion network.
nnr2ff: Update NNT 2.0 feed-forward ne('Nork.
nnr2hop: Update NNT 2.0 Hopfleld recurrent network.
618 MATLAB Environment for Soft Computing Techniques

nm2lin: Update NNT 2.0 linear layer.

nm2lvq: Update NNT 2.0 learning vector quantization network.
nnt2p: Update NNT 2.0 perceptron. ;
nm2rb: Update NNT 2.0 radial basis network
nnt2som: Update NNT 2.0 self.organizing map.
Using networks:
sim: Simulate a neural network.
init: Initialize a neural network.
adapt: Allow a neural nerwork to adapt.
train: Train a neural neMork
disp: Display a neural network's properties.
display: Display rhe name and properries of a neural neLWork variable.
Vecton.:
cell2mat: Combine cell array of matrices into one matrix.
concur: Create concurrent bias vectors.
con2seq: Conven concurrent vectors m sequential vectors.
combvec: Create all combinations of vecmrs.
ind2vec: Convert indices to vectors.
mar2cell: Break mauix up into cell array of matrices.
minmax: Ranges of matrix rows.
nncopy: Copy matrix or cell array.
normc: Normalize columns of a matrix.
normr: Normalize rows of a mauix.
pnormc: Pseudo-normalize columns of a matrix.
quant: Discretiz.e values as multiples of a quantity.
seq2con: Convert sequential vectors ro concurrent vectors.
sumsqr: Sum squared elements of matrix.
vec2ind: Convert vectors to indices.
Weight fimctions:
dist: Euclidean distance weight function.
dotprod: Dot product weight function.
mandist: Manhattan distance weight function.
negdist: Dor product weight function.
normprod: Normalized dot product weight function.
Weight and bias initialization functions:
initcon: Conscience bias initialization function.
initz.ero: Zero weight/bias initialization function.
midpoint: Midpoinr weight initialization function.
randnc: Nonnalized column weight initialization function.
rand.nr: Normalized row weight initialization function.
rands: Symmetric random weight/bias initialization function.
Wt:ight derivative fonctionsr
ddotprod: Dot product weight derivative function.
 19.4 MATLAB Neural Network Toolbox 619

Train and adapt

There are two types of training that are given below.
;~
1. lncremmtal trainini- updating the weights after the presentation of each single training sample.
2. Batch training: updating the weights after each ~resenting the complete data set.

When using adapt, both incremental and batch t~ining can be used. When using train, on the other
hand, only batch training can be used, regardless of the format of the data. The big plus point of train is
that it gives you a lot more choice in training functions (gradient descent, gradient descent w/momenrum,
Levenberg-Marquardt, etc.) which are implemented very efficiently.
The difference between train and adapt is similar as tht difforence between paJm and epochs. When
using adapt, the property that determines how many times the complete training data set is used for
training the network is called net.adaptPacam.passes. But, when using train, the same property is called
net. trainParam.epochs.
net.trainFcn = 'traingdm';
net.trainParam.epochs = 1000;
>> net.adaptFcn = 'adaptwb';
>> ner.adaptParam.passes = 10;

19.4.3 Neural Network Graphical User Interface Toolbox

A GUI can be used to

1. create networks;
2. create data;
3. train the networks;
4. export rhe networks;
5. export the data to the command line workspace.

The salient features of GUI are the following:

1. It is designed to be simple and user-friendly. This tool lets you import potentially large and complex data
sets.
2. It also enables you to create, initialize, train, simulate and manage the networks. It has the GUI
Network/Data Manager window.
3. The window has its own work area, separate from the more familiar command line workspace. Thus,
when using the GUI, one might "export" the GUI results to the (command line) workspace. Similarly,
one might "import" results from the command line workspace to the GUI.
4. Once the Network/Data Manager is up and running, create a network, view it, train it, simulate it and
export the final results to the workspace. Similarly, import data from the workspace for use in the GUI.

This tool lets you import potentially large and complex data sets. The GUl also enables you ro creare,
initialize, train, simulate and manage your networks. Simple graphical representations allow you to visualize
and understand network archiu:crure (see Figure 19-2).
The following example deals with a perceptron network. h gives a step-by-step procedure of creating a
network.

~;,
 -~'=-

620 MATLAB Environment for Soft Computing Techniques

Figure 192 This window displays ponions of ilie neural network GUI. Dialogs and panes allow you ro
visualize your network (rop), evaluate training results (botrom), and manage your
nerworks (ccnrer).

Create a Perceptron Network (NN Tool)

Create a perceptron network to pertOrm the OR function in this example. It has an input vecror clara
I = [0 0 I 1; 0 1 0 l] and a rargervector clara 2 = [0 I 1 1]. The network can be called OR Net. Once created,
rhe network will be trained. Then save the nerwork, its ourput, ere. by "exporting" it m the command line.
Input and target:
To starr, cype NN cool. The window shown in Figure 19-3 appears.
First step is to define rhe network input, data l, as having the particular value [0 0 1 I;
0 l 0 I]. Thus, dte nerwork had a two-element input and four sets of such two-dement vectors are pre-
sented ro it in training. Create New Data appears. Set the Name ro data l, the Value to [0 0 I I; 0 1 0 I].
and make sure that Data Type is set ro Inputs. The Create New Data window will then look like the one
shown in Figure 19-4.
Now dick Create to actuaHy create an input file data I. The Network/Data Manager window comes up
and data 1 is shown as an input.
Step 2 is to create a network target. Click on New Data agajn, and this rime enrer the variable name
i
data 2, specify the value [0 I I I] and click on Target under data rype. The window will look rhe one given
in Figure 19-5.

l
~
 19.4 MATLAB Neural Network Toolbox 621

Figure 193 Neural network launch pad window network manager.

Figure 194 Window for creating new data (inpurs).

Again click on Create and see the resulting Network/Data Manager window that has data 2 as a target as
well a~ the previous data I as an input.
Create network:
Now we want to create a new network, which is OR NET. To do this, click on New Network, and a Create
New Network window appears. Enter OR NET under Network Name. Set the Network Type to Perceptron,
because that is the kind of network to create. The input ranges can be set by entering numbers in that field,
but iris easier to get them from the particular input data that you want to use. To do iliis, click on the down

~
 622
MATLAB Environment for Soft Computing Techniques

r-r;c
. - - - - - - - - - - - - - - - - --
_' C1e.:~tc New D,l\,l

- -- ---1
I----~-- __::------=--=---=:=--=:=4jl ::
a a2_ !I

0 111]
r--1 !I "

r: ~-I i
'1 .

7 - -------
it ..
r~-------~- r~~~~_.:,:.:_-- :-~
-1
r - -
Figure 195 Window for creating new dala (targets).

Figure 196 Window for creating new network.

arrow ar the right side of Input Range. This pull-down menu shows thar you can get rhe inpU[ ranges from
rhe file p. This should lead ro input ranges [0 1; 0 1]. We wane to use a hardlim transfer function and a
learnp learning function, so set those values using the ar:ows for Transfer function and Learning function,
respectively. By now your Create New Network window should look like the one given in Figure 19-6.
Next you might look at ilie nerwork bydicking on View (Figure 19-7). Figure 19-7 shows a nerwork with
a single input (composed of rwo elements), a hardlim transfer function and a single output which is going to
be created. This is the perceptron nerwork. Now click Create to generate the nerwork. Now go back ro the
Network/Data Manager window. Note that OR NET is now listed as a nerwork.
Train the perctptron:
To train the network, click on OR NET to highlight ir. Then dick on Train. This leads to a new window
labeled Network:OR NET (~igure 19~8). At this poim you can view the nerwork again by clicking on the
top tab Train. You can also check on the initialization by clicking on the top tab Initialize. Now click on the
 19.4 MATLAB Neural Network Toolbox 623

Figure 19 7 Window to view the new archirecrure created.

Figure 198 Perceptron net for AND function.

top tab Train. Specify the inpms and outputs by clicking on ilie left tab Training Info and selecting datal
from the pop-down list of inputs and data2 from the puJI-down list of targets (Figure 19-8).
On clicking the Training Parameters rab, ir shows parameters such as the epochs ;md error goal. These
parameters can be changed at this point if desired. Now click Train Network to train the perceptron network
and see the following training results (Figure 19~9).
Thus, the network was trained to zero error in four epochs. (Note rhat other kinds of nerworks commonly
do not train to zero error and their errors commonly cover a much larger range. On that account, plot lheir
errors on a log scale rather than on a linear scale such as that used above for perceprrons.)

r
624 MATI.AB Environment for Soft Computing Techniques

Figure 199 Training resulrs of OR funccion.

To check cltat the trained network does indeed give zero error by using the input p and simulating ilie
nerwork, go to the Network/Data Manager window and click on Network Only: Simulate. This will bring
up the Network: OR NET window. Click there on Simulate. Now use the lnput pull-down menu ro specify
data 1 as the input, and label the output as OR NET_ourputsSim ro distinguish it from the training ourpur.
Now click Simulate Network in the lower.right corner. Look at the Network/Data Manager. It will show
a new variable in rhe output: OR NET_outputsSim. Double-click on it and a small window Data:OR
NET_outputsSim appears with the value [0 I 1 1]. Thus, the nerwork does perform the OR of the inputs,
giving 0 as an output only in this first case, when both inputs are 0.

119.5 Fuzzy Logic MATLAB Toolbox

Fuuy logic in MATLAB can be dealt very easily because of the existing new Fuzzy Logic Toolbox. This
provides a complete set of functions to design and implement various fuzzy logic processes. The major fuzzy
logic operations i_nclude fuzzification, defuzzification and the fuzzy inference. These all are performed by
means of various functions and can be even implemented using the GUI. Many of the applications can be
simulated using the "fuzzy logic controller" simulink block present in Maclab Simulink toolbox. The features .'
are rhe following.
I. It provides tools to create and edit Fuzzy Inference Systems (FIS).
2. It allows integrating fuzzy systems imo simulation with SIMULINK.
3. lr is possible to create stand-alone C programs that call on fuzzy systems built with MATLAB.
The Toolbox provides three categories of tools:
1. command line functions;
2. graphicaJ or interaqive tools;
3. SIMULINK blocks.

j
19.5 Fuzzy loQic MATIAS ToolboK
625

I
- 19.5.1 Commands in Fuzzy Logic Toolbox
The various commands in Fuzzy Logic Toolbox to be operated in command line are as follows:
GUI editorr:
anfisedir: ANFIS training and testing UI toQ1
findduster: Clustering m tool.
fuzzy: Basic FIS editor.
mfedir. Membership function ediror.
ruleedit: Rule editor and parser.
ruleview: Rule viewer and fuzzy inference diagram.
surfview: Output surface viewer.

Membership fonctions:
dsigmf: Difference of rwo sigmoid membership functions.
gauss2mf: Two-sided Gaussian curve membership function.
gaussmf: Gaussian curve membership function.
gbdlmf: Generalized bell-shaped curve membership function.
pimf. Pi-shaped curve membership function.
psigmf: Product of two sigmoid membership functions.
smf: S-shaped curve membership fUnction.
sigmf Sigmoid curve membership function.
trapmf: Trapezoidal membership function.
trimf Triangular membership function.
zmf: Z-shaped curve membership function.

Command line FIS fimctions:

addmf: Add membership function to FIS.
addrule: Add rule ro FIS.
addvar: Add variable to FIS.
defuzz: DefUzzify membership function.
evalfis: Perform fuzzy inference calculation.
evalmf: Generic membership function evaluation.
gensurf: Generate PIS output surface.
getfis: Get fUuy system properties.
mf2mf Translate parameters berween fUnctions .
newfis: Create new FIS.
pars rule: Parse fuzzy rules.
plotfis: Display PIS inpur-output diagram.
plormf: Display all membership fUnctions for one variable.
readfis: Load PIS from disk.
rmmf: Remove membership function from PIS.
rmvar: Remove variable from FIS.
setfis: Set fuzzy system properties.
showfis: Display annotated FIS.
showrule: Display FIS rules.
writefis: Save FIS to disk.
 626
MATLAB Environment for Soft Computing TechniqUes

Advanctd techniqrus:
anfiS: Training routine for Sugeno-rype FIS (MEX only).
fern: Find dusters with fuzzy cmeans clustering.
genfis1: Generate FIS matrix using generic method.
genfts2: Generate FIS matrix using subtractive clustering.
subdust: Estimate cluster centers with subtractive clustering.
Miscellaneous fonctions-. 1.
;
convenfis: Convert vl.O fuzzy matrix ro v2.0 fuzzy structure.
discfis: Discretize a fuzzy inference sysrem.
evalmmf: For multiple membership functions evaluation.
fsrrvcac: Concatenate matrices of varying size.
fuzarich: Fuzzy arithmacic function.
findrow: Find the rows of a matrix chat match the input string.
genparam: Generates initial premise parameters for ANFIS learning.
prober: Probabilistic OR.
sugmax: Maximum ourput range for a Sugeno system.
GUI helpa fikr.
cmfdlg: Add cus[Qmized membership function dialog.
cmrhdlg: Add customized inference method dialog.
fisgui: Generic GUI handling for the Fuzzy Logic Toolbox.
gfmfdlg: Generate FIS using grid partition method dialog.
mfdlg: Add membership function dialog.
mfdrag: Drag membership functions using mouse.
popundo: Pull the last change off the undo stack.
pushundo: Push the current FIS clara onto rhe undo stack.
savedlg: Save before closing dialog.
sratmsg: Display messages in a sraws field.
upddis: Update Fuzz.y Logic Toolbox GUI tools.
wsd.1g: Open from/save to workspace dialog.

19.5.2 Simulink Blocks in Fun.y Logic Toolbox

Once fuzzy system is created using GUI tools or some other merhod, it can be directly embedded imo
SIMULINK using the fuzz.y logic controller (FLC) block as shown in Figure 19-10.
Make sure that the FIS matrix corresponding to the fuzzy system is both in the 1v1ATLAB workspace
and referred to by name in the dialog box associated with this FLC. Although iris possible to use the Fuzzy

~ Fuzzy logic
controller

Figure 19-10 Fu'l.Zy logic controller Simulink block.

 19.5 Fuzzy Logic MAT LAB Toolbox 627

1L]QJ}[]8{I}{l]{I}
Gaussian MF Gaussian2 MF Pi-shaped MF Probabilistic OR S-shaped MF Trapazoidal MF Z-shapedMF

1..:.;:
;:- ITJJillJ[D8WQTI
Genaralized Ball MF OiH. Sigmoidal MF Prod. Sigmoidal MF Probabi~stic Rule Agg Sigmoidal MF Traiangular MF

Figure 19-11 Membership functions.

Logic Toolbox by working srriccly from the command line, in general it is much easier to build a system up
graphically so rhat GUI tools are commonly used for building, editing and observing FIS.
The process of mapping from a given input to an output using fu:a.y logic involves membership functions,
fuzzy logic operators and If-Then rules.
Membership fonctions: This toolbox includes 11 built-in membership function cypes, built from several basic
functions: piecewise linear functions (triangular and traptz<Jida4, the Gaussian distribution function (Gaussian
mrves and generalized beil), the sigmoid curve, and quadratic and cubic polynomial a.~rves (Z, S, and Pi curves)
(Figure 19-11).
Fuzzy logic operators: According ro the fuzzy logical opemions, any number of well-defined methods can fill
in for the AND operation or the OR operation. In the Fuzzy Logic Toolbox, two built-in AND methods are
supported: min (minimum) and prod (algebraic product). Two built-in OR methods are also supported: mllX
(maximum) and the probor (probabilistic OR, also known as algebraic sum).
Based on implication method, rwo built-in methods are supported. These are the same functions that are
used by the AND method, so that, min method truncates the output fuzzy set and prod scales the output
Fuzzy set.
Based on aggregation method, cllree built-in methods are supported: max (maximum), probor (probabilistic
OR) and mm (simply the sum of each rule's output set).
Although centroid calculation is the most popular defuzz.ification method, rhere are five built-in methods
supported: centroid, bisector, middle ofmaximum, largest ofmllXimum nnd smallest ofmaximllm.
If Then rules: Since rules can be edited in three different formats (verbose, symbolic and indexed), verbose
format makes rhe system easier to interpret. Every rule has a weight (a number between 0 and l) which is
applied to the number given by the antecedent. Generally this weight is 1 and so it has no effect at a11 on the
implication process. For example, let us enter a sample rule (rule number one):
Verbose format. 1. if Temperature is warm then Sky is grey {l)
Symbolic format. 1. (Temperature = =wa:rm) =>Sky = grey {l)
Indexedfonnar. 1, 1 (l), I
Here the first "1" corresponds to the inpur variable, the second corresponds to the output variable, the
third displays the weight applied to each rule and the fourth is shorthand that indicates whether this is an
OR (2) rule or an AND (1) rule. So a literal interpretacion of rule number one is: "ifinputl is MFl (the first
membership function associared with input l) then output1 should be MF1 (the first membership function
associated with output 1) with the weight 1". Note that as long as clle aggregacion method is commutative,
the order in which the rules are executed is not important.
Once an FLC is created, it can be saved on a disk (FIS-fi\e is created, i.e., juggler.fis) as an ASCII text
format so that it can be edited and modified. An FLC can also be saved into MATLAB workspace as a matrix
628 MATLAB Environment for Soft Computing Techniques

variable (FIS matrix) so that it can be modified; however, its represemarion is extremely different from PIS-file
represenrarion.

I 19.5.3 Fuzzy Logic GUI Toolbox

The fuzzy logic can be simulated in MATLAB using GUI. On r:yping "fuzzy" in the command prompt, __i.f.
the fuzzy GUI toolbox opens up. The main windows corresponding to Fuzzy GUI tools are shown in ~
Figu<es 19-12-19-17. "'c
In Figure 19-12 the FIS ediror-Mamdani or Sugeno model- is selected. The inference mechanism can be
selected at iliis step. The various mechanisms to be selected in rhe FIS editor are AND method, OR method,
implicacion, aggregation and defuzzificarion. Here, the nwnber of input and output variables can be specified.
The membership function editor is shown in Figure 19-13. In this editor, for the inj>ut variable and the
corresponding ourput variable, the membership functions using linguistic variables along with their range are
defined. Figure 19-13 shows the membership function editor for the input variable and Figure 19-14 shows
the membership function editor for the output variable.
The rules to be formed based on the input variables ro get the output are defined in the rule editor. The
inference of these rules gives the fuzzified ourpur of the problem under consideration. For rule definition either
AND connective or OR connective can be used. Figure 19-15 shows a rule editor with 3 rules formulated.
The formulated rules can be viewed in the rule viewer as shown in Figure 19-16. On viewing these rules,
information about the output can be obtained.
Figure 19-17 shows the surface view of the defmed fuzzy inference editor. The fuzzy logic GUI toolbox
helps us in designing a suitable FLC module for any application.

~ ~I~U!i
'
--
'" "' 0
(5olQt>110)
~- !(U)

"'"""
.'

Figure 19~12 Basic FIS editor.

j_
 .l
~

19.5 Fuzzy_ Logic MATLAB Toolbox

629
:I

l
Ii
I
!

~
Figure 1913 Membership function editor (input 1).

i.
I

Figure 1914 Membership function editor (output I, in Sugeno-style).

630

Rule editor (sample with 3 rules in verbose format).

l=:lgure 1916 Rule viewer (in Sugeno-style).

r
 19.6 Genetic Algorithm MATLAB Toobox 631 ' ,
"

I
~
Figure 1917 Surface viewer (input land inpur 2 versus output 1).

119.6 Genetic Algorithm MATLAB Toolbox

The genetic algorithm (GA) is a method for solving both constrained and unconstrained optimization prob-
lems that are based on na(Ural selection, the process that drives biological evolution. The GA repeatedly
modifies a population of individual solutions. At each step, the GA sdecrs individuals at random from the
current population to be parents and uses them to produce rhe children for the next generation. Over suc-
cessive generations, the population "evolves" toward an optimal solution. One can apply the GA to solve
a variety of optimization problems that are not well suited for standard optimization algorithms, including
problems in which the objective function is discontinuous, nondifferemiable, srochastic or highly nonlinear.
I The GAand direct search roolbox are a collection offuncrions that extend the capabilities of the optimiza~
I
t tion toolbox and the MATLAB numeric computing environment. The GA and direct search toolbox include
routines for solving optimization problems using
l. genetic algorithm;
2. direct search.
These algorithms enable you to solve a variety of optimization problems that lie omside the scope of the
Optimization Toolbox.
The GA uses three main rypes of rules at each step to create the next generation from the current
population:
l. Selection rules select the individuals, called parents, that contribute to the populacion at the nexrgeneration.
2. Crossover ruks combine two parents to form children for the next generation.

r:"
632 MATIAS Environmenl for Soft CompUiing Techniques
~{
,:.:
3. Mutarion roles apply random changes to individual parents to form children.
The GA ar the command line calls the GA fimcrion ga with the syntax lj;::
[x fval] = ga(@fiUtessfun, nvars, options) f'
where @firnessfun is a handle tO the fimess function; nvars is the number of independent variables for rhe
fitness function; options is a strucrure containing options for the GA If you do not pass in this argument,
ga uses its default options.
The results are given by
x: Point at which the final value is attained.
fval: Final value of the fitness function.
The GA tool is a GUI that enables one ro use the GA withom working at the command line. To open the
GA tool, enter

Garool
at dte MATIAB command prompt.
The Optimization Toolbox extends the MATLAB technical computing environment with tools and
widdy uSed algoriduns for standard and large-scale optimization. These algorithms solve conscrained and
unconsrta.ined continuous and discrete problems. The toolbox includes functions for linear programming,
quadratic programming, nonJinear qpcimization, nonlinear least squares, nonlinear equations, multi-objective
optimization and binary integer programming.
i
I 19.6.1 MATLAB Genetic Algorithm Commands

The various commands used in GA MATLAB toolbox are as follows.

bin2im: BINary string to INTeger string conversion.
bin2reaJ: BINary suing to REAL vector conversion.
bindecod: BINary DECODing to binary, integer or real numbers.
Compdiv: COMPute DIVerse things ofGEA Toolbox.
compdiv2: COMPute DNerse things ofGEA Toolbox.
compere: COMPETition between subpopulations.
Comploc: COMPute LOCal model things of toolbox.
Compplm: COMPute PLOT things ofGEA toolbox.
geamain2: MAIN function for Genetic and Evolutionary Algorithm toolbox for maclab.
Initbp: CReaTe an initial Binary Population.
. ~

Inirip: CReaTe an initial {lnreger value) Population.

lnirpop: INITialization of POPulation (including innoculation).
Inirpp: Create an INITial Permutation Population.
Initrp: INITialize a Real value Population.
Migrate: MIGRATion of individuals between subpopulations.
Murare: high level MUTATion function.
Murbin: MUTation for BINary representation.
Mutbmd: real value Mutation like Discrete Breeder generic algoridliil.
murcomb: MUTation for combinatorial probl~ms.
mutes 1: MUTation by.Evolutionary Strategies 1, derandomized sdf adaption.
mures2: MUTarion by Evolutionary Strategies 2, derandomized self adaption.

~~-
 633
19.6 Genetic Algorithm MAllAB Toolbox

mure.xch: Millarion by eXCHange.

mutint: MUTarion for INTeger representation.
Murinven: MUTation by INVERTing variables.
murmove: MUTation by MOVEing variables.
Mutrand: MUTation RANDom.
Muuandbin: MUTation RANDom of binary variables.
Mutrandint: MUTation RANDom of integer variables.
Mutrandperm: MUTation RANDom of binary variables.
mutrandreal: MUTation RANDom of real variables.
Mutreal: real value Mutation like Discrete Breeder genetic algorithm.
Murswap: MUTation by SWAPping variables.
mutswaptyp: MUTation by SWAPping variables ofidencical cype.
r.mkgoaJ, perform goal preference calculation between multiple objective values.
RANK-based fitness assignment, single and multi objective, linear and nonlinear.
Rankintr
rankplt: RANK rwo multi objective values Partially Less Than.
rankshare: SHARing between individuals.
reed is: RECombination DIScrete.
reafp, RECombinarion Double Point.
recd.prs: RECombination Double Point with Reduced Surrogate.
recgp: RECombination Generalized Position.
recint: RECombination extended INTermediate.
reel in: RECombinarion extended LINe.
reclinex: EXtended LINe RECombination.
recmp: RECombination Multi-Point, low level function.
recombin: high level RECOMB!Nat;on function.
recpm: RECombinarion Panial Matching.
recsh: RECombinarion Shuffle.
recshrs: RECombination SHuffle with Reduced Surrogate.
recsp: RECombination Single Point.
recsprs: RECombination Single Point with Reduced Surrogate.
reins: high-level RE-INSertion function.
reinsloc: RE-INSertion of offspring in population replacing parents LOCal.
remsreg: REINSertion of offspring in REGional population model replacing parents.
selection: high level SELECfion function.
seUocal: SELection in a LOCAL neighborhood.
sehws: SELection by Roulene Wheel Selection.
selsus SELection by Stochastic Universal Sampling.
sehour: SELection by TOURnament.
seltrunc: SELection by TRUNCation.
tbx3bin: ToolBoX funaion ro define parameters for optimization of binary variables.
rbx3comp: ToolBoX function to define parameters for COMPeting subpopulation.
tbx3esl; ToolBoX function to define parameters for local oriented optimization of real
variables.
tbx3gu;fun 1' ToolBoX function to define parameters for optimization, test of gui.
ToolBoX function to define parameters for optimization of integer variables.
tbx3im:
ToolBoX function to define parameters for optimization of permutation variables.
rbx3perm:

-

634 MATLAB Environment for Soft Computing Techniques

tbx3plo<l: ToolBoX function to define parameters for grafical display options.

tbx3real: ToolBoX function to define parameters for optimization of real variables.
terminat: TERMINATion function.
Objective fonctions:
initdopi: INI1ialz.ation function for DOuble Integrator objdopi.
initfunl: INI1ialz.ation function for de jong's FUNction 1.
mopfonsecal: MulriObjeccive Problem: FONSECA's function I.
mopfonseca2: MuhiObjeccive Problem: FONSECA's function l.
moptest: MultiObjective function TESTing.
obj4wings: OBJective function FOUR-WINGS.
objbran: OBJective function for BRANin rcos function.
objdopi: OBJective function for DOuble Integrator.
objeaso: OBJective function for EASom function.
objflerwell: OBJective function after FLETcher and PoWELL.
objfracral: OBJective function Fractal Mandelbrot.
objfi.ml: OBJective ftmction for de jong's FUNction 1.
objfun l 0: OBJective function for ackley's path FUNction 10.
objfunll: OBJective function for langermann's FUNction 11.
objfun12: OBJective function for michalewicz's FUNction 12.
~. objfunla: OBJective function for axis parallel hyper~ellipsoid.
:\,
objfunl b: OBJective function for rotated hyper~ellipsoid.
objfunlc: OBJective function for moved axis parallel hyper ellipsoid I c.
objfun2: OBJective function for rosenbrock's FUNction.
" objfun6: OBJective function for rastrigins FUNction 6.
,, objfun7: OBJective function for schwefel's FUNction.
Tr objfun8: OBJective function for griewangk's FUNction.
I
objfun9: OBJective function for sum of different power FUNction 9.
objgold: OBJective function for GOLDstein~price function.
objharv: OBJective function for HARVest problem.
objintl: OBJective function forINT function I.
objim2: OBJective function forINT function 2.
objint3: OBJective function{~r INT function 3.
objim4: OBJective function ~\r INT function 4.
objlinq: OBJective function for discrete LINear Quadratic problem.
objlinq2: OBJective function for LINear Quadratic problem 2.
objonel: OBJective fi.mction for ONEmax function I.
objpush: OBJeccive function for PUSH~cart problem.
objridge: OBJective function RIDGE.
objsix.h: OBJective function for SIX Hump camelback function
objsoland: OBJective function for SOLAND function.
objtspl: OBJective function for the traveling salesman exa~.1ple.
objtsplib: OBJective functi_on for the traveling salesman library.
plotdopi: PL01ing ofDO{Ppel)uble Integration results.
plottsplib: PL01ing of results ofTSP optimization {TSPLIB examples).
simdopi 1: M~file description of the SIMULINK system named SIMDOPil.
simdopiv: SIMulation Modell ofDOPpellmegrator, s~function, Vectorized.
'if
19.6 Genetic Algorithm MATLAB Toolbox
635

simlinql: M-file description of the S!MULINK system named SIMLINQl.

simlinq2: Modell of Linear Quadratic Problem, s~function.
tsp_readlib: TSP utility function, reads TSPLIR'dara files.
tsp_wcity: TSP utility function, reads US City defiriitions.
Plot fimctions:
Fitdistc: FITness DISTance Correlacion computation.
meshvar: create grafics of objective funccions wilh plotmesb.
plormesh: PLOT of objective functions as MESH Plot.
plotmop: PLOT properties of MulriObjective functions.
res\oolc LOOK at saved RESults.
resplot: RESult PLOTing of GEA Toolbox optimization.
samdara: sammon mapping: data examples.
samgrad: Sammon mapping gradient calculation.
sammon: Multidimensional scaling (SAMMON mapping).
samobj: Sammon mapping objective function.
samplot: Plot function for Multidimensional scaling (SAMMON mapping).

19.6.2 Genetic Algorithm Graphical User Interface

The GA tool is a GUI that enables you to use the GA without working at the command line. To open the
GA tool, enter

gatool

at the MATLAB command prompt. This opens the tool as shown in Figure 19~18. To use the GA mol, you
must first enter the following information.
1. Fitness function: The objective function you want to minimize. Emer the fitness ftmcdon in the form
@firnessfun, where fimessfun.m which is an M-file that compmcs the fitness function.
2. Nttmber ofvariables: The number of variables in the given fitness function should be given.

The plot options

1. best fitness;
2. best individual;
3. distance;
4. expectation;
S. genealogy;
I
I.
6. range;
7. score diversity;
8. scores;
9. selection;
10. stopping.
 636
MATLAB Environment for Soft Computing Techniques

Figure 1918 Generic algorithm rool.

i
On the basis of the problem, custom function may also be built. The various parameters essential for running
GA roo I should be specified appropriately. The parameters appear on the right-hand side of the GA tool. The
description is as follows:
I
l. Population: In this case population type, population size and creation function may be selected. The I
initial population and initial score may be specified, if nor, the "GA tool" creates them. The initial range
should be given.
2. Fitness scaling. h should be any of the following:
rank;
proponional;
rop;
shift linear;
cuswm. I
3. Selection: The selection is made on any one of the methods shown in Figure 19-19.
4. Reproduction: In reproduccion dte elite count and crossover fraction shouJd be given. If elite count is nor
specified, it is taken as 2 (Figure 19-20).
5. Mullltion: Generally GaUssian or uniform mutation is carried our. The user may define own custOmized
mutation operation (Figure 19-21).

/;,
....!ii:._
19.6 Genetic Algorithm MATIAS Toolbox 637

Figure 19-19 Selection.

Figure 19-20 Reproduction.

I
\

Figure 1921 Murarion.

6. Crossover. The various crossover techniques are shown in Figure 19-22.

7. Migrmion: The parameter for migration should be defined as in Figure 19-23.
8. Hybrid fimction: Any one of the hybrid functions shown in Figure 19-24 may be selected.
9. Stopping criten'a: The stopping criteria play a major role in simulation. They are shown in Figure 19-25.
The other parameters Ourput function, Display to command window and Vectorize may be suitably defined
by the user.

Running and Simulation

The menu shown in Figure 19-26 helps the user for running the GA mol.
The running process may be temporarily stopped using "Pause" option and permanently stopped using
"Stop" option. The "current generation" will be displayed during the iterarion. Once ilie iterations are
complered, ilie srarus and results will be displayed. Also ilie "final point" for the fitness funcdon will be
displayed.
 638 MATLAB Environment for Soft Computing Techniques

Figure 1922 Crossover.

~. Figure 1923 Migration.

:\

"'
rI '

Figure 1924 Hybrid funaion.

Figure 1925 Stopping criteria.

 639
19.7 Neural Network MATLAB Source Codes

Figure 1926 Run solver.

119.7 Neural Network MATLAB Source Codes

1. Write a program to implement AND function usingADALlNE with bipolar inputs and outputs

Source Code
clear all;
clc;
disp('adaline network for and function bipolar inputs, bipolar
targets');
xl=(l 1 -1 -1]; %input pattern
x2=[1 -1 1 -ll; %input pattern
x3=[1 1 1 1]; %x3 for bias
t=[l -1 -1 -1]; %target
wl=O.l;
w2=0.1;
b=O.l;
\. alpha=O.l;
e=2;
"'"i:: delwl=O;
u:_ delw2=0;
!.~~

~
 .

640
r
MATLAB Environment for Soft Computing Techniques

delb=O;
epoch=O;
while (e>l. 018)
epoch=epoch+l
e=O;
for i=l:4
nety(i)=wl*Xl(il+w2*x2(i)+b;
nt=[nety(i) t(i)]; %netinput,target
delwl=alpha*(t(i)-nety(i))*xl(i);
delw2=alpha*(t(i)-nety(i))*x2(i);
delb=alpha*(t(i)-nety(i)J*x3(i);
wc=[delwl delw2 delb]; %weight chanches
wl=wl+delwl; %updating of weights
w2=w2+delw2;
boo:b+delb;
w= [wl w2 b]; _%weights
x=[xl(i) x2(i) xJ(i)]; %input pattern
pr=[x nt we w] %to print the result
end
for i=l:4
nety(i)=wl*xl(i)+w2*x2(i)+b;
e=e+(t(i)-nety(i)) 2;
end
end

2. Write a program to implemem AND function using MAD ALINE with bipolar inputs and outputs.
Source Code
clear all;
clc;
disp('madaline network for and function bipolar inputs, bipolar
targets');
xl=[1 1 -1 -1]; %input pattern
x2=[1 -1 1 -1]; %input pattern
x3=[1 1 1 1); %x3 for bias
t=[1 -1 -1 -1]; %target
wll=0.1;
wl2=0.1;
w21=0.1;
w22=0.1;
b1=0.1;
b2=0.1;
b3=0.5;
v1=0.5;
v2=0.5;
alpha=0.5;
e=2;
delwll=O;
delwl2=0; I
delw21=0; I
delw22=0;
I
I

.-l
 641
19.7 Neural Network MATLAB Source Codes

delbl=O;
de1b2=0;
delb3=0;
delvl=O;
delv2=0;
epoch=O;
while (e>1.00)
epoch=epoch+ 1
e=O;
for i=l:4
zinl=xl(i)*wll+x2(i)*w21+bl;
zin2=xl(i)*wl2+x2(i)*w22+b2;
z=[zinl zin2];
i f (zinl>=Ol
zl=l;
else
zl=-1;
end
if {zin2>=0l
z2=1;
else
z2=-l;
end
hid=[zl z2];
nety=b3+zl*vl+z2*v2;

i f (nety>=0'
y=1;
else
y=-l;
end
nt=lt(il nety y];

if (t(il==l)
if (zinl<zin2)
delbl=alpha*(l-zinl);
bl=bl+delbl;
delwl1=alpha*(l-zinl)*xl(i);
wll=wll +delwll;
delw21=alpha*(l-zinl)*xl(i);
w2l=w21+delw21;
else
delb2=alpha*(l-zin2);
b2=b2+delb2;
delwl2=alpha*(l-zin2)*x2(i);
wl2=wl2+delwl2;
de1w22=alpha* (l-zin2) *x2 (i);
w22=w22+delw22;
end
 642
MATLAB Environment for Soft Computing Techniques

elseif {t(iJ==-1)
i f (zinl>O)
delbi=alpha*(-1-zinl);
bl=bl+delbl;
delwll=alpha*{-1-zinl)*xl(i);
wll=wll+delwll;
delw21=alpha*(-1-zinl)*xl{i);
w21=w21+delw21;
else
delb2=alpha*(-1-zin2);
b2=b2+delb2;
delw12=alpha*(-1-zin2)*x2{i);
wl,.2=w12+delwl2;
delw22=alpha*(-l-zin2)*x2(i);
w22=w22+delw22;
end
end

del=[delwll delw21 delbl delwl2 delw22 delb2];

in=[xl(i) x2(i) x3(i)];
bi=fvl v2 b3];
~
pr=[in z hid nt del bi]
\ end

for i=l:4
" zinl=bl+xl(i)*wll+x2(i)*w21;
zin2=b2+xl(i)*wl2+x2(i)*w22;
..., z=[zinl zin2];
"' i f (zinl>=O)
zl=l;
else
zl=-1;
end
if (zin2>=0)
z2=1;
else
z2=-l;
end
nety=vl*zl+v2*z2+b3;
e=e+(t(i)-nety) 2;
end
end

3. Write a MATLAB program to construa and resr auco associative network for input vector using HEBB
rule.
Source Code
clear all;
clc;
disp(' AUTO ASSOCIATIVE NETWORK-----HEBE RULE');
 19.7 Neural Network MATLAB Source Codes
643

w=[O 0 0 0 ;0 0 0 0 ;0 0 0 0 ;0 0 0 0 ];
s=[1 1 1 -1];
t=[1 1 1 -1];
ip=[l -1 -1 -1];
disp('INPUT VECTOR');
s
for i=1:4
for j=1:4
w(i, j)=w{i, j )+(s (i) *t(j));
end
end
disp ( 'WEIGHTS TO STORE THE GIVEN VECTOR IS' ) ;
w
disp('TESTING THE NET WITH VECTOR');
ip
yin=ip*w;
for i=1:4
i f yin(i)>O
y{i)=1;
else
y(i)=-1;
end
end
if y==S
disp('PATTERN IS RECOGNIZED')
else
disp('PATTERN IS NOT RECOGNIZED')
end
Output
>> AUTO ASSOCIATIVE NETWORK-----HEBE RULE
INPUT VECTOR
s =
1 1 1 -1
WEIGHTS TO STORE THE GIVEN VECTOR IS
w
1 1 1 -1
1 1 1 -1
1 1 1 -1
-1 -1 -1 1
TESTING THE NET WITH VECTOR
ip =
1 -1 -1 -1
PATTERN IS NOT RECOGNIZED
4. Write a NiATLAB program ro construct and test auto associative network for input veaor using ourer
produa rule.
Source Code
clear all;
clc;

~;
 ?/
644 :.~
MATLAB Environment for Soft Computing Techniques
'!: 1
disp('To test Auto associatie network using outer product rule for
foqoWing input vector');
xl=[l -1 1 -1];
x2=[1 1 -1 -1];
n=O;
wl=xl' "'xl;
1
w2=x2' *x2;
wm=wl+w2;
disp{' input');
x1
x2
disp ( 'Target ) ,
x1
x2
disp ( 'Weights' ) ;
w1
w2
disp( 'Weight matrix using Outer Products Rule');
wm
yin=xl*wm;
yin
for i=l:4
if(yin(i)>O)
y=l;
else
y=-1;
end
ny(i)=y;
if(y==xl(i))
n=n+l;
end
end
ny
if(n==4)
disp ('This pattern is recognized');
else
disp('This pattern is not recognized');
end
n=O;
yin=x2*wm;
yin
for i=1:4
if(yin(i)>O)
y=l;
else
y=-1;
end
ny(i}=y;
if (y==x2 (i))
n=n+l;

I
.. Jii.
 645
19.7 Neural Network MATLAB Source Codes

end
end
ny
if (n==4l
disp( 'This pattern is recognized');
else
disp('This pattern is not recognized');
end
end

Output
>> To test Auto associative network using outer product rule for
following input vector
input
x1 =
1 -1 1 -1
x2 =
1 1 -1 -1
Target
x1 =
1 -1 1 -1
x2 =
1 1 -1 -1
Weights
w1 =
1 -1 1 -1
-1 1 -1 1
1 -1 1 -1
-1 1 -1 1
w2
1 1 -1 -1
1 1 -1 -1
-1 -1 1 1
-1 -1 1 1
Weight matrix using Outer Products Rule
wm
2 0 0 -2
0 2 -2 0
0 -2 2 0
-2 0 0 2
yin =
' _, 4 _,
ny =
1 -1 1 -1
This pattern is recognized
yin =
4 4 -4 -4
ny =
1 1 -1 -1
This pattern is recognized
646 MATLAB Environment for Soft Computing Techniques

5. Write a MATIAB pr?gram to construct and test heteroassociarive nerwork for binary inputs and targers.
Source Code
%To construct and test Heteroassociative network for binary inputs
and targets
clear all;
clc;
disp('Heteroassociative Network');
xl=[l 0 0 O];
x2=[1 1 0 0];
x3=[0001);
x4=[0 0 1 1];
t1=[1 OJ;
t2=[10];
t3=[01];
t4=[01];
n=O;
for i=l:4
for j=l:2
w(i,j)=((2*xl(i))-1)*{(2*tl(j))-1)+((2*x2(i))-1)*((2*t2(j))-1)+
( (2*x3 (i)) -1) * { (2*t3 (j)) -1) +( (2*x4(i)) -1) ( (2*t4 {j)) -1);
end
end
w
yinl=xl*w
yin2=x2*w
yin3=x3*w
yin4=x4*w
t1=[ 1 -1];
t2=[ 1 -1];
t3=[-1 1];
t4=[-1 1];
for i=l:2
if(yinl(i)>O)
yl(i)=l;
elseif (yinl(i)==OJ
yl(i)=O;
else
yl(i)=-1;
end
end
yl
for i=l:t.
if(yl(i)==tl(i))
n=n+l;
end
end
i f (n==2)
disp('The pattern is matched');
else
 19.7 Neural Network MAllAB Source Codes 647

disp('The pattern is not matched');

end
o; n=O;
for i=l:2
if(yin2(i)>O)
y2(il=l;
elseif (yin2(i)==O)
y2(i)=0;
else
y2(il=-l;
end
end
Y2
for i=l:2
if(y2(i)==t2(i))
n=n+l;
end
end
if (n==2l
disp('The pattern is matched');
else
disp{'The pattern is not matched');
end
n=O;
for i=l: 2
if(yin3(il>Ol
y3 (i)=l;
elseif (yin3{i)==Ol
y3{i)=0;
else
y3(i)=-l;
end
end
y3
for i=l::t
i f {y3 {i) ==t3 (i))
n=n+l;
end
end
i f (n==2)
disp('The pattern is matched');
else
disp('The pattern is not matched');
end
n=O;
for i=l:2
if(yin4.(i)>0)
y4{i)=l;
elseif (yin4{i)==0)
y4(i)=O;
648 MATIAS Environment for Soft Computing Techniques

else
y4(i)=-l;
end
end
y4
for i=l:2
if(y4(i)==t4(i))
n=n+l;
end
end
if (n==2)
disp('The pattern is matched');
else
disp('The pattern is not matched');
end
n-0;

6. Write a MATLAB program to implement Discrete Hopfield Network and rest the input panern.
Source Code
clear all;
clc;
disp('Discrete Hopfield Network');
theta=O;
x=[l -1 -1 -1;-1 1 1 -1;-1 -1 -1 1]
%Calculating Weight Matrix
w=x' *x
%calculating Energy
k=l;
while(k<=3)
ternp=O;
for i=l:4
for j=l:4
temp= temp+ (x (k, i) "w(i, j) x(k, j)) ;
end
end
E(k)=(-0.5)*temp;
k=k+l;
end
%Energy Function for 3 samples
E

%Test for given pattern s=[-1 1 -1 -1]

disp('Given input pattern for testing');
xl=[-1 1 -1 -11
temp=O;
for i=l:4
for j=l:4
temp=temp+{xl(i)*w(i,j)*x1(j));
end
end
19.7 Neural Network MATLAB Source Codes 649

SE= {-0. 5.) *temp

disp('By synchronous updation method');
disp('The net input calculated is'};
yin=x1*w
for i=1:4
if(yin(i)>theta)
y(i)=l;
elseif(yin{i)==theta)
y(i)=yin{i);
else
y(i)=-1;
end
end
disp('The output calculated from net input is');
y
temp=O;
for i=l:4
for j=l:4
temp=ternp+(y(i)*w(i,j)*y(j));
end
end
SE=(-0.5)*temp
n=O;
for i=l:3
i f {SE==E{i))
n=O;
k=i;
else
n=n+l;
end
end

if(n==3)
disp('Pattern is not associated with any input pattern');
else
disp ('The test pattern');
x1
disp('is associated with');
x(k, :)
end

Output

>> Discrete Hopfield Network

1 -1 -1 -1
-1 I 1 -1
-1 -1 -1
650 MATl.AB Environment for Soft Computing Techniques

w .~
3 -1 -1 -1
~k
:.~<.:
-1 3 3 -1
-1 3 3 -1 '....fJ,_
-1 -1 -1 3 ,.,;::
E = -10 -12 -10
Given input pattern for testing
xl = -1 1 -1 -1
SE = -2
By synchronous updation method
The net input calculated is
yin = -2 2 2 -2
The output calculated from net input is
y = -1 1 1 -1
SE = -12
The test pattern
xl = -1 1 -1 -1
is associated with
ans = -1 1 1 -1

7. Write a program ro implement Kohonen self~organizing feature maps for given inpm pattern using
learning rate as 0.6.
Source Code
clear all;
clc;
disp('Kohonen self organizing feature maps');
disp('The input patterns are');
x=[l 1 0 0; 0 0 0 l; 1 0 0 0 ; 0 0 1 1]
t=l;
alpha(t)=0.6;
e=l;
disp('Since we have 4 input pattern and cluster unit to be formed
is 2, the v1eight matrix is' ) ;
w=[0.2 0.8; 0.6 0.4; 0.5 0.7; 0.9 0.3]
disp( 'The learning rate of this epoch is');
alpha
while(e<=3)
i=l;
j=l;
k=l;
m=l;
disp( 'Epoch =');
e
while ( i<=4)
for j=1:2
ternp=O;
for k=l:4
temp= temp+ {(w(k,j)-x(i,k)) 2);
end
 19.7 Neural Network MATLAB Source Codes 651

~ D{j)=temp
k: end
if(D(l)<D{2))
J_ J=l;
else
J=2;
end
disp('The winning unit is ');
J
disp ('Weight updation');
for m=l:4
w(rn,J)=w(m,J) + (alpha(e) * (x(i,m)-w(m,J)));
end
w
i=i+l;
end
temp= alpha (e);
e=e+l;
alpha(e)=(0.5*temp);
%disp ('First Epoch completed') ;
%disp('Learning rate updated for second epoch');
alpha(e)
end

8. Write a MATLAB program to imp\emenr full counter propagation network for a given input pattern.

Source Code
clear all;
clc;
disp('FULL COUNTERPROPAGATION NETWORK');
x=[l 0 0 0];
y=[1 OJ;
alpha=0.4;
beta=0.3;
a=0.2;
b=O.l;
e=l;
v=\0.8 0.2; 0.8 0.2; 0.2 0.8; 0.2 0.8];
w=[O.S 0.5; 0.5 0.5];
t=[0.6 0.4 0.4 0.6);
u=[0.7 0.7];

while(e<=3)
m=l;
n=l;
for j=l:2
temp=O;
for k=l:4
temp= temp+ ((v{k,j)-xlk)l~ 2);
end
 ...,_...-~~
--'--1-
O I

652
MATLAB Environment for Soft Computing Techniques

for k=1:2
I
temp= temp+ ((w{k,j)-y(k)j' 2); I
end '
D(j)=ternp I
end I
if(D(l)<D(2))
J=l; I
else
J=2;
end
disp('The winning unit is ');
J
disp('Weight updatiOn');
for m=l:4
v(m,Jl=v(m,J) + (alpha(e) * (x(m)-v(m,J)));
end
v
for n=l:2
w(n,J)=w(n,J) + (beta(e) * (y(n)-w(n,J)));
end
w

temalpha=alpha{e);
tembeta=beta {e);
tema=a (e);
temb=b(e);
oe=e;
te(e)=e;
e=e+l;
te(e)=e;
t:el(oe)=oe;
alpha(e)=(O.S*temalpha);
alpha
beta(e)=(O.S*tembeta);
beta

disp('for Weight updation from cluster unit to output unit');

for m=l:4
v(m,J)=v(m,J) + (alpha(e) * (X(m)-v(m,J)));
end
v
for n=l:2
w(n,J)=w(n,J) + (beta(e) * (y(n)-w(n,J)));
end
w
for m=l:4
t(m)=t(m) + (b(oe) (x{m)-t(m)));
end
t
for n=l:2

~..L
19.7 Neural Network MATLAB Source Codes 653

u(n)=u(n) + (a(oe) * (y(n)-u(n)));

end
u

a(e)=(O.S*tema);
b(e)=10.5*temb);
a
b
end
tel(e)=e;
xl=te;
x2=tel;
yl=alpha;
y2=beta;
y3=a;
y4=b;
figure (1)
h=plot(x1 yl Xl y2,x2,y3,x2,y4)
1 1 1

set (h, {'Color'} r'; 'g'; 'b 'm'})

1 {
1 1
;

grid on
xlabel ('EPOCH' )
ylabel ( 'ERROR RATE' )
title('COUNTERPROPAGATION NETWORK')
legend(h, 'alpha', 'beta', 'a', 'b')
The error rate vcrsu~ epoch for full coumer propagation network is shown in F\gure 19-27.

.. ; . . . ... -
~.

Figure 1927 Epoch vs error r:ne for full coumer propagation nerwork.
 654
MAnAS Environment for Soft Computing Techniques

9. Implement a back propagation network for a given input pattern by a suirable MATLAB program.
Perform 3 epochs 9f operation.
Source Code
%back propagation network
clear all;
clc;
disp('Back prOpagation Network');
v=[0.7 -0.4;-0.2 0.3]
X=[O 1]
t= [1]
w=[O.S;O.l]
tl=O;
wb=-0.3
vb=[0.4 0. 6)
alpha=0.2S
e=l;
temp=O;
while (e<=3)
e
for i=l:2
for j=l:2
c
temp= temp+ (v (j, i) *x(j));
I' end
zin(i)=temp+vb(i);
templ=e (-zin(i)};
fz (i) =(1/ ( l+templ));
z(i)=fz{i);
fdz (i) =fz (i) * (1-fz (i));
temp=O;
end

for k=l
for j=l:2
temp=temp+z{j)*w(j,k);
end
yin(k)=temp+wb(k);
fy(k)=(l/(l+(e -yin(k))));
y [k) =fy (k)'
temp=O;
end

for k=l
fdy(k)=fy(k)*(l-fy(k));
delklkl =I t(kl -ylkl I fdy(kl,
end
for k=l
fa~ j=l:2
dw(j,k)=alpha*delk(k)*z(j);
end
-fl
~-

... 19.7 Neural Network MATLAB Source Codes 655

-I dwb(k)=alpha*delk(k);
end

for j=l:2
for k=l
delin(j)=delk(k)*w(j,k);
end
delj(j)=delin(j)*fdz(j);
end

for i=1:2
for j=l:2
dv(i,j)=alpha*delj(j)*x(i);
end
dvb(il=alpha*delj(i);
end

for k=l
for j=l:2
w(j,k)= w(j,k)+dw(j,k);
end
wb(k)=wb(k)+dwb(k);
end
w,wb

for i=1:2
for j=l:2
v(i,j)=v(i,j)+dv(i,j);
end
vb(i)=vb(i)+dvb(i);
end
v,vb
te(e)=e;
e=e+l;
end

' 10. Write a program to implement ART 1 network for clustering input vectors with vigilance parameter.

Source Code
clear all;
clc;
disp('Adative Resonance Theory Network 1');
L=2;
m=3;
n=4;
rho=0.4;
te=L/ (L-l+n);
te=te/2;
b=[te te te;te te te;te te te;te te tel
656 MAnAB Environmenl for Soft Compuling Techniques
---
~\7:

1"

t=ones(3,4)
s=[1 1 0 0;0 0 0 1;1 0 0 0;0 0 1 11
e=1;

while(e<=4)
temp=O;
for i=1:4
temp=ternp+s(e,i);
end
ns=ternp;
x(e,:)=s(e,:);
for i=1:3
ternp=O;
for j=1:4
ternp=temp+(x(e,j)*b(j,i));
end
yin(i)=temp;
end
j=1;
if (yin(j)>=yin(j+1)& yin(j)>=yin(j+2))
J=1;
elseif (yin(j+1)>=yin(j)&yin(j+l)>=yin(j+2))
J=2;
else
J=3;
end
J
for i=l:4
xl(i)=x(e,i)*t(J,i);
end
x1;
temp=O;
for i=1:4
temp=temp+x1(i);
end
nx=ternp;
m=nx/ns;
if (m<rho)
yin(J)=-yin(J);
j=1;
if (yin(j)>=yin(j+1l&yin(j)>=yin(j+2))
J=l;
elseif (yin(j+l)>=yin(j)&yin(j+1)>~yin(j+2)J
J=2;
else
J=3;
end
J
end
for i=1:4

I';"

"'""AJ.
 1!.

"I"j'l'
19.7 Neural Nelwork MATLAB Source Codes 657
il
j;,
temp=.O;
ill
ternp=L-1+nx;
lil
b(i,J)=(L*x1(i))/temp; ~1:
end
b
for i=1:4
t(J,i)=xl(i);
end
t
e=e+l;
end

Output

>> Adative Resonance Theory Network 1

b
0"2000 0.2000 0.2000
0.2000 0.2000 0.2000
0.2000 0.2000 0.2000
0.2000 0.2000 0.2000

t
1 1 1 1
1 1 1 1
1 1 1 1
s =
1 1 0 0
0 0 0 1
1 0 0 0
0 0 1 1
J = 1
b
0.6667 0.2000 0.2000
0.6667 0.2000 0.2000
0 0.2000 0.2000
0 0.2000 0.2000
t
1 1 0 0
1 1 1 1
1 1 1 1
J =2
b
0.6667 0 0.2000
0.6667 0 0 2000
0 0 0.2000
0 1.0000 0.2000
t
1 1 0 0
0 0 0 1
1 1 1 1
 ~
~
658 MATLAB Environment for Soft Computing Techniques

J = 1
b
1.0000 0 0.2000
0 0 0.2000
0 0 0.2000
0 1.0000 0.2000
t
1 0 0 0
0 0 0 1
1 1 1 1
J =2
b
1.0000 0 0.2000
0 0 0.2000
0 0 0.2000
0 1.0000 0.2000
t
1 0 0 0
0 0 0 1
1 1 1 1

11. Implement adaptive r~onance theory nerwork 2 for given inputs by a MATLAB program. Perform 2
iremions only.
Source Code
clear all;
clc;
disp('Adative Resonance Theory 2');
s=[O.B 0.6]
a=lO;
b=lO;
cooO.l;
d=0.9;
e=O;
rho=0.9;
theta=0.7;
wb=[7.0 7.0];
wt=[OO];
alpha=0.6;
it=l;
u=[O.O 0.0]
tem=O;
for i=l:2
tern=s(i) 2+tem;
end
n&=sqrt (tern);
p='[o oJ
for i=l:2
x(i)=s(i);
w(i)=s(i);
~~
~ I
659
19.7 Neural Network MATLAB Source Codes

q(i)=p(i);
end
X
w
q
temp=O;
templ=O;
for i=l:2
terop=w(i) 2+temp;
templ=p(i) 2+templ;
end
nw=sqrt (temp) ;
np=sqrt {templ J ;

for i=l:2
i f (x(i)>=theta)
fx=x(i);
else
fx=O;
end
i f (q(i)>=theta)
fq=q(i);
else
fq=O;
end
v(i)=fx+ (b*fq);
end
v

tem=O;
for i=l:2
tem=tem+v(i) 2;
end
nv=sqrt (tern) ;

disp('Updating Fl activation again');

for i=l:2
' ' u(i)=v{i);
w{i)=s{i)+(a*u{i));
p{i)=u(i);
end
u
w
p
tem=O;
temp=O;
for i=l:2
tem=tem+w(i) 2;
temp=temp+p(i) 2;
end
660
MATI.AB Environ~ent for Soft Computing Techniques

nw=sqrt {tern) ; .
np=sgrt (temp) ;
for -i=l:2
X(i)=w(i);
q{i)=p(i);
end
X
q
for i=l:2
if (x(i)>=theta)
fx.=x(i);
else
fx=O;
end
if (q(iJ>=theta)
fq=q(i);
else
fq=O; [
end I
v{i) =fx+ (b*fq);
end
v
disp('Computing signal to F2');
for i=1:2
t~p=O;

temp=temp+wb(i)*p(i);
y(i)=temp;
end
y
temp=O;
templ=O;
for i=l:2
temp=temp+v(i) 2;
templ=templ+u(i) 2;
end
nv=sqrt(ternp);
nu=sqrt ( ternpl);
for i=l:2
u (i) =v{i) i . )

p(i)=u(iJ+(d*wt(i));
end
u
p
ternp=O;
for i=l:2
ternp=ternp+p(i) 2;
end
np=sqrt (temp) ;
for i=l:2
r(i) = (u(i) +c*p(i)) I (e+nu+ (c*np));

.~
19.7 Neural Network MATI.AB Souroe Codes 661
lr;-i
1'
end ~
te.ntp=O;
for i=l:2
temp=r(il 2+temw;
end
nr=sqrt(temp);
i=l;
if {y(i)>=y(i+ll)
J=l;
else
J=2;
end
%Check for RESE~
i f (nr>=rho)
for i=1:2
w(i)=s(iLtairu(i);
v (i) =fx+l;l':"-fq;
ternp=o-;
tem=O;
for i=l:2
terop=temp'tw(i) 2;
tem=tem+p(~) 2;
end
nw=sqrt (temp) ;
np=sqrt(tem): .\
x(i)=w(i)/{e+nw); I
q{i)=p{i)/{e+np);
end
end
disp{'Update weights for 2 iterations');
while (it<=2)
wt(J)=(alpha*d*u(J)J+l+{alpha*d)*wt(J);
wb(J) = (alpha*d*u(J) )fl+ (alpha*d) *wb(J);
wt
wb
for i=l:2
u(i)=v(i),
p(i)=u(i)+d*wt(i);
w(i)=s(i)+a*u(i);
x(il=w(i);
q(i)=p{i);
v(i)=fx+b*fq;
end
it=it+l;
end
12. A perceprron neucaJ net uses a hard~ limit tcinsfer function. Plot this transfer funCtion.
Source Code
%Plot of bard limit transfer fUnction
X= -4:0.1:4;
 662
MATl.AB Environment for Soft Computing Techniques

y = hardlim(x);
plot(x,y)

Output

Figure 1928 Plot of perccptron hard limit transfer function.

13.
Create a perceptron network using the command "newp" and obrain irs performance.
Source Code

%Program to create a perceptron network using command 'newp

net= newp([ -2 2; -2 2], 1);
%No. of epochs is given as 4
net.trainParam.epochs = 4;
%Let define the input vectors and the target
p = I 12 ;2 I [1; -21 1-2 ; vector
21, 1-1; 11 I;
t=[0101l;
%The net can be train with
net = train(net, p, t) ;
%Finally simulate the trained network for each of the inputs.
a = sim(net,p)

~~
TRAINC, Epoch 0/4
TRAINC, Epoch 3/4
TRAINC, Performance goal met.
-~=
;11
m_:-
19.7 Neural Network MATlAB Source Codes 663

a=
0 1 0 1

Figure 1929 Training performance for a perceptron net.

14. Write a l'viATLAB program to create a feed forward network and perform Batch training
Source Code
%Program to create a feed forward network and perfrom batch training
%Create a training set of inputs p and targets t.
%For batch training, all of the input vectors are placed in one matrix.
p = (-1 -1 2 2;0 50 5);
t={-1-111);
%Create the feedforward network. The function minmax is used to
%determine the range of the inputs to be used in creating the network.
net=newff (mirunax (p) , (3, 1] , {' tansig' , purelin' } , 'traingd') ;
%Set training parameters.
net.trainParam.show = 50;
net.trainParam.lr = 0.05;
net.trainParam.epochs =
300;
net.trainParam.goal = le-5;
% Now train the network.
(net,tr)=train(net,p,t);
% The training record tr contains information about the progress
of training.
% Now the trained network can be simulated to obtain its response
to the inputs in the
% training set.
a = sim(net,p)
664 MATLAB EnViionment for Soft Computing Techniques

()uqput
>> TRAINGD, EpOch 0/300, MSE 0.69466/le-005, Gradient 2.29478/le-010
TRAINGD, Epoch 50/300, MSE 4.17837e-005/1e-005,
Gradient 0.00840093/le-010
TRAINGD, Epoch 68/300, MSE 9.35073e-006/le-005,
Gradient 0.0038652/le-010
TRAINGD, Performance goal met.
a
-1.0008 -0.9996 1.0053 0.9971

k
rt'
t:

Figure 1930 Training performance of a feed-forward network.

15. A radial basis nerwork is a nerwork with two layers. lr consisrs of a hidden layer of radial basis neurons
and an output layer of linear neurons. Plot a radial basis function.
Source Code
%Plot of radial basis function
clear all;
clc;
X = -5: .1:5;
y = radbas (x);
plot{x,y)

~,.~
 !~
665 i:
19.7 Neural Network MATLAB Source Codes
~.,,
Output ~
"

Figure 1931 Radial basis &merion.

16. Consider a surface described by z = sin(x)cos(r) defined on a square -3 S. x S. 3, -3 ~J S 3.

Plot the surface z as a function of x andy.
Design a neural network which will fit the data. You should study different alternatives and rest the
final result by studying the fitting error.

Source Code
%Generate data
X = -3:0.25:3
y = -3:0.25:3
z = sin(x)'*cos(y)
surf(x,y,zl
xlabel ('X axis');
ylabel ( 'Y axis');
zlabel('z axis');
title('surface z = sin(x)cos{y)');
%Store data in input matrix P and output vector T
P = [x;y];
T = z;
%Set small number of neurons in the first layer, say 25, 25 in
%the output.
%Initialize the network
666 MAllAB Environment for Soft Computing Techniques

net=newff([-3 3; -3 3), {25 25], {'tansig' 'purelin'},'trainlm');

%Apply Levenberg-Marquardt algorithm
%Define parameters
net.trainParam.show = 50;
net.trainParam.lr = 0.05;
net.trainParam.epochs = 300;
net.trainParam.goal = le-3;
%Train network
netl = train(net,P,T);
a= sim(netl,P);
surf(x,y,a)

Output
TRAINLM, Epoch 0/300, MSE 6.57445/0.001, Gradient 1010.2/le-010
TRAINLM, Epoch 4/300, MSE 0.000424834/0.001, Gradient 10.0448/le-010
TRAINLM, Performance goal met.

Figure 1932 Surface for z = sin(x) cos(y).

 667
19.'l Neural Network MATLAB Source Codes

Figure 1933 Training performanc,..

figure 1934 Surface of (x,y, a).

668 MATIAS Environment for Soft Computing Techniques

~
17. Find a neural networ~ model, which produces the same behavior as Van der Pol equation:
,
X+(x' -l)x+x= o
Solution: The given Van der Pol equation can be represented in state space form as

;,, =-'2(1-x{) -x,

X2 =X)

Here various initial functions can be used. On applying vector norations, the above state space form is
given as

x=f(x)

where

x= [:] and f(x) = ~~~::n = [-'2(!-~)-x,]

For building chis Vander Pol model, Simulink is used.
Source Code
The simulink model for Vander Pol equation is shown in Figure 19~35.

Product1

To workspace
Product

simot~t1

To workspace

Figure 1935 Simulink model for Vander Pol equation.

% Define the simulation parameters for Van der Pol equation

% The period of simulation: tfinal 15 seconds; =
tfinal = 15;
% Solve Van der Pol differential equation
[t,x]=sim('vandpol',tfinal);
% Plot the states as function of time
plot(t,x)
xlabel('time (sees)');
ylabel('xl ~d x2 -states');
title('Van D~ Pol Equation');

~
 669 l1
19.9 Fuzzy Logic MATLAB Source Codes ,,
grid
%Plot of training vectors
p = t';
T =X';
plot(P,T, '+');
title{'Training Vectors');
xlabel('InPut vector P');
ylabel {''Target Vector T' ) ;
% Define the learning algorithm parameters- a feed forward
network chosen
net=newff([O 20], [10,2], {'tansig','purelin'),'trainlm');
%Define parameters
net.trainParam.show = 100;
net.trainParam.lr =
0.05;
net.trainParam.epochs = 500;
net.trainParam.goal = le-3;
%Train network
netl =train(net, P, T);

Output
TRAINLM, Epoch 0/500, MSE 6.91369/0.001, Gradient 408.177/le-010
TRAINLM, Epoch 100/500, MSE 0.0208639/0.001, Gradient 0.112283/1e-010
TRAINLM, Epoch 200/500, MSE 0.0208277/0.001, Gradient 0.00613187/1e-010
TRAINLM, Epoch 300/500, MSE 0.0208226/0.001, Gradient 0.0603704/le-010
TRAINLM, Epoch 400/500, MSE 0.0208181/0.001, Gradient 1.62252/le-010
TRAINLM I
EPoch
-~----
500/~00. MSE 0.0208168/0.001, Gradient 0.0403124/le-010
TRAINLM. Maximum epoch reached, performance goal was not met.

The states of the Van der Pol equation are plotted as function of time as shown in Figure 19~36.
The training vectors are shown in Figure 19~37.
The convergence has not occurred (performance goal not met), since network structure is simple. .A5
a result, by modifying its structure, perform further iterations to achieve the performance goal.
Figure 19-38 shows the rtaining performance.

119.8 Fuzzy Logic MATLAB Source Codes

1. Write a MATLAB program to implemem fuz:z.y set operation and properties
Source Code
%Program for fuzzy set with properties and operations
clear all;
clc;
disp('Fuzzy set with properties and operation');
a=[O 1 0.5 0.4 0.6];
b=[O 0.5 0.7 0.8 0.4];
c=[0.3 0.9 0.2 0 1];
phi=[O 0 0 0 0];
disp( 'Union of a and b');
670 MATLAB Environment for Soft Computing Techniques

Figure 1936 Plot of states vs dme (Vander Pol equation).

++++

+
+ + +
+ +
+ + + ++_,.
++
+ + +
+ + +
+ + +
+ + +
'f
+ + + +
+ + +
+ + +
+
++++ +++++
+

Figu~ 1~-37 Plot of training vectors.

 671
19.8 Fuzzy Logic MAll.AB Source Codes

Figure 19-38 Training performance (goal not mer).

au=max(a,b)
disp('lntersection of a and b');
iab=min(a,bl
disp('Union of band a');
bu=max(b,a)
if (au==bu)
disp('Cornmutative law is satisfied');
else
disp('Commutative law is not satisfied');
end
disp( 'Union of band c');
cu=max(b,c)
disp('a U (b U c)');
acu=max."(a, cu)
disp('(a U b) U c)');
auc=rnax(au,c)
if (acu==auc)
disp('Associative law is satisifed');
else
disp{'Associative law is not satisfied');
end
disp( 'intersection of b and c') ;
ibc=min(b,c)
disp('a U (b I c)');

i
j,_;;
I~
 ;:~~:-

672
MATLAB Environment for Soft Computing Techniques

dls=max{a, ibc).
disp('Union of a and c');
uac=rnax{a,c)
disp (' (a U b) I (a u c)');
drs=min(au,uac)
i f (dls==drs)
disp('Distributive law is satisfied');
else P._

disp('distributive law is not satisfied'); i5.

end
disp('a U a'),
idl=rnax{a,a)
a
if (idl==a)
disp('Idempotency law is satisfied');
else
disp('Idempotency law is not satsified');
end
disp('a U phi');
idtl=max(a,phi)
a
if (idtl==a)
disp('Identity law is satisfied');
else
disp('Identity law is not satisfied');
end
disp('a I phi');
idtl=min(a,phi)
phi
i f (idtl==phi)
disp('Identity law is satisfied');
else
disp('Identity law is not satisfied');
end
disp('Complement of (a I b)');
for i=-1:5
ciab(i)=-1-iab(i);
end
ciab
disp('Complement of a');
for i=l:S
ca._(i)=l-a(i);
end
ca
disp('Complement of b');
for i=l:S
cb(i) =1-b(i);
end
cb
disp('a Complement U b Complient');

A_
19.8 Fuzzy Logic MAn.AB Source Codes 673

drnl=max(ca,cb)
i f (drnl==ciab)
disp{'Demorgans law is satisfied');
else
disp{'Demorgans law is not satisfied');
end
disp('Complement of complement of a');
for i=1:5
cca{i) =1-ca(i);
end
cca
a
if (a==cca)
disp('Involution law is satisified');
else
disp('Involution law is not satisfied');
end
2. Write a program to implement composition of Fuzzy and Crisp relations
Source Code
%program for composition on Fuzzy and Crisp relations
clear all;
clc;
disp('Composition on Crisp relation');
a=[0.2 0.6]
b=[0.3 0.5]
c=[0.6 0.7)
for i=l:2
r(i)=a(i)*b(i);
s(i)=b(i) c(i);
end
r
s
irs=min(r,s)
disp('Crisp- Composition of rands using max-min composition');
crs=max(irs)
for i=l:2
prs(i) =r(il s (i) ,
end
prs
disp('Crisp- Composition of rands using max-product composition');
mprs=max(prs)
disp('Fuzzy Composition');
firs=rnin(r,s)
disp('Fuzzy- Composition of rands using max-min composition');
frs=max(firs)
for i=l:2
fprs(i)=r(i)*s(i);
end
fprs
674 MATLAB Environment for Soft Computing Techniques

disp('Fuzzy- Composition of rands using max-product composition');

fmprs=max(fprsl
3. Consider the following fuzzy sets

1 0.4 0.6 0.3)

A= [ -+-+-+-
2 3 4 5
0.3 0.2 0.6 0.5)
B= [ -+-+-+-
2 3 4 5
Calculate, AU B, An B, A, l3 by a MATLAB program.
Source Code
%Program to find union,. intersection and complement of fuzzys sets
% Enter the two Fuzzy sets
u=input('enter the first fuzzy set A');
v=input('enter the second fuzzy set B');
disp('Union of A and 8');
w=max(u,v)
disp{'Intersection of A and B');
p=min(u,v)
[m) =size (u);
disp ('Complement of A' ) ;
ql=ones(m)-u
[n] =size (V);
disp('Complement of B');
q2=ones(n)-v
Output
enter the first fuzzy set All 0.4 0.6 0.3]
enter the second fuzzy set 8[0.3 0.2 0.6 0.5]
Union of A and B
w 0
1.0000 0.4000 0.6000 0.5000
Intersection of A and B
p 0

0.3000 0.2000 0.6000 0.3000

Complement of A
q1 ::
0 0.6000 0.4000 0.7000
Complement of B
q2 ::
0.7000 0.8000 0.4000 0.5000
4. Find wherher the following relation is a tolerance relation or not by writing a MATLAB file.

11 11 00 00 0]0
R= 0 0 I 0 0
[00 00 00 11 11
 675
19.8 Fuzzy Logic MAT_!..A;S Source Codes

Sow-ce'Code
% Program to check whether the given relation is tolerance relation
or not
p=input{'enter the relation')
sum=O;
suml=O;
[m, n) =size (p);
if (m==nl
for i=l:m
if(p(1,1)==p(i,i))
else
fprintf(' the given relation is irrelexive and ');
sum1=1;
break;
end
end
if(suml ..... = 1)
fprintf('the given relation is reflexive and');
end
for i=1:m
for j=l:n
if(p(i, j )==p(j, i))
else
fprintf('not symmetry hence ');
sum=l;
break;
end
end
i f (sum==1)
break;
end
end
if (sum-=1)
fprintf{'symmetry hence');
end
end
if (surn1 ..... =1)
if (sum-=1)
fprintf('the given relation tolerance relation');
else
fprintf(' the given relation is not tolerance relation');
end
else
fprintf(' the given relation is not tolerance relation');
end

Output
et:ter the relation[l 1 0 0 0;1 1 0 0 0;0 0 1 0 0;0 0 0 1 1;0 0 0 1 1]
676 MATLAB Environmenl for Soft Computing Techniques

p
1 1 0 0 0
1 1 0 0 0
0 0 1 0 0
0 0 0 1 1
0 0 0 1 1

The given relation is reflexive and symmetry h!l'nce the given relation is a tolerance relation.
5. To find whether the following rdation is equivalence or not using a MATLA.B program.

) 0.87 0
0.87 .1 0.46 0.13 0.35]
0 0.98
R= 0. 0.46 I 0 0
[ 0.13 0 0 1 0.54
0.24 0.98 0 0.54 1
Source Code
%Program to check whether the given relation
%is an Equivalence relation or not
p=input('enter the matrix')
sum=O;
suml=O;
sum2=0;
sum3=0;
[m, n] =size (p);
l=m;
i f (m==n)
for i=l:m
if(p(l,l)==p(i,i))
else
fprintf(' the given relation is irreflexive ');
suml=l;
break;
end
end
if{suml -= 1)
fprintf(' the given relation is reflexive');
end
m;
n;
[m,n]=size(p)
for i=l:m
for j=l:n
if(p(i,j)==p(j,i})
else
fprintf(' , not symmetrY');
sum=l;
break;
end
end
19.8 Fuzzy Logic MATLAB Source Codes
677

if(sum==l)
break;
end
end
if(swn--=1)
fprintf(' ,symmetry');
end
for i=1:m
for j=1:n
for k=l:-1:1
larnbda1=p ( i, j) ;..
larnbda2=p ( j, k~ ;
lambda3=p(i,kl;
q=min(lambda1,lambda2);
if(lambda3 >= q)
else
surn2=1;
break;
end
end
end
end
if (surn2 "'= 1)
fprintf(' and transitivity hence '); !:]
else
fprintf(' and not transitivity hence');
end
if(surn1--=1)
if (surn"'=ll
if (surn2-=ll
fprintf{' the given relation is eqUivalence relation');
else
fprintf{'the given relation is not eqUivalence relation');
end
else
fprintf('not eqUivalence relation');
end
else
fprintf('not eqUivalence relation');
end
end
Output
enter the rnatrixti 0.87 0 0.13 0.35;0.87 1 0.46 0 0.98; 0 0.46 1 0 0;
0.13 0 0 1 0.54;0.24 0.98 0 0.54 1]
p
1.0000 0.8700 0 0.1300 0.3500
0.8700 1. 0000 0.4600 0 0.9800
0 0.4600 1. 0000 0 0
0.1300 0 0 1.0000 0.5400
0.2400 0.9800 0 0.5400 1. 0000
678 MATI.AB Environment for Soft Computing Techn~ues

The given rdation is reaexive, not symmetry and not transitivity and hence not an equivalence relation.
6. Find the fu22y relation using fuzzy max-min method for the following using MATLAB program:

0.2 0.3 0.4] 0.1 I ]

R ~ 0.3 0.5 0.7 and S ~ 0.4 0.2
[ I 0.8 0.6 [ 0.3 0.7

%Program to find a relation using Max-Min Composition

%enter the two vectors whose relation is to be find
R=input('enter the first vector')
S=input ('enter the second vector')
% find the size of two vectors
[m, n) =size (R)
[x,y]=size(S)
if (n==x)
for i=l:m
for j=l:y
c=R(i.:)
d=S {:, j)
f=d"
%find the minimum of two vectors
q=min(c,fl
%find the maximum of two vectors
h(i, j)=max(q);
end
end
%print the result
display('the fuzzy relation between two vectors is');
display(h)
else
display('The fuzzy relation cannot be find')
end

Output
enter the first vector[0.2 0.3 0.4;0.3 0.5 0.7;1 0.8 0.6]
R
0.2000 0.3000 0.4000
0.3000 0.5000 0.7000
1.0000 0.8000 0.6000
enter the second vector[0.1 1;0.4 0.2;0.3 0.7]
s
0.1000 1.0000
0.4000 0.2000
0.3000 0.7000
ans
the fuzzy relation between two vectors is
h
0.3000 0.4000
0.4000 0.7000
0.4000 1.0000
 679
19.8 Fuzzy Logic MAll.AB Source Codes

7. Use MATLAB commands ro display the triangular and Gaussian membership funaion. Given x = 0
to 10 with increment ofO.l. Triangular membership function is deHned between [56 71 and Gaussian
funaion is defined between 2 and 4.

Source Code
%Program to depict membership functions
x::(O:O.l:lO)';
y1=gaussmf{x, {2 4]);
%Plot of Gaussian membership function
plot(x,yl)
hold
%Plot of Triangular membership function
y2=trimf (x, [5 6 7));
plot(x,y2)

Output

Figure 19-39 Gaussian and triangular membership functions.

1
8. Find the fuzzy relation between rwo vectors RandS using max-product method by a M.ATI.AB program.

0.2 0.3 0.41 [0.1 I

R ~ 0.3 0.5 0.7 md 5 ~ 0.4 0.2
[ I O.B 0.6 0.3 0.7
 _;;

c': I!
680
MATl.AB Environment for Soft Computing Techniques

Source Code
%Program to f'ind a relation using Max-Product Composition
%enter the two input vectors
R=input('enter the first vector')
S=input{'enter the second vector')
%find the size of the two vector
[m,n]:::size(R);
[x,y]=size(S);
if (n==x)
for i=l:m
for j=l :y
c=R(i,:);
d=SI', j);
[f,g]=size(c);
[h,qJ=size(d);
%finding Product
for l=l:g
e(l,l)=c (1.1) *d(l,l);
end
%finding maximum
t(i,j)=max(e);
end
end
disp('Max-product composition relation is');
disp(t)
else
display( 'Cannot find relation using max product composition');
end

9. Using MATLAB program find the cnsp lambda cur set relations for lambda = 0.6. The fuzzy matrix is
given by

0/0.6 0.8
0.7 0.4
1
02]
IS~
[o 0.6
0.1 0.5
0.5
0.9
Source code
%Lambda Cut method of defuzzification
% Enter the given relational matrix
R~input('Enter the relational matrix')
% Enter the lambda value
lambda=input('enter the lambda value')
[m, n] =size (R);
for i=l :m
for j=l:n
if(R(i,j)<lambda)
b(i,j)=O;
 681
19.9 Genetic Algorithm MATLAB Source Codes

else
b(i,j)=1;
end
end
end
% output value
display('the crisp value is')
displB;y(bl
Output
Enter the relational matrix[O.l 0.6 0.8 1;1 0.7 0.4 0.2;0 0.6 1 0.5;
0.1 0.5 1 0.9]
R
0.1000 0.6000 0.8000 1. 0000
1. 0000 0.7000 0.4000 0.2000
0 0.6000 1. 0000 0.5000
0.1000 0.5000 1.0000 0.9000
enter the lambda value 0.6
lambda =
0.6000
ans
the crisp value is
b
0 1 1 1
1 1 0 0
0 1 1 0
0 0 1 1

119.9 Genetic Algorithm MATLAB Source Codes

1. Write a MATI.AB program for maximizingf(x) = :l using GA. where xis ranges from 0 to 31. Perform
5 iterations only.
Steps involved
Step 1: Generate initial four populations of binary srring with 5 bits length.
Step 2: Calculate corresponding x and fitness value J(x) =:?.
Step 3: Use the tournament selection method to generate new four populations.
Step 4: Apply crossover operator to the new four populations and generate new populations.
Step 5: Apply mutation operator for each population.
Step 6: Repeat the steps 2-5 for 5 iterations.
Step 7: Finally prim the result.

Source Code
%program for Genetic algorithm to maximize the function f (x) =xsquare
clear all;
682
MATLAB Environment for Soft Cnrr.puting Techniques
clc;
%x ranges from 0 to 31 2power5 = 32
%five bits are enough to represent x in binary representation
n=input('Enter no. of population in each iteration');
nit=input('Enter no. of iterations');
%Generate the initial population
[oldchrom]=initbp{n,S)
%The popultion in binary is converted to integer
FieldD=[5;0;31;0;0;1;1)
for i=l:nit
phen=bindecod(oldchrom,FieldD,3);
of the binary population % phen gives the integer value
%obtain fitness value
sqx=phen. ~ 2;
sumsqx=sum ( sqx) ;
avsqx=sumsgx/n;
hsqx=rnax(sqx);
pselect=sqx./surnsqx;
sumpselect=sum(pselect);
avpselect=sumpselect/n;
hpselect=rnax(pselect);
%apply roulette wheel selection
FitnV=sqx;
Nsel=4;
newchrix=selrws{FitnV, Nsel);
newchrom=oldchrom(newchrix, :);
%Perform Crossover
crossrate=l;
newchromc=recsp(newchrom,crossrate); %new population after crossover
%Perform mutation
vlub=O: 31;
mutrate=O.OOl;
newchrormn=mutrandbin (ne~lchromc, vlub, 0. 001); 'lmew population
after mutation
disp('For iteration');
i
disp('Population');
oldchrom
disp( 'X' 1;
phen
disp( 'f(X) 'J;
sqx
oldchrom=newchromm;
end

Output

Enter no. of population jn each iteration4

Enter no. of iterations5
At the end of fifth iteration, the output i~
 19.9 Genetic Algorithm MATI.AB Source Codes 683

For iteration
i =
5
Population
oldchrom =
0 0 0 0 1
0 0 0 1 0
0 1 1 0 0
0 1 0 0 0
X
phen
1
2
12
8
f lXI
sqx
1
4
144
64

2. Use Gatool and minimize the quadratic equation j(x) =? + 3x + 2 within th~ range -6 ::: x::: 0.
Function Definition
Defme rhe given function f(x) = :l + 3x + 2 in a separate m-file as shown in Figure 19-40.

%funccion ~o minimize
function z=qudr;atic (Xi
z= (x1;x+3~x+2);

I
I
Figure 19~40 M-61e showing defined quadratic function.
I
I Creation of Gatool
On ryping "gatool" in the command prompt, the GA molbox opens. In rool, for f1mess value cype
@qudraric and mention the number of variables defined in the function. Select best fitness in plot and
specify the other parameters as shown in Figure 19-41.
 fFl=
-I

684 MATlAB Environment for Soft Computing Techniques i

Figure 19-41 Genetic algorirhm tool for quadratic equation.

Output
The ourput showing the best fitness for SO generations is shown in Figure 19-42.
The status and results for this functions for 50 generations are shown in Figure 19-43.
3. Create a Gatool to maximize the functionj(xl,x2):::: 4x, + 5X2 within the range 1-l.l.
Function Definition
Define the given functionj(xl,x2) == 4xt + 5X2 in a separate m-file as shown in Figure 19-44.
Creation of Garool
On typing "garool" in rhe command prompt, rhe GA toolbox opens. In roo!, for fitness value type
@twofunc and mention the number of variables defined in the function. Select best fitness and best
individual in plm and specify the other parameters as shown in Figure 19-45.
Output
The ourpur for 50 generations is as shown in Figure 19-46. The output also shows the best inidvidual.
The status and result for this function are shown in Figure 19-47.
4. UseGatoolandminimize thefunctionf(xl,x2, x3) = -5 sin(xl) sin(x2) sin(x3)+f- sin(5xl) sin(Sx2) sin(x3)],
whereO :::.xi ::S.pi,for I :5. i::S. 3.
Function Definition
Define the given function
f(xl,x2, x3) = -5 sin(xl) sin(x2) sin(x3) + [- sin(5xl) sin(5x2) sin(x3)]
in a separate m~file ~ shoWn in Figure 19-48.

..
 --- I
~~~9-=9~G~e~n~et~ic~A~Ig~o~ri~th~m~M~A~JLA~~B~S~o~"~~e~Co~d~~-------------------------------------- 685

..
.. .. . ..
...
...................................:

Figure 1942 Output response (best fitness).

Figure 1943 StatuS and resuhs.

686 MATLAB Environment for Soft Computing Techniques J
funecion 2~tuofunc(x)

=t4xTn+sxt2l l:

Figure 1944 M-file showing defined function.

Figure 1945 Genetic algorithm tool for given function.

Creation of Gatool
On typing "gamol" in the command prompt, rhe GA toolbox opens. In tool, for fimess value cype @sinefn
and memion the munber of variables defined in the function. Select best fitness in plot and specifY the
other parameters as shown in Figure 19-49.
Output
The output for 100 generations is as shown in Figure 19-50.
The status and result for this function are shown in Figure 19-51.
J
I
.
'
19.9 Genetic Algorithm MATU.B Source Codes
687

oO
...:::.:::::
....... .
I I

...
*Zs

Figure 1946 Output response (besr fimess and best individual).

4-

Figure 1947 Status and results.

688 MATI.AB Environment for Soft Computing Techniques
-1--
--

%Function to minimize the given function

tunction z~sinefn(x)
(~(Ssin(x(1))'sin(x(2))sin(x(3)))+
j(-(o1n(Sx(1))sin(Sx(2))sin(x(3)))))

Figure 19~48 M-fi1e showing defined sine function.

"'

Figure 1949 Genetic algorithm tool for sine equation.

 689
19.9 Genetic Algorithm MATLAB Source Codes

Figure 1950 Ourpm response (besr fitness).

Figure 1951 Statll'i and results.

 690 MATLAB Environment for Soft Computing Techniques

119.10 Summary
In this chapter soft computing techniques are implememed using MATLAB software. MATLAB software is
very user-friendly and enables the user to simulate the ideas of soft compucing for their applications. This
chapter provides an overview of the various commands and GUI module involved in MATLAB for neural
networks, fuzzy logic and GA approaches. The source codes developed using these commands and GUI
toolbox for the soft computing techniques have also been included for the ready reference of the reader.

I 19.11 Exercise Problems

1. Implement the AND funcrion using perceprron 10. Minimize Ra.migin's function using MATI.AB
network using a MATIAB program. GUI GA toolbox.
2. Write a MATLAB program to apply back 11, Given a polynomial equation of the form f(x) =
propagation network for a pattern recognition 4x4 + 3~ + 2J?. + x + 7, find its roots using
problem. GA approach.
3. Implement OR function with bipolar inputs 12. Consider a hyperbolic tangent function. Max-
and targets with a NADALINE neural net. imize it within the range 0<x<22/7. Apply
4. Write a program to create an ART 1 network to two-point crossover and tournament selection
cluster 7 input units and 3 cluster units. process. Construct a GA GUI toolbox.

~
I,
I
5. Develop a Kohonen self-organizing feature map
for a image recognition problem.
13. Find the roots of rhe quadratic equation using
genetic algorithm. The quadratic equation is
II., J(x) = 6x' + 5x+ 3.
6. Write a program to implement various opera-
I tions of fuzzy sets. 14. Find the solution of the function J(x) =
I
sin(7rr x) + 10 with rhe consuaim -3 < x < 3
7. Implement rhe properties of fuzzy sets using an
by using genetic algorithm and MATLAB pro-
m-file.
gramming.
8. Develop an m-file to perform compositional
15. Write a program ro minimize "cosine" function.
opemions in fuzzy relations.
9. Maximize Rosenbrock's function using a
MATIAB program.
 f
=
eX;

,~
Bibliography
r~
I.e;.
l~~{
;;;
1. Pal, S. K {1998, January-April) Soft computing tools and pattern recognition. JETE Journal of RLsearch,
44(1-2), 61-87.
2. Rao, D. H. (1998, July-Ocrober) Fuzzy neur.ll networks. IETE],urnal ,J&>w-ch, 44(4-5), 227-236.
3. Russo, M. {1998, August) FuGeNeSys- A fwzy generic neural system for full}' modeling. IEEE Tranracrionr
'"Fuzzy SJ"""' 6(3), 373-388.
4. Pal, S. K. and Srimani, P. K. {1996) Neurocompuring motivation, models, and hybridization. Proceedings of
IEEE Cl)mpuurs, 2~28.
5. Jain, A. K. and Mao, J. {1996, March) Artificial neural networks: A tutorial. Proceedings ofIEEE Computers,
31-54.
6. Lippmann, R P. {1987, April) An inuoducrion ro computing with neural nets. IEEEASSP Magazine, 4-22.
7. Hush, D. R. and Horne, B. G. {1993, January) Progress in supervised NN. IEEE Signal ProuSJing Magazine,
8-32.
8. Kohonen, T. (1990, September) The self organizing map. Proceedings ofIEEE, 78(9), 1464-1478.
9. Kangas, J. A., Kohonen, T. K and Laaksonen, J. T. (1990, March) Variants of self-organizing maps. IEEE
Tramactionr o11 Ntural Networks, 1(1), 93-99.
10. Pal, N. R., Bezdek, J. C. and Tsao, E. C-K {1993, July) Generalized clustering networks and Kohonen's
self-organizing scheme. IEEE Transnc#ons on Neural Networks, 4(4), 549-557.
11. Carpenter, G. A. and Grossberg, S. (1992, September) A self-organizing neural nenvork for supervised
learning, recognition, and prediction- can neural networks learn m recognize new objects without forgetting
familiar ones? IEEE Commrmicarion Magazine, 38-49.
12. Carpenter, G. A., Grossberg, S. and Rosen, D. B. {1991) Fuzzy ART: fast stable learning and categorization
of analog patterns by an adaptive resonance system. Neural Networks, 4, 759-771.
13. Frank, T., Kraiss, K-F. and Kuhlcn, T. (1998, May) Compar:uiveanalysisoffuzzy ART andART-2A, ncnvork
clustering performance. IEEE Transactions on Neural Networks, 9(3), 544-559.
14. Fernandez-Delgado, M. and Ameneiro, S. B. (1998, January) ~T: A multichannel ART-based neural
network. IEEE Transactions on Neural Networks, 9{1), 139-150.
." 15. Seriono, R. and Liu, H. (1996, March) Symbolic represenrarion of neural networks. IEEE Comp11ter,
71-76.
16. Shang, Y. and Wah, B. W. (1996, March) Glo~al optimization for neural network training. IEEE Computer.
17. Jain, A. K. and t>4ao, J. (1996, March) ANN: A rutorial. IEEE Computa-.
18. Ng, K and Lippmann, R. P. (1990, May) A Comparative Study ofthe PracticalCharactuisticsofNeural.Network
and Conventional Pattmz Cla.ssifim, Masters Thesis, Deparunent of Electrical Engineering and Computer
Science, MassachusettS lnsci.rute of Technology.
19. Bortoian, G., Degani, R. and Williams, J. L. (1991) Neural networks for ECG cla.ssificadon. IEEE Ntural
Networla, 269-272.
20. Geczy, P. and Usui, S. (1997) Learning performance measures for MLP nerWorks. IEEE.l]CNN, 1845-1849.