UNIT-I

NEURAL NETWORKS - II
(Bnck Puobogalien Netoouk)
PocehtKons : Jhe netiosk consiat os a dut o
dursouy wnii tat consili the inhut
tidden Jayus os tompulalion nodes
mae
computalion node.
and an sutbut Jaye o he netosk in
aignal
he inthutdinclion uspagaliü though Ihue NNs
-touoand la - by - layu basis.
mullilayu heeiptrns CLPS).
Hutlilayeu eublions taue bwn asaliud ducetully da
sslue oma dillicult and divose þuoblems by Jiaining
hubuisd mann with a
them in a
Rnon as the wL back - huotagaion algsithm.
lgauithm kaAd on the eua - cauielion ianing
tule.
back- puopagalion Lanning emaisi y lio þas
Ewar
dhuaugh the dittont Jayus the netask:
jouwand ass
-a baekwand pass.
aclnity þattiun (input wecla)
In the touoand þass, unodes ! the neteuk, and ii
dmbay
ib aplud do the by Jaye.
etet ubagalis thsugh he netwak ayu
o/es is busdued as the aclial sbone
finally, a dut o weighli y the netaks ane
thu yw. the synaplie
all
naplie
the backioaud pass, the sy e coeighli
Couwfon sle.
Bning accodamee wth the -
adiuatid = duind olp - acliat o/p
ignal
 uspagalid backiand thisujh
asigmal is then connciou
the netwo, against theL dineeion o4 sy nablic
ten the name eaL back ob agalion. Ge
Back - þuGpagali on algouithm. the auning fuoCIas
pyoumud wth the aloihon çs called back -þuotagalion
Juawning. dislinelie chavalüsies:
st mullilaye fueklion tas thuee
Un the
utosk ineluds
each
each mewm
4 he moolul
a nonlina aclivalion tunc, But the nom-linail
smooth (ie ditounlablu)
siq moidal nonlineanily clinad ay loqulie
tune:

ocal teld
whe U; is inducd om.
y. b
dayus a tidden
the nutwk conlains e
2 olp the
onb thot aue not at the Ip
menons
enons enable. the netwsk
natw ut. thea iddn
Jäsks by exliacli ng uogLssively
Jo Jaun comple
meaningu yealis m the yp tattiuns.
exhibis a high degus of connecli iiy
3. hhe mutisak synapsu o the netosk. dchauge
delimi nud y the netuok cuguies chaugl
in the conmeciuiy q the thu weigkti.
dynaplie connuelions o
çu the papulallom of
 2-3

he back- þuopagalion algothm uoidus computalionally

e4ficient mthod ou the saining mullilan hunenblKons.
>funelin signals
(Mull la <- E sig nals
Puuslon)

1 funelion &ignals t tunll on aigmal is an

that come n at the Ip eud the
netook , uopagals ooad thugh the netoscK, and
at the olp eud oq the netosk as an o/ p ignal.
emugu
is eswmd Jo hufoum a wsnful tune atat th

op the nyw
each Lon, the signal is calulalsd es
a tune t|ps and alsocialid wi
o the I|Ps

2. EO
ELL Sig nals i- sn signal auiginalis at
al a

o|P newon oy the yw , ard þuss agals

backwaud thsuaugh the w ye g
4 the etwk iuelues an e a -clezendert
tune im ansthu.
iweues the computaln an ealimal the
yaadient veclau (quadieni a the eau cuyae t wt)
 24
Back- Paskag alien sgoithm
at the o<p y nHn at itaion

is an olp ode.

Inatantansus value n) o th total ergy

al the olp Jay:
(2)
jEC
Sat C çnelude al the nnons

the ole la
N duno the toal mo 4 palins conained in
dt
Set.
the Jaaining
Avoage sguand
Gn) (3)
N

Cout tunelin as Juaning

tuyoumanel.
tee paKamelid
Aaning
we conai d dimple methsd aaining
shich the Oe updalid on a patlin'- by
þatlun basis umtu on eboch.
 2.·5

djln)
Wioln) =b;(n)

-4

tig.; which debicls on i.

i
Conaide the aboue
Jhe indueed Jocal tiuld ’ (4)

telal no. inls (ucdi-g

wtee m us the

he tunelion signal
’ (5)

Backtnapagalion algouithm applis

rspoulicnal o the
þaulial duivalie n)dwiln).
chain sule caleulas :
Aeesding
’(6)

dansl vit tacla, duliuining the

di soclion duaneh in wt. shaee tau the cynale
 bth dides ay eq (2) wit. b e, (r)
9et :
’ (7)

-Ayoun alng both sids o es (4) w-t.

oat 4.(n) we get
Je; n) z -1 13(8)

euniating e (s) wt. ;(n),t get

(9)

sitenaing es (4) wat. wn)

’ (10)

< egs () J5 (1o) in eg (6) we get

Caueecliom Awj; (n) applid do wji n) is dui ned

the d a e:

wte sali baMameli

Auawning
ea, (42) accounls jae gtadi ent
desent in wight shac.
 Aeonding'y julda
’(3)

loeal guadient s; (n) = - ofln

yhe dscal guadient oins 5 uquind changs

synaplie weighli.
sdceeding o ea, (14), the lscal quadint &;(n) toe
Olp nton Jo the pusduct the cauebonding
Signal e, (n) oL that won eud th deiva live
P cn)) 4 the associalid aevali on tene
from e (13) #(14) we noi that wigkt odjustment
debunds sigual ej (n).
Two Cass comes
us desending ubon wtheL in te
nor i scalid

Case t : Nenon
aboue munlond case.

tidden Nodle :
Case 2: Nueer
lscalid in a hiddern day
netwk, thee s
sAeilied duiud uibonie
th
 28

deconding tot
scal qadient 6; (n) n) a t;(n)

-3G(n) ñ;(ycn), ’45)

mUwon j s hidden

h in seond in ue hawe usd ea

To calcdat the þauial duivaliue ) aycn).

may pusLid ollous:
faom tu
Ein) = ecn)
K Ec en egz) e
’(16) hae usd ke

k.
the allal
Net we
the
odeivalie ecny Qnd Heusli ea )
4;n)
eguivalunt tom Deetn) ) ’U8)

we sli that

ey n) = deln)- Yn)
)
node.
 ttenc
’ (20)

’ (2)
whe Tolal no I<ps opplind ds
K.

the synaplie weight wo (n) is equal

egat
tuu agin, cost
neunon k, and the
the bias b (n) abhlud Jo
he rale tL.
-onding Ijp ts tixad att

’(22)

By ing e 2), (2.2)

K

’(23)

Fuom e 04)

Finally ey (23) 4 U5)

we
back-prspagaion
local radient ój (n) as:

k hi dden.
 t ’ 1
)
al
+
N2
N3
N
Jayu kiodun
wih
XORosl
by Soue
cdass
() 4o)(I, Co,), (0,0), sg. unik
poblu XoR Cordidu we
hyucse pt Each
ykucube. unit the painscbsiying
in
case shscial
atpuoblm
þatun) oxablu Jinsuly
sep Non Puotlum
( XOR
L0 2,
 2:||

bË : -3
2.

the deeision bounday conaliuelid lay

-1. o,)
his

(o,o)

nunon 3:

w3 -2

has n eneitaláy Ctiue)

the bottom hËdd
wha
Conneclon a e ojp nuon )

inhibiay (-ie) conelion t

a songu
Op n o n .
wtn I|p (o,o) ’ o/P Hewas .
bLeaue the inhbi lby
luge -ie wt
conneed

w IlP Co,) ,o)

+i ne wweigh
betom adew weuOn.

Top wiclelu
 212

Vauioli ens o< ostandakd Backbuotagalion yuttm

1. ecenental Ialion Poscadune
slahut descent method , the mini mum
In coenlional
iss achiewed by vaying the Vaxiable as
-t.

imbuovemunt, the Vaue ay 'K

wthun a sas weals no coninud.
edued aud the ocess is
adso, shen impusuement is exci ed
in BPN he comliwd
Similaudy
duease
ese, te ainiug an
coeiins and momndim
diHuent set y dauning
uut
factes, Juthu denating the
ea.
netwah eachs a laliau
the the
Usually e jne, deadi at dis ag aud
and t
Thaini ug can
dishenad witl sadueed
contnd wit puious weigkli ueiug
Hhn
omanliim jaeliu and Jaunig coicient.
2 Adaplie Backhuabagalon (Aealualid Launing)
he
Rackbspagalion (ABp) à duessud
ddasliue
dlenondu et al. (1994) aud sliya. et al. (1992) laei
by
duelebmunt a a fault detulion aul ideutitialion syi
oseus tombasd dlineet cusunt wsto., a
jal a
)
aud associalid bii ng sylm.
tHhe

Jn the ABP
choen
daawnig algo, the no
uch hat
igut pdali
tunclien is foeed
Jo behawe culaiu mannw tuat acelealis conegece
 2-13

In Jtd BPN:
Jheu) - w(old)- 1

AGP algo"e
D(re) - (ald) Ple) (2E)

E’Tetal e e .
sale
PcE) = LE
jazla

Sinee PlE) = 1E

styouthm Baud Boukhstagalon

3 Geeneie

piueiple
oual wutalio

:
Ueomes the
falman's
puobem BPN.
 2-14
weigkt updati wethad:
E, + Awot. AV 2E. +
Enr
AwAV +
2.

Quick bop's wt ubdale pxoeduke dahunds

obnoi nalioy
wt aoduee alavely itle ttut
quadiut sbuved aat stuothu w
each wt. u
2 he
quadalie
(scally
5 Augmenlid BPGP Netweks:
std. BPN wite augmntid
Jhe aehilelie is tha
netueks de dogauittmic aud extporsnial
LLMOnS addad E nwl etwauk's p ud olp laes.
the uineiple augming the no is:
Lersiiue w
ighly
the bouidauy doain , tdy ailibaing
conlicfon oceunate wafhing wodll's
baundauy dau P
Jw demalis each vaiable wit mulliple
2.
aguly iulûaelie
tune's on iddu nons
ailsuing
 2-15

Bi
Ci

A
C
idd
D

achiliie s tuat a std BPN

HowewL dogaithnue 4 entonulial
added to L|P 4 o/P layes
in tthe LP lay eis a
nwon
he agonitmie ug
nalial ogauithmie ams Jaumaon g e cassondi
L]p wae a e liaini ug dala
A; = n (1115 X;
+l543)
ie nalnal
mewOns
layus enponnt caesbonding
L)Pvalue
jauwalion a the
enponente Jians
iaiuiug dal:
-l313
Bi O-854 exb (xi)
This Liansjoullo
bepoial.
is shou y:
-o5

-05
 216.

C; = Ln (18 Y; +1)
D : exs (0. 6931 Y; -1)
D:

igly dunsile
laqaithuie
Swall Yalue
Enbonnt nuon
rale donw
faciliali
Bhis Leaie ca ws bounday daai.
acat aepig
Ro
In BPN,

X’ s)P vauabl.
La ABPN
Y: Yx, A,8)) B enponut

Neal
puatod e wtod . Ju s mhod
Zhaug
tioldu aaldid at a e. It slál
4 Jnc. Sswall.
 2-17

Ejet Tuming Pouamalis the Bockpnohaguli on

Newal Netwak

-A bus deleclion iking bauamalis ssh

menlim Jacto, saning collicient, dig mai dal gain
an theshold value sauid a ei ci ent daning
aud digni ng a ostable the wts should nl
momtum 'u that rious changs n
nt diuti on o moemnt in weigr on tha momenlem
adyutment is made basd
weiglt
methed.
the momenlim has oe in decidiug
digmii ant usdueeHapid
the vales Laning suali that wl
dauni ng
biass.
lacla s zuo Suethening is mini
mini mm
H momnlüm eom the newly
aciusment comus
aud th enlie wt.
caleulalid chaug adjualment is
ignaud aud he þewiss aak1alid.
asjusfwut is Suostlud
wt momenlum
awsnt facláu

without aeillalions.
Jadig
cottiiet dasends
alue
o. dagur
i/ each pattun consideud
Jeawning coidu is eblainud
 A18

-4 te daning codici ent is asg. >os.

sall, <o2,
aL chawgd duasically . 14 thu dauing s
chauged in sualL iuewins, us causing
the syim do conieige m

Leauning ali
esy

q
8s

0.9 1

nale
Laning
)6UD
talix
ea
45

35
36
2S
Momenim lacai
n20
e Aali

O9

Momnlim facla
GUD
80 LIualf ons wAUs
x
70
ali

e 30
20

2S. 80 125 |70 260 250

TIualins

Sigmoidal qain
fune is selaelid, )p - op talionhip
I siqmsidal
he neuo

CI+ecre)
wh
sig mai dal gaiu
neen abacls Two cali naulon.
Onalog
Jhe vae af sigmaidal qain ao ayeels back husagalion.
Imbou combinaliou of e Scaliug facl, Liaiug
uali aud momenlim joclá wigt
aud þersu cogone.
as the scaliug yaclai is
To ger taded oP womunlim Jalo haue lo
inwrasd, Jsaniug ale
Aeet osilalioS.
 ab .,
Thueshod ale
ün the above eq,. is commonly calad as thhold
value the bias L thu moi Jaets
o gonnalis an o|p 4 the weighlid
Sum sy the Ie eids the thushd value.
a dmallL yalue
One methsd is da dimaly aaaign
Ond do not
chauge
method s
sujuing.
o inilialy chose some saudom
he sthe
chauge thum duuig iaining.
and chaug
values s and
Esay
alt
xio
14

)3 Hiddun nods
I2.

6 20
Hicdolu Nedes

vauisus þanamelos in BPN

Seledlion o
No tidden Nedes minimum nods
ciliom is to select the
guiding
imbaiu the metosk hyoemanee
which o not ould the weighs
dumand a ataing the
that the memoy
be Rekt i imum.
H M- 1
deannisg
f alemns in
Sat.
 Momenlim Coellicient d
- Momonlim Jaclou enhanes the Jiaining uscess. h
wylnee he momun lim wighr chauge is stoun
in tig: whue

[aw1
(weight chag i tout
womntum)

(Momnlim lim)

Momnlim also h et lseal minima.

momunm Jsm wl ain cay
chauge u u thusuk
duo gobal minimal

bualical sos
loal Hinima ; the
invelues the inlioduelion a a "shock"
4.

spueilic a aandom amounls.

wtich chgs all weighs by
hen te wsut aei cal
this fail, aaiing all u .
all
aud lat the
sunandomig the weights
(n)
deauning Co yiientngaliue. woud chauge
annet be (as it
- Ir

wst weclio uiu

I
( nyw is wnstable4
he yoR oblcm

(onnt cla

ko,)

(1

þatte
Paten

We cawwnst depate o'patewt n I

Pateni
 7

beis b,

Each
He hts Hodet,

ae

t|

-o.

=0, X |
wi2+|
w
0)Xg= O,
 D()t to ()-0S

NNeon
(AND) ( oR) At Newn 3
-)(os)
( ) ol-2) - o()

(o s)
() o(-2) + L()-1
1

I(-2) +I() - (0-s)

(w)

S<o

,(,)
becss bounday newn
Canotuele
(9-)
Deainin deunday
Conatu
,6),

Cowle
(o,e) (1,)
 TKoann
*This Aechaigne
1982
oM's are
o

.Onol tained

-liennnae
ver
eti-inewionel data t
uhic lo eamy

&on's hae tuo

Rnd''
here are'n' nit n the
wn the
 Soyaini Ayoita

Quitialiaten
lnilel wee
th

that a
neeeo t Can dae Caleueli

Euclielean dis lauee

Jhe i/fneclei
each ) | m

) Ne aleulelimi- nd the neoey

weelo

bleilie he ig
 Mplei

w2:( o 3, oS, o7, o-1, \-o)

net with two
krieney dey wngi
Coide a

O3

Dj)- (oix
Drs)t(o3-o)
2

D() +(o7-)+
2 (o:S- 2

(I-0)t(o9-o-s)
=
D(i)= -34 2

D(2) o:3-o) (o
 pdauan?

,,(oca ) t (x,- o(ead) )

w Cnew)
wsI lne)
Neus
(-1)xko:3)
Vo
nel Caloulate
the
lapr -fer
2,
t}
ueln nelunn Act
e ()2|te -
o3, - Vo Vi,
0-y,0-S
[o6, o Vsy Vi,, are
fuum -tetinlon oleldijei
Vo
(2)
1-
W
finel
Ne How
do
 Aetinlnf l n
1(ini) |-e
|te

-| 2
-Zin -
|-c
oS32

() aleulele he
for

4ctinelm,
|-e - 0 | | 22

S,o-la3) (o.yq33) =0-S41)

 D'2 x0Sy1| o1343.

y
Compe

Siny

Sa(Zn 2)

o-o2y
2

Vo
 -O23-oo6y1

V9 (ne)

wlne)

3264
Vo lne) o
So9
Vog (ew) = o