as digyimi funchon payameteyc

Naive gayes assumption)

(disc re n uo(k})

Gotxsg€anNaive Baya .
Smooåhing4ap Jace

LineaY Reoyecion OLS

f)nalyl-iccxl
ienb Descen b
Probabilishc
Cogs41C Regression % GneYa±ive model •Toiot likelibood
Discrimmnt model
likelihood
Reye-Bien

gyacheDJc descefr lc-

updat€ Yuleg aye
GeDeYaIgd ltneay mdels same .
Exponenffal o? dig-Wibation
Sufficienl 6kdis&icS•
SofEmox Reor$i0D GILD)
Kane) melhods I Tyick
Cineay non-I?neaY is also
no-Erequired
—y type loill encuye
Kernel {Y10ck
n--da[Q poiDE ex-tsåenc-Qoe , foy
Kernel mab{ix mercevS-Ihm
2M)
1-
31 31 mercers Kernel
CSymßEbic positive
semi—definite)
 (I—AS)= -2-(I-asJWA 4mnebicW.

is

Line-ay Classifier
6epaTa{inq hypeyplane

Assumphon: Obsdvahong QTe separable

Predichon: sloo
CT-F ic noYma\ hupeyplane»
Hen equahon of hypeyplane
is -6
diskanc-c obgeYV—
hypeyplane shoulÅ
moodmised.
 f(ĚÄ+ po)

-funclion maYgin [Lihe confldef)ce)

define — min

=> objec&ive max Mťłnfi

Ô objecJcive max
po,e

—-f(2Po
blxl obJec±iVcfunck
chaoaeg)

IIAI

11211
Ě ( Xi-3i
Ilŕ)l
IIÔI) =ô

d—tnin
rmx q 11+11-1
ffN8iÛ
ensuyeskhal
is genmekYic
fnN9i n.

Wii addilioml consWaınE

Η I aiVendat-9

min s+

maX—maY0iYl claSSİfieT OY
sup paye vecbv clasifiev R)i+h
hayd
poinks on mayon aye
Suuppoyl vec-lpys. nok on
maygif)» Iheq cofrlYiblAEe
choîSe CF hypeYplane
Suppoyl veclo« clasçif(ey-
This is optimal claşifveY
Sepyakinq hg pey pime clasgiÇfCY
mox fhaygîD clasgiFjey hard margifl
@ maxH q
]Jğ2/b
Û Tüll min4 llwllL
yegcaled SLACh

QTI -Fbz )
-ı
Theorem The above 407mulntions aye equivalent.

equivalent of -Foymulahon@'
max

MMX max

miD-l
S•t• g.) -bbl)

Relaxin khe lineally separable assumption

boo much Viola

—fioo is al lowe

max—maY91Dclass wi&b son maY9jD

min
c-cos{ para

yi I—Éi y f)

4euDobseyvabons
--5 no viola-hoocwell sepaya-ted
+ coyongSide of rynYqiD/bu±on coyyeclside
coront side of
of byperplane
ÉpL
bypeyplarr
 c=ooEHayd-rrayon
loto—valueof c
Hi9b-va(ue 0C C -ò NarYouYY
less-viohtlon
cause oveYfitFrg
Loidey mayqin, less generalisa+ioo
more-violoo.ee4ion *
lead-to Undeyfit.
Quich recap cons4yaìned o
pyobletf\
min -Ê(Q)
tfon

GleneYQl
Md Lagvarvian
f(Q) hiCu)

Considey ep(u) = max k ( Q, A/ e)

••sa{isftes
co

O
lf L) : feosble pJc Esa{jsfies
Solu±ion-b ep(L7)'is game as
coDsEya-lDIop pyoble;
min
 Gerwic Congl-yain-b problem

prîmal min ț(w)

—or) SR. 9i(L)) = min 9(Q)
pyoblem

General ised [agrar,giar)

Lagăange mul-łipliers
max e)
-Țeasible Convex opti
ep@)— fa)
o-lbeyajłse blen
min min max dual forma
lałi0T)
ConcHeT,.eo( = min
max (Ă/ e ma..Xmin L

e c, e) min p

E max min min max

Mîn H Q)

kkT condiłîons : seł necessart*condi±ions

OpłÎmal musł
be holds

-P easîbl )

Ș gI complemenăry Shckncg
 gi Convex
( bf inkycepo

-men ptimal dual optimal soluåons Cist and

min

Objective : To dual
max 9 a) = max mif)
g,
i=
(2)

RecoyileO usin @ q @ enbal minimaloptim-æab)

GMQ)
Yj d i

2Q(i Yi =O

. Convex 40 _ ?
L * generic dual
KhT cordi16crg ø-goY
coovex .
SQ±hc$ed

Solvedud
-J using @
 are on rnaYoir%

= —M ax Xi
i;gri

-Sivgn( <Zi/J> +
non-żeyodi *CoYYegpond guppoY± vecłors.
Non-lîneaY -5 (ce keynelfuncłien
Łbe fot Sofł—roayqjn
, n} c(Rd
cohîch
unkrówn.
/appyox•
choice oȚ loss funcŁi0D (deperds OD obj)
IȚISE ( — łbe frainîng eyyoy)
Ehôin•łm-łsinł
Głereval(tQhoD anseendQłQ P(X./Y)
E ExpecŁedmsE)
obj-> m)njmiSîngbaining eYYOY
1--5alco Min imigin$ oeneYalisabd) eyyoy
E vav
 一 f31
E「

bias= E囹 一
 Co sitio de-
BiaS—VaYnnce
EXECłed -Iesł eyroy

fCÂY
(E (ȚCŁ))—
VaY(Ț(ț)) +
—(ŃoiSe)+ @iaŕ(ȚII))
hon much olkY
model vaYies?

loco hiqh
VaYiancc vaWiance
locobias

high bias

quadvałîc
6
l-łneavrnael
6
-ș hiob ŁYaîDin8loc peysis&
undeYȚi
Hing
model îs veyg Cimple ancl
nol expres î ve enovgh Yeflecł relati hips among
vavîąbles.
Trcreasing daąaseł(ęîE no-[help-Pul
Cîmpley -3 Comp&x li kel(d FEIp.
High degYee Țlgnomial
--3fił -O sparious patterns

îrryease
 5mp\e7nobel

large bias : lack expYessiVi-l% (no mat\ey OF

data)
mit i qoåon make model complex
keep move relations, btw fea{-uY6
(Tncreasina datagtte dogno-€ help much)
lack of or tooexprßive rndel
o increastno
+ Simpler model Chiqh bios, low vaD
meaSRYe

norms o?
RegudQY'ltQ-h•oo -to contYol bios—VAYiOJÄQ{Yodeoff

ea e
gbYiDl
ReguÄritahon c-lyengtb
+ I-t— Ridge
regression
= Il WIIo f- _= # of nonR.eyoin la
allan-
—a-hVeJ C= Wil)
reg
Chg-go
Cleas& absolutte sbfinkaqe
Selection
ReylaYi2a-h-00
the lecæuY%toe discused
high bias i S cavsed b! lack OP model'
exerci €1 9 Complex model-
On Ehe handJ Complexmodel {end to give high
vQViance, artl coeshould look simpler model -to
fake coxe high Vaynnce
 gives Yîse łYade-frFF blo bîasąl variance .
RegulaYitQl-ionîs one-toag corrłțol bias-vafiancę
ŁYade off.
oeraîc -Țoymulałîon, RegulaYiž•ed -RxnCłÎOn has

min R(u)
voheye T(e) : novmal loss AncŁion (ex: MSF)
R(L) : RegulariteY
S-lYenoŁh
common examples•
I I Alfr q- RĐklQYlăbn
RO) II12110 OF non-żeyoes în g)
Ag kerdg {o make ceveyal voj's2eYo,
cboîce khis has -lendencg
ModelSpaysey(ag nill be EYo q on(yTeu.)non-t@fo
(Di will be pyesenĐ
Jôouoevey, main pyobleml? iŁš noe dŕPPeyenŁiQbIę.
*T-nslead ased

Regulayižafron/ Ridge Recesion

coYTesgp0DdinĄ Los-fincJo-on

Ț(e) (s logg
TRO) =
Tn nOłafion,

We aYe looking

min
 -AnalHkjco.( SolU±ion
Takino a deyivakve U
C uTxTxw +4T4
¯öl.o
rJTx12 : sca lar
Foy min

Reroayk:One canalso {o offain the help of

gradienl decerÆ met-hd
Compare (,oyidqe 1.0i4hanalgåjcal SollkhDß oe least
Cquaye Megression
=CxTx)
COLSmay noe be esbn-u±ed poorly es-Gma-tedwhen
x TX (S singu(aY OY reaYlg sin€ulaY. hoc,JeveY,-frat is
{be cag wridqe
-
{Yade-oPf in Ridge Regression
Bias—VaYiarce
balances the {Yode off bliA) bias vayiance
--3 A increases- bias incfeaEs f, vaYiaræ decreases
J
leading -b Ampler mode(
coil-h f, yeoulayization term gel-s
impoY±anceq hence{he fiys-[ kym may not have
flexibi {o increoc VallkeSof . offeYwiS€/
ID!(l make hiqheY.

SirT\PeY rnoAeI VedacecL mael

 mveyælg,as A vaYiance increases
complexmdel 3cYeased model
We U)

Töb(ewoy
Eyroy/bia;/ T Vayiance
VaYiance

Bias
model
comp
ty) in EYms of A
>Toen( eTYor
Bias
eyyof
bios/
VaYianæ

Variqnce

aye minT(L) -VA

Il

4—Ridff reqyession
Lasso
Ridge
vayiance
Ac A OF bias •incveages,
simpler mode) T—-n
conveysely, as A J, vayiance 't
 (X ±A.ä I toil
nxd Crnalyixncka60t)
is no-bdefined at O
2nd±evm
* Need+0 al&eyna-bvemejchodobey
+han analytical coluå-ion qyadienl descen±• t'
* uses Q called
min C WI LOi_l
LhSD
Solving univayiale
obeys are -hxed
OPH pyoblem
SVM» SMOC Sequential MinimiSa-bo%)op-bmita-bbj
descene•
R jdqe Regression
q: IDhlALhSSO resulJs in eslimats
aye exac-uy teYo, Ridge is unable to
do so ?
LPrSSO* also booYls as Vayiab\e SeleckoY

Ridge
dose

LhSSo
min

lines (came 4 n vaue)

CO I +1.,02__
 I (Dil+lU)21 SS

—-5901k) aye To choose bl(D above

me-buds
O model Selection --3A CValtAÖ
@ model asessmenE Geneya)itabion eYYOY
MNING TEST
bm---h-

('o f-—es4imaEed øeneyou(itation

eYY6Y .

TRB(NINGI
CET Dato

tvlæel Improvement
Wedichon accuracy
r modelInleypre{abilit-u
Non-lineaY variaLle -fransfoymation
Kernel -brick
Link function in Ccan be
trfroduce nonÅineayi@J
Assume Iineav is ctnfFcierrbzlow bias
 sanpl$/l # OF variab16 pyed€cbs
p ( based on theiY re layonship,
variance can be high I Iou)
S low varianceC4endercD
p {enderEy-Jø ovevfikard hioh variance
Solvåion ..-3 t.oags
Subsev
@ shytnkqoe me-khod selec-kior)

11m do e ßeffessi on
Ll LhSSO Re8Te-8sion
( some ae closeb &YO,
hencecan be removed)
Jdon -b Iotoey dimensiDj.
. CpYöjec
subsu selection
Best
glec-hon X e (RP/ subsetsjte k
l. Null mæel/ -z: sample for a Oven
1-3no mean
pred i cåoYS.
2. Foy k=lx
@ -Fil- ( Bbdelg conbinlnq k pyedlc-brs
@ Selec+ / call
(t) models Mk
best RSS C Residual sum og Sq(AQYe)

es&ima-æon a sel independen&-promæyainint

data
of features.
f Simpler all psi bleCombiocUions
vomp1k±Qt-ioy-nllb expensive.
4 -YerdenQd
Of hÜh vaYlÄhCc.
S&epuoiseSelection
ebepwise Se lechon
O Null model MO: ujl±b no predictor
@ for . , pH
@ consideyall p-k models,bb addinq one new
predic±ov in Mk
@ choose aypnq models call M kH
on -LYatntnØdata
@ bese amono MO, M),
once added, cooril be dropped models aye
go,Tf Somecombinabonof cornpaved
features cAYe but since (leg— Compu&a-
Eon)
noE addd not
used
fill combinabons aye no-I
conSideYd
ßackwaYd Stepcoig selection
O MP•full model all predicbYs
@ foy k- p, pH,.
@ considerall k models Containingall buf one of
+e predictoyclfeaturcsl vaYS in for tota( of
pyedic±ols
@ bes-l among k models tvlk-l
@ c hoose M M
öTrÅependen4set (validation Seb k direct estimate
@ TrBivec+ es4imaåe
0+ eyYöT {hyoccah adjusbn-
wenb b -khe Waininq eYYOY.
mal IoW' Cp SlatiS±iCS
g Akaike tnfoyma&ion
C gayestoninformation OTC)
p n ( RSS4
corvespn dingto # of pyedicbs
in model
 ee6rnake
pc-Heynative
test eyYoY)

indivec Jc es-hmaJce-+va\/dev
cp
Cmkaike inf cri4eyion)
Blk ( Bayesian inf.

1-->irÜlYecJc ecJc
{egl- eyroy
fourued on
information 9)eoYY
kL—divev e c Compare how pyobabiIt-by
p -from a pyobabili+y

I-b es%mates {he vela-kve amount loss-

AIC = 2k --2 In (C) k + of- payame&evs
C---> maximised Value of like-Il
-Func&ior)
Regress jon model pyobabili±fc mdel
YéqyessionCGaugsiao osgxmpebon)
equiva len b -
n €Q_C Cup±o consD
Cp ATC if) {his gcerayjo
bic
L-bderived from bay,es-nn peyspec±ive
blc kin
qmpls
payam eEevs
Bic higbevpeoaft(d
Dfmensiorulill.åReductionCpyincipal component analyst)
Capture maximum
VaYiancc in data.
d

I.oilh menn Q
%mpkcovariant

+tssume, low-dimensional Yepvesen-tgtion

OYffonorma-.( ma-bYix
biT bJ -O •i*J
Obiec&ive PYOjec+ da±A C IR d
V —Rd (dim (
• pyojecæ da±ovX] in V
06
n€yma-fior) Los
max varianc perspective— gpYeod
CVaYi arc)
sequenhal appyoach
I-blækföY yec-bT e Rd -that
data.
maximieegHoe vaYianc of pyodecked
BTXi

—o

maximise V)/ ±hen many choices 4 [Dould

come) can be made afbi±YaYiIy layqe-—
hen cc making VI layqe
cnax vayiancz
4Yans(akd naX
CoyyeSponding Lograngian

— 1--11

: eioen of-S
Giqen-value decompcibon : coyyeBpoDd(n6
eco—value
Aemnxn

Define p d (anal
co•PchA-i entMis

hi is elder)value of
g, b, aye coyyegponding eigen—
VeC-bys
0—0

6ymmelflCma4Yix can be diagornlised

| blT

chooSe ei8en-value
hi t e-vec{oy
Varance perspeckive

projec&ionof Xi on V
Y BTXi
a..2i

var V -L

max VI moX

layqec-l e gen-
—Vaue Al
bi : coyregpondirn
pol eigen-vec-lvo
coff)poenE
S peelYaL7beovem
symne&ic ma4Yjx A has -Che fack-ofl
Yeal A= •
e-vec-(-oys Q
Akbgpaa mox vayiancg
Aggumc•.We have m-l pyincipaA Componentcs as
e-vec±bYs associated larges Jc
e-value.
TAA
OBS.
: PYoiec{-îon
ma-ËYix
pyojec+if)g onto
the %kbspaQ
by
 : asgœìakd coi±b
layaesE e—valueoe
C Sek of
Of S Cl afe iden±ica)

layqesjc oF q./
eìaer—value OB S.
B —ym —eig6) vec-œys
OF S ave
ASSocia4ed
{QYae.SJc
e—vaUeS •
Relaijve variance e.21QlDedvay;ana-:
captured
2Ài (Anexpla,nd vayiance•.

Xi erkdy erkd minimiz-e I lXi-X•llV

Auto-encodeY —y

OVkh0h0Yrnal basic
bi
= Cibi / Ci C R

= ajibj C V
 miniMite avg- Te COYEbcxc+i0D eyTöY

0--0Ydı•natüti L bagic vecl-oy

PCA: OF avg. yecons-lyuc!on eYYOY
G IRA coyyegponding oylbönoymal

Zi Ş- çj-bj , 9 e R
J-ı
zcjdj

diM(U) Bo

Re mçEYuc+-iöf) eYYOY
40Y
TMz%9vg II

z)
gj/Ej EllXi-%ll L - I

D7-i
(zı-7j)T
DX
Daji

Daji
(li-13 blkJ
-z (Zi%j ¯aji) —0
aji

pYojech6nof
 -qâbdJJȚ)Ă

OCLGęosxo

ô&-CuE
-0 ąono.k
SWogpO.CL
płoăechon

in *o-be pyincip-.l
SlkbSpaQ
Je.2CbĐpyircipd subSFncĐ

C Symm&ic Makyix Yank mReach Tank L)

Findg bes& yank M appyoxîrną€ion

of
i ĄenŁiły małYiX.

Młl
j=
 bjTXiXITbj
n i j=cni-l

sna(le-ge d -M
of S
*Min variance o-P data pRecJed
oy%oaonal œmplement OF ffe principal-
componen+ •
Key Sleps in pcfr..

Cro makL ry\ean

8=0)
@ : Divide by 64-d de viation
-b mahe vayiance
Of CwarianQ rmtYiX
fiaen Êecomp051tìon