N

M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
Professor

ee ls
rin an
g d
Ap
p
Dr. M.K. Bhuyan

lic
at
io
ns
Indian Institute of Technology Guwahati, India.
Department of Electronics and Electrical Engineering,
-- Fundamentals and Applications
Machine Learning and Deep Learning

Introduction to Machine Learning

 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 






N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
Speech recognition

lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
Machine Perception

at E l am
i ng en
in ta
ee ls
Fingerprint identification

rin an
g d
Ap
p lic
at
io
DNA sequence identification

n s
Biomedical Signal Processing
Biomedical Image Processing
OCR (Optical Character Recognition)
Build a machine that can recognize patterns:
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
appropriate output.

ns
Computer Vision

the same way that human vision does and then provide
enabling computers to see, identify and process images in
Computer Vision is a field of computer science that works on
 Image Analysis (Computer Vision)
 Fundamental Steps
 Preprocessing (Image enhancement and Image
reconstruction)

ns
io
at
p lic
 Segmentation (Isolating objects )

Ap
d
an
ls
ta
 Feature Extraction (Representation and description)

en
am

g
rin
nd

ee
Fu

in
 Classification (Object recognition)

--

ng
ng

at E
ni

l
ah ca
ar

i
i
Le

uw tr
G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep

gy E
itu f al an a

lo nd
st t o m g ific
X1

no s a
In en Ka nin ert

ch ic
an m s ar C
di rt a e e
X2

In epa an e L lin
D r. M hin On
D ac EL
M PT
XN
N

Image Feature
Preprocessing & Classification
Segmentation
Extraction

Input Object Feature

Image Image Vector Object type
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 Examples of applications

• Handwritten: sorting letters by postal code,

ns
io
input device for PDA‘s.

at
p lic
Ap
• Printed texts: reading machines for blind

d
• Optical Character

an
ls
ta
people, digitalization of text documents.

en
am

g
rin
nd

ee
Fu
Recognition (OCR)

in
--
• Face recognition, verification, retrieval.

ng
ng

at E
ni

l
ah ca
ar

i
i
Le

uw tr
• Finger prints recognition.

G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep

gy E
itu f al an a

lo nd
st t o m g ific

no s a
In en Ka nin ert
• Biometrics • Speech recognition.

ch ic
an m s ar C
di rt a e e
In epa an e L lin
D r. M hin On
D ac EL
M PT
N

• Medical diagnosis: X-Ray, ECG analysis.

• Diagnostic systems • Machine diagnostics.

• Automated Target Recognition (ATR).

• Military applications
• Image segmentation and analysis (recognition
from aerial or satelite photographs).
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
Components of PR system

ns
io
at
lic
Sensors and Feature Class

p
Ap
Pattern Classifier

d
an
preprocessing extraction assignment

ls
ta
en
am

g
rin
nd

ee
Fu

in
--

ng
ng

at E
ni

l
ah ca
ar
Learning algorithm

i
Teacher

i
Le

uw tr
G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep

gy E
itu f al an a

lo nd
st t o m g ific

no s a
In en Ka nin ert

ch ic
an m s ar C
di rt a e e
In epa an e L lin
D r. M hin On
• Sensors and preprocessing. D ac EL
M PT
N

• A feature extraction aims to create discriminative features good for classification.

• A classifier.
• A teacher provides information about hidden state -- supervised learning.
• A learning algorithm sets PR from training examples.
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
Classifier
A classifier partitions feature space X into class-labeled regions such that

ns
io
at
X = X 1 ∪ X 2 ∪  ∪ X |Y | and X 1 ∩ X 2 ∩  ∩ X |Y | = {0}

p lic
Ap
d
an
ls
ta
en
am

g
rin
nd

ee
Fu
X1

in
--

ng
ng

at E
X1

ni
X3

l
ah ca
ar
X1

i
i
Le

uw tr
G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep

gy E
itu f al an a

lo nd
st t o m g ific

no s a
In en Ka nin ert

ch ic
an m s ar C
X2
di rt a e e
In epa an e L lin
D r. M hin On
D ac EL
M PT

X2
N

X3

The classification consists of determining to which region a feature vector x belongs to.
Borders between decision boundaries are called decision regions.
 Representation of classifier

ns
A classifier is typically represented as a set of discriminant functions

io
at
p lic
gi (x) : X → ℜ, i =1, ,| c |

Ap
d
an
ls
ta
gi (x) > g j (x) ∀j ≠ i

en
am
The classifier assigns a feature vector x to the i-the class if

g
rin
nd

ee
Fu

in
--

ng
ng

at E
ni

l
g1 (x)

ah ca
ar

i
i
Le

uw tr
G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep

gy E
itu f al an a

lo nd
st t o m g ific

no s a
In en Ka nin ert

ch ic
an m s ar C
di rt a e e
In epa an e L lin
g 2 ( x)
D r. M hin On
x D ac EL
M PT
N
max y
Feature vector  Class identifier

g |c | ( x )

Discriminant function
 Bayesian Decision Theory

ns
• Design classifiers to make decisions subject to

io
at
p lic
Ap
minimizing an expected ”risk”.

d
an
ls
ta
en
am

g
– The simplest risk is the classification error (i.e.,

rin
nd

ee
Fu

in
--

ng
assuming that misclassification costs are equal).

ng

at E
ni

l
ah ca
ar

i
i
Le

uw tr
G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep

gy E
– When misclassification costs are not equal, the risk

itu f al an a

lo nd
st t o m g ific

no s a
In en Ka nin ert

ch ic
an m s ar C
can include the cost associated with different
di rt a e e
In epa an e L lin
D r. M hin On
D ac EL
misclassifications.
M PT
N
 Terminology
• State of nature ω (class label):

ns
io
at
– e.g., ω1 for sea bass, ω2 for salmon

p lic
Ap
d
an
ls
ta
en
am

g
rin
nd
• Probabilities P(ω1) and P(ω2) (priors):

ee
Fu

in
--

ng
ng

at E
ni

l
ah ca
– e.g., prior knowledge of how likely is to get a sea bass

ar

i
i
Le

uw tr
G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep

gy E
itu f al an a
or a salmon

lo nd
st t o m g ific

no s a
In en Ka nin ert

ch ic
an m s ar C
di rt a e e
In epa an e L lin
D r. M hin On
D ac EL
M PT
N

• Probability density function p(x) (evidence):

– e.g., how frequently we will measure a pattern with
feature value x (e.g., x corresponds to lightness)
 Decision Rule Using Conditional
Probabilities
• Using Bayes’ rule:

ns
io
at
p lic
p ( x / ω j ) P (ω j ) likelihood × prior

Ap
d
an
P (ω j / x) =
=

ls
ta
en
am

g
p( x) evidence

rin
nd

ee
Fu

in
--

ng
ng

at E
2

where p( x) = ∑ p( x / ω j ) P(ω j )

ni

l
ah ca
ar

i
i
Le

uw tr
(i.e., scale factor – sum of probs = 1)

G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep

gy E
itu f al an a

lo nd
st t o m g ific
j =1

no s a
In en Ka nin ert

ch ic
an m s ar C
di rt a e e
In epa an e L lin
D r. M hin On
Decide ω1 if P(ω1 /x) > P(ω2 /x); otherwise decide ω2
D ac EL
M PT
N

or
Decide ω1 if p(x/ω1)P(ω1)>p(x/ω2)P(ω2); otherwise decide ω2
or
Decide ω1 if p(x/ω1)/p(x/ω2) >P(ω2)/P(ω1) ; otherwise decide ω2
likelihood ratio threshold
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 Types of Learning
Supervised: Learning with a labeled training set
Example: email classification with already labeled emails

ns
io
at
Unsupervised: Discover patterns in unlabeled data

p lic
Ap
d
Example: cluster similar documents based on text

an
ls
ta
en
am

g
rin
nd
Reinforcement learning: learn to act based on feedback/reward

ee
Fu

in
--

ng
Example: learn to play Go, reward: win or lose

ng

at E
ni

l
ah ca
ar

i
i
Le

uw tr
G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep

gy E
itu f al an a

lo nd
st t o m g ific

no s a
In en Ka nin ert

ch ic
an m s ar C
di rt a e e
In epa an e L lin
D r. M hin On
class A
D ac EL
M PT
N

class B

Classification Regression Clustering

Anomaly Detection
Sequence labeling
…
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 Species
An Example

N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
Salmon

in ta
ee ls
rin an
Sea bass

g d
Ap
p lic
at
io
ns
“Sorting incoming Fish on a conveyor,
according to species using optical sensing”
 






etc…
Width

N
Length

M PT
D ac EL
D r. M hin On
Lightness

In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
Problem Analysis

ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g
Position of the mouth

d
Ap
plic
at
io
n s
Number and shape of fins

= set of features to use in our classifier!

Take sample images … to extract features
Preprocessing
 Use a segmentation operation to isolate

ns
io
at
lic
fish from one another

p
Ap


d
an
ls
ta
and from the background

en
am


g
rin
nd

ee
Fu

in
--

ng
ng

at E
ni

l
ah ca
ar

i
i
Le

uw tr
G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep

gy E
Information from each single fish is sent to a

itu f al an a

lo nd
st t o m g ific

no s a
In en Ka nin ert


ch ic
an m s ar C
di rt a e e
In epa an e L lin
feature extractor, whose purpose is to reduce
D r. M hin On
D ac EL
M PT
N

the data by measuring certain features

 The features are passed to a classifier

 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 
relationship

N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
p
of lightness, to minimize cost

lic
at
io
ns
Task of decision theory
reduce number of sea bass classified as salmon!
Move decision boundary toward smaller values
Threshold decision boundary and cost
 

the fish

Fish
N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
p lic
at
io
ns
Lightness
xT = [x1, x2]

Width
Adopt the lightness and add the width of
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 






Eg …
N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
an optimal performance

uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
Perhaps add other features

rin an
g d
Ap
p lic
at
io
not correlated with current ones

n s
Best decision boundary == one that provides
Warning: “noisy features” will reduce performance
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
“Optimal Performance” ??
 



Goal:
Objective:

N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
l
Dealing with Novel Data

at E am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
Performance on NOVEL data

at
io
ns
Issue of generalization!
Performance on TRAINING DATA !=
Optimal performance on NOVEL data
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
p
(test data)

lic
at
io
ns
Overfitting

http://wiki.bethanycrane.com/overfitting-of-data

training data very well, even

Learned hypothesis may fit the

generalize to new examples

outliers (noise) but fail to
Pattern Recognition Systems
 Sensing

ns
io
at
p lic
Using transducer (camera, microphone, …)

Ap
d


an
ls
ta
en
PR system depends of the bandwidth, the

am

g
rin


nd

ee
Fu

in
--
resolution sensitivity distortion of the

ng
ng

at E
ni

l
ah ca
ar

i
i
Le

uw tr
G lec
of ct uy De n
transducer

te Ele Bh d tio
Te ron an ep

gy E
itu f al an a

lo nd
st t o m g ific

no s a
In en Ka nin ert

ch ic
an m s ar C
di rt a e e
In epa an e L lin
D r. M hin On
D ac EL
M PT
N

 Segmentation and grouping

 Patterns should be well separated
(should not overlap)
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 Feature extraction
 Discriminative features

s
Want features INVARIANT w.r.t. translation,

n
io
at


p lic
Ap
rotation, scale.

d
an
ls
ta
en
am
Classification

g
rin
nd

ee
Fu


in
--

ng
ng

at E
ni

l
ah ca
ar
Using feature vector (provided by feature extractor)

i
i
Le

uw tr
G lec
of ct uy De n


te Ele Bh d tio
Te ron an ep

gy E
itu f al an a

lo nd
st t o m g ific
to assign given object to a category.

no s a
In en Ka nin ert

ch ic
an m s ar C
di rt a e e
In epa an e L lin
D r. M hin On
D ac EL
Post Processing
M PT

N

 Exploit context info not from the target pattern

itself to improve performance.
 





N
M PT
D ac EL
D r. M hin On
In epa an e L lin
Training

di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
Evaluation

te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
The Design Cycle

no s a ar
Model Choice

lo nd ni
gy E ng
G lec --
Data collection
Feature Choice

uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
Computational Complexity
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
Data Collection

st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
and testing the system?

uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
large and representative set of examples for training
How do we know when we have collected an adequately
 





domain.
Ideally…

N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
Feature Choice

st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
Simple to extract

no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
i
Insensitive to noise.

ah ca nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
Invariant to irrelevant transformation
Depends on characteristics of problem
 


N
M PT
D ac EL
Model Choice

D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i
(e.g., of our fish classifier)

ng en
in ta
ee ls
rin an
g d
Ap
p lic
at
io
n s
Consider another class of model
If not satisfied with performance
 





models
Training

N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
identify best model

itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
p lic
at
io
n s
determine appropriate parameters
Use data to obtain good classifier

Many procedures for training classifiers and choosing

 



Evaluation

another one
N
M PT
D ac EL
D r. M hin On
~= performance

In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
Measure error rate

st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
p lic
at
io
ns
May suggest switching from one set of features to
 


N
M PT
categories?
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
and performance?

lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
Computational Complexity

number of features, patterns or

How algorithm scales as function of
Trade-off between computational ease
 





input patterns
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
in the training set

In en Ka nin ert
st t o m g ific
itu f al an a
Supervised learning

te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
Unsupervised learning

lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
Learning and Adaptation

The system forms clusters or “natural groupings” of the

A teacher provides a category label or cost for each pattern
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 What is Clustering?
Also called unsupervised learning, sometimes called

ns
io
at
classification by statisticians.

p lic
Ap
d
an
ls
ta
en
am

g
rin
nd

ee
Fu

in
• Organizing data into classes such that there is

--

ng
ng

at E
ni

l
ah ca
ar

i
i
Le

uw tr
G lec
of ct uy De n
te Ele Bh d tio
Te ron an ep
• high intra-class similarity

gy E
itu f al an a

lo nd
st t o m g ific

no s a
In en Ka nin ert

ch ic
an m s ar C
di rt a e e
In epa an e L lin
• low inter-class similarity
D r. M hin On
D ac EL
M PT
N

• Finding the class labels and the number of classes directly

from the data (in contrast to classification).
• More informally, finding natural groupings among objects.
0.23
N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng

3
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
p lic
at
io
ns
Peter Piotr
Defining Distance Measures

342.7
between O1 and O2 is a real number denoted by D(O1,O2)
universe of possible objects. The distance (dissimilarity)
Definition: Let O1 and O2 be two objects from the
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
Simpson's Family
N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert

School Employees
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
Females
Clustering is subjective

Males
What is a natural grouping among these objects?
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns
 N
M PT
D ac EL
D r. M hin On
In epa an e L lin
di rt a e e
an m s ar C
In en Ka nin ert
st t o m g ific
itu f al an a
te Ele Bh d tio
of ct uy De n
Te ron an ep
ch ic Le
no s a ar
lo nd ni
gy E ng
G lec --
uw tr Fu
ah cai nd
at E l am
i ng en
in ta
ee ls
rin an
g d
Ap
plic
at
io
ns