Scribd
Upload
32 views

Foundations of Machine Learning: Sudeshna Sarkar IIT Kharagpur

This document discusses feature extraction and dimensionality reduction techniques. It begins by defining feature extraction as mapping a set of features to a reduced feature set that maximizes a learner's ability to classify patterns. It then discusses principal component analysis (PCA) and how it works to find a projection of data to a lower dimensional space while preserving as much of the variance as possible. However, PCA does not consider class labels. Linear discriminant analysis (LDA) is then introduced as a technique that finds projections that maximize the distance between class means while minimizing within-class variance, with the goal of better separating classes.

Uploaded by

Venkatesh Gaduputi
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
32 views

Foundations of Machine Learning: Sudeshna Sarkar IIT Kharagpur

This document discusses feature extraction and dimensionality reduction techniques. It begins by defining feature extraction as mapping a set of features to a reduced feature set that maximizes a learner's ability to classify patterns. It then discusses principal component analysis (PCA) and how it works to find a projection of data to a lower dimensional space while preserving as much of the variance as possible. However, PCA does not consider class labels. Linear discriminant analysis (LDA) is then introduced as a technique that finds projections that maximize the distance between class means while minimizing within-class variance, with the goal of better separating classes.

Uploaded by

Venkatesh Gaduputi
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 19

Foundations of Machine Learning

Module 3: Instance Based Learning and Feature


Reduction

Part C: Feature Extraction

Sudeshna Sarkar
IIT Kharagpur
Feature extraction - definition
••  Given a set of features
the Feature Extraction(“Construction”) problem is
is to map to some feature set that maximizes the
learner’s ability to classify patterns
Feature Extraction
• Find
  a projection matrix w from N-dimensional to M-
dimensional vectors that keeps error low
• Assume that N features are linear combination of vectors

• What we expect from such basis


– Uncorrelated, cannot be reduced further
– Have large variance or otherwise bear no information
Geometric picture of principal components (PCs)
Geometric picture of principal components (PCs)
Geometric picture of principal components (PCs)
Algebraic definition of PCs
Given a sample of p observations on a vector of N variables

 x , x ,, x   
1 2 p
N

define the first principal component of the sample


by the linear transformation
N
z1  w x j   wi1 xij ,
T
1 j  1,2,  , p.
i 1

where the vector w1  ( w11 , w21 , , wN 1 )


x j  ( x1 j , x2 j ,, x Nj )
is chosen such that var[ z1 ] is maximum.
PCA
PCA
• Choose
  directions such that a total variance of data
will be maximum
1. Maximize Total Variance
• Choose directions that are orthogonal
2. Minimize correlation

• Choose orthogonal directions which maximize total


variance
PCA
• -dimensional
  feature space
• symmetric covariance matrix estimated from
samples
• Select largest eigenvalue of the covariance matrix
and associated eigenvectors
• The first eigenvector will be a direction with largest
variance
PCA for image compression

p=1 p=2 p=4 p=8

Original
p=16 p=32 p=64 p=100 Image
Is PCA a good criterion for classification?

• Data variation
determines the
projection direction
• What’s missing?
– Class information
What is a good projection?
Two classes
• Similarly, what is a
overlap
good criterion?
– Separating different
classes

Two classes are


separated
What class information may be useful?

• Between-class distance
– Distance between the centroids
of different classes

Between-class distance
What class information may be useful?
• Between-class distance
– Distance between the centroids of
different classes
• Within-class distance
• Accumulated distance of an instance
to the centroid of its class

• Linear discriminant analysis (LDA) finds


most discriminant projection by
• maximizing between-class distance
• and minimizing within-class distance Within-class distance
Linear Discriminant Analysis
• Find
  a low-dimensional space such that when
is projected, classes are well-separated
Means and Scatter after projection
Good Projection
• Means are as far away as possible
• Scatter is small as possible
• Fisher Linear Discriminant

 m1  m2 
2

J  w 
s s
2
1
2
2
Thank You

You might also like