Data Mining

Practical Machine Learning Tools and Techniques

Slides for Chapter 4 of Data Mining by I. H. Witten, E. Frank and
M. A. Hall
 Algorithms: The basic methods

●
Inferring rudimentary rules
●
Statistical modeling
●
Constructing decision trees
●
Constructing rules
●
Association rule learning
●
Linear models
●
Instance-based learning
●
Clustering

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 2

 Simplicity first

●
Simple algorithms often work very well!
●
There are many kinds of simple structure, eg:
♦ One attribute does all the work
♦ All attributes contribute equally & independently
♦ A weighted linear combination might do
♦ Instance-based: use a few prototypes
♦ Use simple logical rules
●
Success of method depends on the domain

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 3

 Inferring rudimentary rules

●
1R: learns a 1-level decision tree
♦ I.e., rules that all test one particular attribute
●
Basic version
♦ One branch for each value
♦ Each branch assigns most frequent class
♦ Error rate: proportion of instances that don’t belong
to the majority class of their corresponding branch
♦ Choose attribute with lowest error rate
(assumes nominal attributes)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 4

 Pseudo-code for 1R

For each attribute,

For each value of the attribute, make a rule as follows:
count how often each class appears
find the most frequent class
make the rule assign that class to this attribute-value
Calculate the error rate of the rules
Choose the rules with the smallest error rate

●
Note: “missing” is treated as a separate attribute value

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 5

 Evaluating the weather attributes
Outlook Temp Humidity Windy Play

Sunny Hot High False No Attribute Rules Errors Total

errors
Sunny Hot High True No
Outlook Sunny → No 2/5 4/14
Overcast Hot High False Yes
Overcast → Yes 0/4
Rainy Mild High False Yes
Rainy → Yes 2/5
Rainy Cool Normal False Yes
Temp Hot → No* 2/4 5/14
Rainy Cool Normal True No
Mild → Yes 2/6
Overcast Cool Normal True Yes
Cool → Yes 1/4
Sunny Mild High False No
Humidity High → No 3/7 4/14
Sunny Cool Normal False Yes
Normal → Yes 1/7
Rainy Mild Normal False Yes
Windy False → Yes 2/8 5/14
Sunny Mild Normal True Yes
True → No* 3/6
Overcast Mild High True Yes
Overcast Hot Normal False Yes
Rainy Mild High True No * indicates a tie

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 6

 Dealing with numeric attributes
●
Discretize numeric attributes
●
Divide each attribute’s range into intervals
♦ Sort instances according to attribute’s values
♦ Place breakpoints where class changes (majority class)
♦ This minimizes the total error
●
Example: temperature from weather data
64 65 68 69 70 71 72 72 75 75 80 81 83 85
Yes | No | Yes Yes Yes | No No Yes | Yes Yes | No | Yes Yes | No

Outlook Temperature Humidity Windy Play

Sunny 85 85 False No
Sunny 80 90 True No
Overcast 83 86 False Yes
Rainy 75 80 False Yes
… … … … …

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 7

 The problem of overfitting
●
This procedure is very sensitive to noise
♦ One instance with an incorrect class label will probably
produce a separate interval
●
Also: time stamp attribute will have zero errors
●
Simple solution:
enforce minimum number of instances in majority class
per interval
●
Example (with min = 3):
64 65 68 69 70 71 72 72 75 75 80 81 83 85
Yes | No | Yes Yes Yes | No No Yes | Yes Yes | No | Yes Yes | No

64 65 68 69 70 71 72 72 75 75 80 81 83 85
Yes No Yes Yes Yes | No No Yes Yes Yes | No Yes Yes No

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 8

 With overfitting avoidance

●
Resulting rule set:

Attribute Rules Errors Total errors

Outlook Sunny → No 2/5 4/14
Overcast → Yes 0/4
Rainy → Yes 2/5
Temperature ≤ 77.5 → Yes 3/10 5/14
> 77.5 → No* 2/4
Humidity ≤ 82.5 → Yes 1/7 3/14
> 82.5 and ≤ 95.5 → No 2/6
> 95.5 → Yes 0/1
Windy False → Yes 2/8 5/14
True → No* 3/6

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 9

 Discussion of 1R
●
1R was described in a paper by Holte (1993)
♦ Contains an experimental evaluation on 16 datasets (using
cross-validation so that results were representative of
performance on future data)
♦ Minimum number of instances was set to 6 after some
experimentation
♦ 1R’s simple rules performed not much worse than much
more complex decision trees
●
Simplicity first pays off!

Very Simple Classification Rules Perform Well on Most Commonly

Used Datasets
Robert C. Holte, Computer Science Department, University of Ottawa

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 10

 Discussion of 1R: Hyperpipes

●
Another simple technique: build one rule for each class
♦ Each rule is a conjunction of tests, one for each attribute

♦ For numeric attributes: test checks whether instance's value is

inside an interval
● Interval given by minimum and maximum observed in

training data
♦ For nominal attributes: test checks whether value is one of a

subset of attribute values

● Subset given by all possible values observed in training data

♦ Class with most matching tests is predicted

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 11

 Statistical modeling

●
“Opposite” of 1R: use all the attributes
●
Two assumptions: Attributes are
♦ equally important
♦ statistically independent (given the class value)
● I.e., knowing the value of one attribute says nothing about
the value of another (if the class is known)
●
Independence assumption is never correct!
●
But … this scheme works well in practice

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 12

 Probabilities for weather data
Outlook Temperature Humidity Windy Play
Yes No Yes No Yes No Yes No Yes No
Sunny 2 3 Hot 2 2 High 3 4 False 6 2 9 5
Overcast 4 0 Mild 4 2 Normal 6 1 True 3 3
Rainy 3 2 Cool 3 1
Sunny 2/9 3/5 Hot 2/9 2/5 High 3/9 4/5 False 6/9 2/5 9/ 5/
Overcast 4/9 0/5 Mild 4/9 2/5 Normal 6/9 1/5 True 3/9 3/5 14 14
Rainy 3/9 2/5 Cool 3/9 1/5
Outlook Temp Humidity Windy Play
Sunny Hot High False No
Sunny Hot High True No
Overcast Hot High False Yes
Rainy Mild High False Yes
Rainy Cool Normal False Yes
Rainy Cool Normal True No
Overcast Cool Normal True Yes
Sunny Mild High False No
Sunny Cool Normal False Yes
Rainy Mild Normal False Yes
Sunny Mild Normal True Yes
Overcast Mild High True Yes
Overcast Hot Normal False Yes
Data Mining: Practical Machine Learning Tools and Techniques
Rainy (Chapter
Mild 4)
High True No13
 Probabilities for weather data
Outlook Temperature Humidity Windy Play
Yes No Yes No Yes No Yes No Yes No
Sunny 2 3 Hot 2 2 High 3 4 False 6 2 9 5
Overcast 4 0 Mild 4 2 Normal 6 1 True 3 3
Rainy 3 2 Cool 3 1
Sunny 2/9 3/5 Hot 2/9 2/5 High 3/9 4/5 False 6/9 2/5 9/ 5/
Overcast 4/9 0/5 Mild 4/9 2/5 Normal 6/9 1/5 True 3/9 3/5 14 14
Rainy 3/9 2/5 Cool 3/9 1/5

●
A new day: Outlook Temp. Humidity Windy Play
Sunny Cool High True ?

Likelihood of the two classes

For “yes” = 2/9 × 3/9 × 3/9 × 3/9 × 9/14 = 0.0053
For “no” = 3/5 × 1/5 × 4/5 × 3/5 × 5/14 = 0.0206
Conversion into a probability by normalization:
P(“yes”) = 0.0053 / (0.0053 + 0.0206) = 0.205
P(“no”) = 0.0206 / (0.0053 + 0.0206) = 0.795

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 14

 Bayes’s rule
Probability of event H given evidence E:
●

Pr [E∣H]Pr [H]
Pr [H∣E]=
Pr [E]

A priori probability of H :
●
Pr [H]
● Probability of event before evidence is seen
A posteriori probability of H :
●
Pr [H∣E]
● Probability of event after evidence is seen

Thomas Bayes
Born: 1702 in London, England
Died: 1761 in Tunbridge Wells, Kent, England

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 15

 Naïve Bayes for classification

●
Classification learning: what’s the probability of
the class given an instance?
♦ Evidence E = instance
♦ Event H = class value for instance
●
Naïve assumption: evidence splits into parts (i.e.
attributes) that are independent

Pr [E1∣H]Pr [E2∣H]Pr [En∣H]Pr [H]

Pr [H∣E]=
Pr [E]

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 16

 Weather data example
Outlook Temp. Humidity Windy Play
Evidence E
Sunny Cool High True ?

Pr [ yes∣E]=Pr [Outlook =Sunny∣yes]

×Pr [Temperature=Cool∣yes]
Probability of ×Pr [Humidity=High∣yes]
class “yes” ×Pr [ Windy=True∣yes]
Pr [ yes]
×
Pr [E]
2 3 3 3 9
× × × ×
9 9 9 9 14
=
Pr [E]
Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 17
 The “zero-frequency problem”

●
What if an attribute value doesn’t occur with every class
value?
(e.g. “Humidity = high” for class “yes”)
♦ Probability will be zero! Pr [Humidity=High∣yes]=0
♦ A posteriori probability will also be zero! Pr [yes∣E]=0
(No matter how likely the other values are!)
●
Remedy: add 1 to the count for every attribute value-class
combination (Laplace estimator)
●
Result: probabilities will never be zero!
(also: stabilizes probability estimates)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 18

 Modified probability estimates

●
In some cases adding a constant different from 1
might be more appropriate
●
Example: attribute outlook for class yes

2 /3 4 /3 3 /3

9 9 9

Sunny Overcast Rainy

●
Weights don’t need to be equal
(but they must sum to 1)
2 p1 4 p2 3 p3
9 9 9

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 19

 Missing values
●
Training: instance is not included in frequency count for
attribute value-class combination
●
Classification: attribute will be omitted from calculation
●
Example:
Outlook Temp. Humidity Windy Play
? Cool High True ?

Likelihood of “yes” = 3/9 × 3/9 × 3/9 × 9/14 = 0.0238

Likelihood of “no” = 1/5 × 4/5 × 3/5 × 5/14 = 0.0343
P(“yes”) = 0.0238 / (0.0238 + 0.0343) = 41%
P(“no”) = 0.0343 / (0.0238 + 0.0343) = 59%

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 20

 Numeric attributes
● Usual assumption: attributes have a normal or
Gaussian probability distribution (given the
class)
● The probability density function for the normal

distribution is defined by two parameters:

●
Sample mean µ 1
n
= ∑ x i
n i=1
●
Standard deviation σ
√
n
1 2
σ= ∑
n−1 i=1
(x i −μ)
● Then the density function f(x) is

(x−μ)2
1 −
2 σ2
f (x)= e
√2 π σ
Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 21
 Statistics for weather data

Outlook Temperature Humidity Windy Play

Yes No Yes No Yes No Yes No Yes No
Sunny 2 3 64, 68, 65,71, 65, 70, 70, 85, False 6 2 9 5
Overcast 4 0 69, 70, 72,80, 70, 75, 90, 91, True 3 3
Rainy 3 2 72, … 85, … 80, … 95, …
Sunny 2/9 3/5 µ =73 µ =75 µ =79 µ =86 False 6/9 2/5 9/ 5/
Overcast 4/9 0/5 σ =6.2 σ =7.9 σ =10.2 σ =9.7 True 3/9 3/5 14 14
Rainy 3/9 2/5

●
Example density value:
66−732
1 −
2⋅6.2
2

f temperature=66∣yes= e =0.0340
2 6.2

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 22

 Classifying a new day

●
A new day: Outlook Temp. Humidity Windy Play
Sunny 66 90 true ?

Likelihood of “yes” = 2/9 × 0.0340 × 0.0221 × 3/9 × 9/14 = 0.000036

Likelihood of “no” = 3/5 × 0.0221 × 0.0381 × 3/5 × 5/14 = 0.000108
P(“yes”) = 0.000036 / (0.000036 + 0. 000108) = 25%
P(“no”) = 0.000108 / (0.000036 + 0. 000108) = 75%

●
Missing values during training are not included
in calculation of mean and standard deviation

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 23

 Probability densities

●
Relationship between probability and density:
 
Pr [c− xc ]≈×f c
2 2
●
But: this doesn’t change calculation of a posteriori
probabilities because ε cancels out
●
Exact relationship:

Pr [axb]=∫ f tdt
a

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 24

 Multinomial naïve Bayes I
●
Version of naïve Bayes used for document classification using
bag of words model
● n1,n2, ... , nk: number of times word i occurs in document
● P1,P2, ... , Pk: probability of obtaining word i when sampling from
documents in class H
●
Probability of observing document E given class H (based on
multinomial distribution):
k ni
Pi
Pr [E∣H]≈N!×∏
i=1 ni !

●
Ignores probability of generating a document of the right length
(prob. assumed constant for each class)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 25

 Multinomial naïve Bayes II
●
Suppose dictionary has two words, yellow and blue
●
Suppose Pr[yellow | H] = 75% and Pr[blue | H] = 25%
●
Suppose E is the document “blue yellow blue”
●
Probability of observing document:
0.751 0.252 9
Pr [{blue yellow blue}∣H]≈3 !× 1! × 2! = 64 ≈0.14
Suppose there is another class H' that has
Pr[yellow | H'] = 10% and Pr[yellow | H'] = 90%:
0.11 0.92
Pr [{blue yellow blue}∣H']≈3!× 1!
× 2!
=0.24
●
Need to take prior probability of class into account to make final
classification
●
Factorials don't actually need to be computed
●
Underflows can be prevented by using logarithms

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 26

 Naïve Bayes: discussion

●
Naïve Bayes works surprisingly well (even if independence
assumption is clearly violated)
●
Why? Because classification doesn’t require accurate
probability estimates as long as maximum probability is
assigned to correct class
●
However: adding too many redundant attributes will cause
problems (e.g. identical attributes)
●
Note also: many numeric attributes are not normally
distributed (→ kernel density estimators)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 27

 Constructing decision trees

●
Strategy: top down
Recursive divide-and-conquer fashion
♦ First: select attribute for root node
Create branch for each possible attribute value
♦ Then: split instances into subsets
One for each branch extending from the node
♦ Finally: repeat recursively for each branch, using only
instances that reach the branch
●
Stop if all instances have the same class

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 28

Which attribute to select?

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 29

Which attribute to select?

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 30

 Criterion for attribute selection

●
Which is the best attribute?
♦ Want to get the smallest tree
♦ Heuristic: choose the attribute that produces the
“purest” nodes
●
Popular impurity criterion: information gain
♦ Information gain increases with the average purity of
the subsets
●
Strategy: choose attribute that gives greatest
information gain

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 31

 Computing information

●
Measure information in bits
♦ Given a probability distribution, the info required to

predict an event is the distribution’s entropy

♦ Entropy gives the information required in bits

(can involve fractions of bits!)

●
Formula for computing the entropy:

entropy p1, p 2, ... ,p n=−p1 log p1−p2 log p2 ...−p n log pn

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 32

 Example: attribute Outlook

●
Outlook = Sunny :
info[2,3]=entropy 2/5,3/5=−2/5 log 2/5−3/5 log 3/5=0.971 bits
●
Outlook = Overcast : Note: this
info[4,0]=entropy 1,0=−1 log 1−0 log0=0 bits is normally
undefined.
●
Outlook = Rainy :
info[2,3]=entropy 3/5,2/5=−3/5 log 3/5−2/5 log 2/5=0.971 bits
●
Expected information for attribute:
info[3,2],[4,0],[3,2]=5/14×0.9714/14×05/14×0.971=0.693 bits

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 33

 Computing information gain

●
Information gain: information before splitting –
information after splitting
gain(Outlook ) = info([9,5]) – info([2,3],[4,0],[3,2])
= 0.940 – 0.693
= 0.247 bits

●
Information gain for attributes from weather data:

gain(Outlook ) = 0.247 bits

gain(Temperature ) = 0.029 bits
gain(Humidity ) = 0.152 bits
gain(Windy ) = 0.048 bits

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 34

Continuing to split

gain(Temperature ) = 0.571 bits

gain(Humidity ) = 0.971 bits
gain(Windy ) = 0.020 bits

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 35

 Final decision tree

●
Note: not all leaves need to be pure; sometimes
identical instances have different classes
⇒ Splitting stops when data can’t be split any further

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 36

 Wishlist for a purity measure

●
Properties we require from a purity measure:
♦ When node is pure, measure should be zero
♦ When impurity is maximal (i.e. all classes equally
likely), measure should be maximal
♦ Measure should obey multistage property (i.e. decisions
can be made in several stages):

measure[2,3,4]=measure [2,7]7/9×measure[3,4]
●
Entropy is the only function that satisfies all three
properties!

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 37

 Properties of the entropy

●
The multistage property:
q r
entropy p ,q , r=entropy p ,qrqr×entropy  qr , qr 

●
Simplification of computation:
info[2,3,4]=−2/9×log 2/9−3/9×log3/9−4/9×log 4/9
=[−2×log 2−3×log 3−4×log 49×log 9]/9
●
Note: instead of maximizing info gain we could
just minimize information

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 38

 Highly-branching attributes

●
Problematic: attributes with a large number of
values (extreme case: ID code)
●
Subsets are more likely to be pure if there is a
large number of values
⇒ Information gain is biased towards choosing
attributes with a large number of values
⇒ This may result in overfitting (selection of an
attribute that is non-optimal for prediction)
●
Another problem: fragmentation

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 39

 Weather data with ID code
ID code Outlook Temp. Humidity Windy Play
A Sunny Hot High False No
B Sunny Hot High True No
C Overcast Hot High False Yes
D Rainy Mild High False Yes
E Rainy Cool Normal False Yes
F Rainy Cool Normal True No
G Overcast Cool Normal True Yes
H Sunny Mild High False No
I Sunny Cool Normal False Yes
J Rainy Mild Normal False Yes
K Sunny Mild Normal True Yes
L Overcast Mild High True Yes
M Overcast Hot Normal False Yes
N Rainy Mild High True No

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 40

 Tree stump for ID code attribute

●
Entropy of split:
infoID code=info[0,1]info[0,1]...info[0,1]=0 bits
⇒ Information gain is maximal for ID code (namely
0.940 bits)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 41

 Gain ratio

●
Gain ratio: a modification of the information gain that
reduces its bias
●
Gain ratio takes number and size of branches into
account when choosing an attribute
♦ It corrects the information gain by taking the intrinsic
information of a split into account
●
Intrinsic information: entropy of distribution of instances
into branches (i.e. how much info do we need to tell
which branch an instance belongs to)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 42

 Computing the gain ratio

●
Example: intrinsic information for ID code
info[1,1,. ..,1]=14×−1/14×log 1/14=3.807 bits
●
Value of attribute decreases as intrinsic
information gets larger
●
Definition of gain ratio:

gain_ratioattribute=gainattribute
intrinsic_infoattribute

●
Example:

gain_ratioID code= 0.940 bits

3.807 bits
=0.246

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 43

 Gain ratios for weather data

Outlook Temperature
Info: 0.693 Info: 0.911
Gain: 0.940-0.693 0.247 Gain: 0.940-0.911 0.029
Split info: info([5,4,5]) 1.577 Split info: info([4,6,4]) 1.557
Gain ratio: 0.247/1.577 0.157 Gain ratio: 0.029/1.557 0.019
Humidity Windy
Info: 0.788 Info: 0.892
Gain: 0.940-0.788 0.152 Gain: 0.940-0.892 0.048
Split info: info([7,7]) 1.000 Split info: info([8,6]) 0.985
Gain ratio: 0.152/1 0.152 Gain ratio: 0.048/0.985 0.049

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 44

 More on the gain ratio

●
“Outlook” still comes out top
●
However: “ID code” has greater gain ratio
♦ Standard fix: ad hoc test to prevent splitting on that type of
attribute
●
Problem with gain ratio: it may overcompensate
♦ May choose an attribute just because its intrinsic
information is very low
♦ Standard fix: only consider attributes with greater than
average information gain

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 45

 Discussion

●
Top-down induction of decision trees: ID3,
algorithm developed by Ross Quinlan
♦ Gain ratio just one modification of this basic algorithm
♦ ⇒ C4.5: deals with numeric attributes, missing values,
noisy data
●
Similar approach: CART
●
There are many other attribute selection criteria!
(But little difference in accuracy of result)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 46

 Covering algorithms
●
Convert decision tree into a rule set
♦ Straightforward, but rule set overly complex
♦ More effective conversions are not trivial
●
Instead, can generate rule set directly
♦ for each class in turn find rule set that covers all
instances in it
(excluding instances not in the class)
●
Called a covering approach:
♦ at each stage a rule is identified that “covers” some
of the instances

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 47

 Example: generating a rule

If true If x > 1.2 and y > 2.6

then class = a then class = a
If x > 1.2
then class = a

●
Possible rule set for class “b”:
If x ≤ 1.2 then class = b
If x > 1.2 and y ≤ 2.6 then class = b
●
Could add more rules, get “perfect” rule set

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 48

 Rules vs. trees

Corresponding decision tree:

(produces exactly the same
predictions)

●
But: rule sets can be more perspicuous when decision
trees suffer from replicated subtrees
●
Also: in multiclass situations, covering algorithm
concentrates on one class at a time whereas decision tree
learner takes all classes into account

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 49

 Simple covering algorithm

●
Generates a rule by adding tests that maximize rule’s
accuracy
●
Similar to situation in decision trees: problem of
selecting an attribute to split on
♦ But: decision tree inducer maximizes overall purity
●
Each new test reduces
rule’s coverage:

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 50

 Selecting a test

●
Goal: maximize accuracy
♦ t total number of instances covered by rule
♦ p positive examples of the class covered by rule

♦ t – p number of errors made by rule

⇒ Select test that maximizes the ratio p/t

●
We are finished when p/t = 1 or the set of
instances can’t be split any further

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 51

 Example: contact lens data

If ?
●
Rule we seek: then recommendation = hard
●
Possible tests:

Age = Young 2/8

Age = Pre-presbyopic 1/8
Age = Presbyopic 1/8
Spectacle prescription = Myope 3/12
Spectacle prescription = Hypermetrope 1/12
Astigmatism = no 0/12
Astigmatism = yes 4/12
Tear production rate = Reduced 0/12
Tear production rate = Normal 4/12

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 52

 Modified rule and resulting data

●
Rule with best test added:
If astigmatism = yes
then recommendation = hard
●
Instances covered by modified rule:
Age Spectacle prescription Astigmatism Tear production Recommended
rate lenses
Young Myope Yes Reduced None
Young Myope Yes Normal Hard
Young Hypermetrope Yes Reduced None
Young Hypermetrope Yes Normal hard
Pre-presbyopic Myope Yes Reduced None
Pre-presbyopic Myope Yes Normal Hard
Pre-presbyopic Hypermetrope Yes Reduced None
Pre-presbyopic Hypermetrope Yes Normal None
Presbyopic Myope Yes Reduced None
Presbyopic Myope Yes Normal Hard
Presbyopic Hypermetrope Yes Reduced None
Presbyopic Hypermetrope Yes Normal None

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 53

 Further refinement

If astigmatism = yes
●
Current state: and ?
then recommendation = hard

●
Possible tests:

Age = Young 2/4

Age = Pre-presbyopic 1/4
Age = Presbyopic 1/4
Spectacle prescription = Myope 3/6
Spectacle prescription = Hypermetrope 1/6
Tear production rate = Reduced 0/6
Tear production rate = Normal 4/6

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 54

 Modified rule and resulting data

●
Rule with best test added:
If astigmatism = yes
and tear production rate = normal
then recommendation = hard

●
Instances covered by modified rule:
Age Spectacle prescription Astigmatism Tear production Recommended
rate lenses
Young Myope Yes Normal Hard
Young Hypermetrope Yes Normal hard
Pre-presbyopic Myope Yes Normal Hard
Pre-presbyopic Hypermetrope Yes Normal None
Presbyopic Myope Yes Normal Hard
Presbyopic Hypermetrope Yes Normal None

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 55

 Further refinement
●
Current state:
If astigmatism = yes
and tear production rate = normal
and ?
then recommendation = hard

●
Possible tests:
Age = Young 2/2
Age = Pre-presbyopic 1/2
Age = Presbyopic 1/2
Spectacle prescription = Myope 3/3
Spectacle prescription = Hypermetrope 1/3

●
Tie between the first and the fourth test
♦ We choose the one with greater coverage

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 56

 The result

●
Final rule: If astigmatism = yes
and tear production rate = normal
and spectacle prescription = myope
then recommendation = hard

●
Second rule for recommending “hard lenses”:
(built from instances not covered by first rule)

If age = young and astigmatism = yes

and tear production rate = normal
then recommendation = hard

●
These two rules cover all “hard lenses”:
♦ Process is repeated with other two classes

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 57

 Pseudo-code for PRISM
For each class C
Initialize E to the instance set
While E contains instances in class C
Create a rule R with an empty left-hand side that predicts class C
Until R is perfect (or there are no more attributes to use) do
For each attribute A not mentioned in R, and each value v,
Consider adding the condition A = v to the left-hand side of R
Select A and v to maximize the accuracy p/t
(break ties by choosing the condition with the largest p)
Add A = v to R
Remove the instances covered by R from E

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 58

 Rules vs. decision lists

●
PRISM with outer loop removed generates a decision
list for one class
♦ Subsequent rules are designed for rules that are not covered
by previous rules
♦ But: order doesn’t matter because all rules predict the same
class
●
Outer loop considers all classes separately
♦ No order dependence implied
●
Problems: overlapping rules, default rule required

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 59

 Separate and conquer

●
Methods like PRISM (for dealing with one class)
are separate-and-conquer algorithms:
♦ First, identify a useful rule
♦ Then, separate out all the instances it covers
♦ Finally, “conquer” the remaining instances
●
Difference to divide-and-conquer methods:
♦ Subset covered by rule doesn’t need to be explored
any further

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 60

 Mining association rules

●
Naïve method for finding association rules:
♦ Use separate-and-conquer method
♦ Treat every possible combination of attribute values as a
separate class
●
Two problems:
♦ Computational complexity
♦ Resulting number of rules (which would have to be
pruned on the basis of support and confidence)
●
But: we can look for high support rules directly!

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 61

 Item sets

●
Support: number of instances correctly covered by
association rule
♦ The same as the number of instances covered by all tests in
the rule (LHS and RHS!)
●
Item: one test/attribute-value pair
●
Item set : all items occurring in a rule
●
Goal: only rules that exceed pre-defined support
⇒ Do it by finding all item sets with the given minimum
support and generating rules from them!

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 62

Weather data
Outlook Temp Humidity Windy Play
Sunny Hot High False No
Sunny Hot High True No
Overcast Hot High False Yes
Rainy Mild High False Yes
Rainy Cool Normal False Yes
Rainy Cool Normal True No
Overcast Cool Normal True Yes
Sunny Mild High False No
Sunny Cool Normal False Yes
Rainy Mild Normal False Yes
Sunny Mild Normal True Yes
Overcast Mild High True Yes
Overcast Hot Normal False Yes
Rainy Mild High True No

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 63

 Item sets for weather data

One-item sets Two-item sets Three-item sets Four-item sets

Outlook = Sunny (5) Outlook = Sunny Outlook = Sunny Outlook = Sunny
Temperature = Hot (2) Temperature = Hot Temperature = Hot
Humidity = High (2) Humidity = High
Play = No (2)
Temperature = Cool (4) Outlook = Sunny Outlook = Sunny Outlook = Rainy
Humidity = High (3) Humidity = High Temperature = Mild
Windy = False (2) Windy = False
Play = Yes (2)
… … … …

●
In total: 12 one-item sets, 47 two-item sets, 39
three-item sets, 6 four-item sets and 0 five-item sets
(with minimum support of two)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 64

 Generating rules from an item set

●
Once all item sets with minimum support have been
generated, we can turn them into rules
●
Example:
Humidity = Normal, Windy = False, Play = Yes (4)

●
Seven (2N-1) potential rules:
If Humidity = Normal and Windy = False then Play = Yes 4/4
If Humidity = Normal and Play = Yes then Windy = False 4/6
If Windy = False and Play = Yes then Humidity = Normal 4/6
If Humidity = Normal then Windy = False and Play = Yes 4/7
If Windy = False then Humidity = Normal and Play = Yes 4/8
If Play = Yes then Humidity = Normal and Windy = False 4/9
If True then Humidity = Normal and Windy = False
and Play = Yes 4/12

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 65

 Rules for weather data
●
Rules with support > 1 and confidence = 100%:
Association rule Sup. Conf.
1 Humidity=Normal Windy=False ⇒ Play=Yes 4 100%
2 Temperature=Cool ⇒ Humidity=Normal 4 100%
3 Outlook=Overcast ⇒ Play=Yes 4 100%
4 Temperature=Cold Play=Yes ⇒ Humidity=Normal 3 100%
... ... ... ...
58 Outlook=Sunny Temperature=Hot ⇒ Humidity=High 2 100%

●
In total:
3 rules with support four
5 with support three
50 with support two

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 66

 Example rules from the same set
●
Item set:
Temperature = Cool, Humidity = Normal, Windy = False, Play = Yes (2)

●
Resulting rules (all with 100% confidence):
Temperature = Cool, Windy = False ⇒ Humidity = Normal, Play = Yes
Temperature = Cool, Windy = False, Humidity = Normal ⇒ Play = Yes
Temperature = Cool, Windy = False, Play = Yes ⇒ Humidity = Normal

due to the following “frequent” item sets:

Temperature = Cool, Windy = False (2)

Temperature = Cool, Humidity = Normal, Windy = False (2)
Temperature = Cool, Windy = False, Play = Yes (2)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 67

 Generating item sets efficiently

●
How can we efficiently find all frequent item sets?
●
Finding one-item sets easy
●
Idea: use one-item sets to generate two-item sets, two-item
sets to generate three-item sets, …
♦ If (A B) is frequent item set, then (A) and (B) have to be
frequent item sets as well!
♦ In general: if X is frequent k-item set, then all (k-1)-item

subsets of X are also frequent

⇒ Compute k-item set by merging (k-1)-item sets

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 68

 Example

●
Given: five three-item sets
(A B C), (A B D), (A C D), (A C E), (B C D)

●
Lexicographically ordered!
●
Candidate four-item sets:
(A B C D) OK because of (A C D) (B C D)
(A C D E) Not OK because of (C D E)
●
Final check by counting instances in dataset!
●
(k –1)-item sets are stored in hash table

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 69

 Generating rules efficiently

●
We are looking for all high-confidence rules
♦ Support of antecedent obtained from hash table
♦ But: brute-force method is (2N-1)
●
Better way: building (c + 1)-consequent rules from c-
consequent ones
♦ Observation: (c + 1)-consequent rule can only hold if all
corresponding c-consequent rules also hold
●
Resulting algorithm similar to procedure for large
item sets

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 70

 Example
●
1-consequent rules:
If Outlook = Sunny and Windy = False and Play = No
then Humidity = High (2/2)

If Humidity = High and Windy = False and Play = No

then Outlook = Sunny (2/2)

Corresponding 2-consequent rule:

If Windy = False and Play = No
then Outlook = Sunny and Humidity = High (2/2)

●
Final check of antecedent against hash table!

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 71

 Association rules: discussion

●
Above method makes one pass through the data for each
different size item set
♦ Other possibility: generate (k+2)-item sets just after (k+1)-item
sets have been generated
♦ Result: more (k+2)-item sets than necessary will be considered
but less passes through the data
♦ Makes sense if data too large for main memory
●
Practical issue: generating a certain number of rules (e.g. by
incrementally reducing min. support)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 72

 Other issues

●
Standard ARFF format very inefficient for typical
market basket data
♦ Attributes represent items in a basket and most items are
usually missing
♦ Data should be represented in sparse format
●
Instances are also called transactions
●
Confidence is not necessarily the best measure
♦ Example: milk occurs in almost every supermarket
transaction
♦ Other measures have been devised (e.g. lift)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 73

 Linear models: linear regression

●
Work most naturally with numeric attributes
●
Standard technique for numeric prediction
♦ Outcome is linear combination of attributes
x=w 0w 1 a1w2 a 2...w k a k
●
Weights are calculated from the training data
●
Predicted value for first training instance a(1)
k
w0 a1
0 w a
1 1
1
w a
2 2
1
...w a
k k
1
=∑ j=0 w a
j j
1

(assuming each instance is extended with a constant attribute with value 1)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 74

 Minimizing the squared error

● Choose k +1 coefficients to minimize the squared error

on the training data
● Squared error:
●
∑ni=1 x i −∑kj=0 w j aij  2

● Derive coefficients using standard matrix operations

● Can be done if there are more instances than attributes
(roughly speaking)
● Minimizing the absolute error is more difficult

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 75

 Classification
●
Any regression technique can be used for classification
♦ Training: perform a regression for each class, setting the
output to 1 for training instances that belong to class, and 0
for those that don’t
♦ Prediction: predict class corresponding to model with
largest output value (membership value)
●
For linear regression this is known as multi-response
linear regression
●
Problem: membership values are not in [0,1] range, so
aren't proper probability estimates

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 76

 Linear models: logistic regression
●
Builds a linear model for a transformed target
variable
●
Assume we have two classes
●
Logistic regression replaces the target
P[1∣a1, a2, ....,a k ]

by this target
P[1∣a1, a2, .... ,ak ]
log  1−P[1∣a 1, a 2, ...., a k ]


●
Logit transformation maps [0,1] to (-∞ , +∞ )

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 77

 Logit transformation

●
Resulting model:
1
Pr [1∣a 1, a2, ..., ak ]= 1e −w 0−w 1 a1 −...− w k ak


Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 78

 Example logistic regression model
● Model with w0 = 0.5 and w1 = 1:

●
Parameters are found from training data using
maximum likelihood

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 79

 Maximum likelihood
●
Aim: maximize probability of training data wrt
parameters
●
Can use logarithms of probabilities and maximize log-
likelihood of model:
 i
∑ni=1 1−xi log1−Pr [1∣ai
1 , a i
2 ,... , a k ]
i  i i i
x log Pr [1∣a1 ,a2 ,... , ak ]
where the x(i) are either 0 or 1
● Weights wi need to be chosen to maximize log-
likelihood (relatively simple method: iteratively re-
weighted least squares)
Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 80
 Multiple classes
●
Can perform logistic regression independently
for each class
(like multi-response linear regression)
●
Problem: probability estimates for different
classes won't sum to one
●
Better: train coupled models by maximizing
likelihood over all classes
●
Alternative that often works well in practice:
pairwise classification

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 81

 Pairwise classification
●
Idea: build model for each pair of classes, using only
training data from those classes
●
Problem? Have to solve k(k-1)/2 classification problems
for k-class problem
●
Turns out not to be a problem in many cases because
training sets become small:
♦ Assume data evenly distributed, i.e. 2n/k per learning

problem for n instances in total

♦ Suppose learning algorithm is linear in n

♦ Then runtime of pairwise classification is proportional

to (k(k-1)/2)×(2n/k) = (k-1)n

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 82

 Linear models are hyperplanes
●
Decision boundary for two-class logistic regression is
where probability equals 0.5:

Pr [1∣a1, a 2, ...,a k ]=1/1exp−w0−w1 a 1−...−w k ak =0.5

which occurs when −w 0−w 1 a1−...−w k ak =0

●
Thus logistic regression can only separate data that can be
separated by a hyperplane
●
Multi-response linear regression has the same problem.
Class 1 is assigned if:

w1
0 w 1
1 a 1 ...w 1
k a k w 2
0 w 2
1 a 1 ...w k ak
2

⇔w1
0 −w 2
0 w 1
1 −w 2
1 a1 ...w 1
k −w 2
k a k 0

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 83

 Linear models: the perceptron
●
Don't actually need probability estimates if all we want to do is
classification
●
Different approach: learn separating hyperplane
●
Assumption: data is linearly separable
●
Algorithm for learning separating hyperplane: perceptron
learning rule
●
Hyperplane: 0=w0 a0 w 1 a 1w2 a2...w k a k
where we again assume that there is a constant attribute with
value 1 (bias)
●
If sum is greater than zero we predict the first class, otherwise
the second class

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 84

 The algorithm
Set all weights to zero
Until all instances in the training data are classified correctly
For each instance I in the training data
If I is classified incorrectly by the perceptron
If I belongs to the first class add it to the weight vector
else subtract it from the weight vector

●
Why does this work?
Consider situation where instance a pertaining to the first class has
been added:
w 0a0 a0w1a1a 1w 2a2 a2...w k a k a k
This means output for a has increased by:

a0 a0a 1 a 1a2 a2 ...a k a k

This number is always positive, thus the hyperplane has moved into the correct
direction (and we can show output decreases for instances of other class)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 85

 Perceptron as a neural network

Output
layer

Input
layer

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 86

 Linear models: Winnow
●
Another mistake-driven algorithm for finding a separating
hyperplane
♦ Assumes binary data (i.e. attribute values are either zero

or one)
●
Difference: multiplicative updates instead of additive updates
♦ Weights are multiplied by a user-specified parameter α >
1(or its inverse)
●
Another difference: user-specified threshold parameter θ
♦ Predict first class if

w 0 a 0w 1 a1w 2 a2...w k ak 

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 87

 The algorithm
while some instances are misclassified
for each instance a in the training data
classify a using the current weights
if the predicted class is incorrect
if a belongs to the first class
for each ai that is 1, multiply wi by alpha
(if ai is 0, leave wi unchanged)
otherwise
for each ai that is 1, divide wi by alpha
(if ai is 0, leave wi unchanged)

●
Winnow is very effective in homing in on relevant
features (it is attribute efficient)
●
Can also be used in an on-line setting in which new
instances arrive continuously
(like the perceptron algorithm)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 88

 Balanced Winnow
●
Winnow doesn't allow negative weights and this can be a drawback in
some applications
●
Balanced Winnow maintains two weight vectors, one for each class:
while some instances are misclassified
for each instance a in the training data
classify a using the current weights
if the predicted class is incorrect
if a belongs to the first class
for each ai that is 1, multiply wi+ by alpha and divide wi- by alpha
(if ai is 0, leave wi+ and wi- unchanged)
otherwise
for each ai that is 1, multiply wi- by alpha and divide wi+ by alpha
(if ai is 0, leave wi+ and wi- unchanged)

●
Instance is classified as belonging to the first class (of two classes) if:

w 0 −w0− a 0w 1 −w 2− a1...w k −w k− a k 

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 89

 Instance-based learning

● Distance function defines what’s learned

● Most instance-based schemes use Euclidean
distance:
a 1
1 −a2
1 2
a 1
2 −a 2 2
2  ...a 1
k −a 2 2
k 

a(1) and a(2): two instances with k attributes

● Taking the square root is not required when
comparing distances
● Other popular metric: city-block metric
● Adds differences without squaring them

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 90

 Normalization and other issues

●
Different attributes are measured on different scales ⇒
need to be normalized:
v i −min v i
ai = max v i−min vi

vi : the actual value of attribute i

●
Nominal attributes: distance either 0 or 1
●
Common policy for missing values: assumed to be
maximally distant (given normalized attributes)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 91

 Finding nearest neighbors efficiently

●
Simplest way of finding nearest neighbour: linear scan of
the data
♦ Classification takes time proportional to the product of the
number of instances in training and test sets
●
Nearest-neighbor search can be done more efficiently using
appropriate data structures
●
We will discuss two methods that represent training data in
a tree structure:

kD-trees and ball trees

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 92

 kD-tree example

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 93

 Using kD-trees: example

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 94

 More on kD-trees
●
Complexity depends on depth of tree, given by logarithm of
number of nodes
●
Amount of backtracking required depends on quality of tree
(“square” vs. “skinny” nodes)
●
How to build a good tree? Need to find good split point and split
direction
♦ Split direction: direction with greatest variance

♦ Split point: median value along that direction

●
Using value closest to mean (rather than median) can be better if
data is skewed
●
Can apply this recursively

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 95

 Building trees incrementally
●
Big advantage of instance-based learning: classifier can be
updated incrementally
♦ Just add new training instance!

●
Can we do the same with kD-trees?
●
Heuristic strategy:
♦ Find leaf node containing new instance

♦ Place instance into leaf if leaf is empty

♦ Otherwise, split leaf according to the longest dimension

(to preserve squareness)

●
Tree should be re-built occasionally (i.e. if depth grows to
twice the optimum depth)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 96

 Ball trees
●
Problem in kD-trees: corners
●
Observation: no need to make sure that regions
don't overlap
●
Can use balls (hyperspheres) instead of
hyperrectangles
♦ A ball tree organizes the data into a tree of k-
dimensional hyperspheres
♦ Normally allows for a better fit to the data and thus
more efficient search

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 97

 Ball tree example

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 98

 Using ball trees
●
Nearest-neighbor search is done using the same backtracking
strategy as in kD-trees
●
Ball can be ruled out from consideration if: distance from
target to ball's center exceeds ball's radius plus current upper
bound

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 99

 Building ball trees
●
Ball trees are built top down (like kD-trees)
●
Don't have to continue until leaf balls contain just two points:
can enforce minimum occupancy
(same in kD-trees)
●
Basic problem: splitting a ball into two
●
Simple (linear-time) split selection strategy:
♦ Choose point farthest from ball's center
♦ Choose second point farthest from first one
♦ Assign each point to these two points
♦ Compute cluster centers and radii based on the two subsets to
get two balls

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 100
 Discussion of nearest-neighbor learning

● Often very accurate

● Assumes all attributes are equally important
● Remedy: attribute selection or weights
● Possible remedies against noisy instances:
● Take a majority vote over the k nearest neighbors
● Removing noisy instances from dataset (difficult!)
● Statisticians have used k-NN since early 1950s
●
If n → ∞ and k/n → 0, error approaches minimum
● kD-trees become inefficient when number of attributes is
too large (approximately > 10)
● Ball trees (which are instances of metric trees) work well
in higher-dimensional spaces

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 101
 More discussion
●
Instead of storing all training instances, compress them into
regions
●
Example: hyperpipes (from discussion of 1R)
●
Another simple technique (Voting Feature Intervals):
♦ Construct intervals for each attribute
● Discretize numeric attributes

● Treat each value of a nominal attribute as an “interval”

♦ Count number of times class occurs in interval

♦ Prediction is generated by letting intervals vote (those that contain
the test instance)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 102
 Clustering
●
Clustering techniques apply when there is no class to be
predicted
●
Aim: divide instances into “natural” groups
●
As we've seen clusters can be:
♦ disjoint vs. overlapping

♦ deterministic vs. probabilistic

♦ flat vs. hierarchical

●
We'll look at a classic clustering algorithm called k-means
♦ k-means clusters are disjoint, deterministic, and flat

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 103
 The k-means algorithm

To cluster data into k groups:

(k is predefined)
0. Choose k cluster centers
♦ e.g. at random
1. Assign instances to clusters
♦ based on distance to cluster centers
2. Compute centroids of clusters
3. Go to step 1
♦ until convergence

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 104
 Discussion
●
Algorithm minimizes squared distance to cluster centers
●
Result can vary significantly
♦ based on initial choice of seeds

●
Can get trapped in local minimum
♦ Example: initial
cluster
centres

instances

●
To increase chance of finding global optimum: restart with
different random seeds
●
Can we applied recursively with k = 2

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 105
 Faster distance calculations
●
Can we use kD-trees or ball trees to speed up the
process? Yes:
♦ First, build tree, which remains static, for all the
data points
♦ At each node, store number of instances and sum of
all instances
♦ In each iteration, descend tree and find out which
cluster each node belongs to
● Can stop descending as soon as we find out that a node
belongs entirely to a particular cluster
● Use statistics stored at the nodes to compute new
cluster centers

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 106
 Example

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 107
 Multi-instance learning
●
Simplicity-first methodology can be applied to
multi-instance learning with surprisingly good
results
●
Two simple approaches, both using standard
single-instance learners:
♦ Manipulate the input to learning
♦ Manipulate the output of learning

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 108
 Aggregating the input
●
Convert multi-instance problem into single-instance one
♦ Summarize the instances in a bag by computing mean,
mode, minimum and maximum as new attributes
♦ “Summary” instance retains the class label of its bag
♦ To classify a new bag the same process is used
●
Results using summary instances with minimum and
maximum + support vector machine classifier are
comparable to special purpose multi-instance learners on
original drug discovery problem

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 109
 Aggregating the output
●
Learn a single-instance classifier directly from the original
instances in each bag
♦ Each instance is given the class of the bag it originates from
●
To classify a new bag:
♦ Produce a prediction for each instance in the bag
♦ Aggregate the predictions to produce a prediction for the bag as
a whole
♦ One approach: treat predictions as votes for the various class
labels
♦ A problem: bags can contain differing numbers of instances →
give each instance a weight inversely proportional to the bag's
size

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 110
 Comments on basic methods
●
Bayes’ rule stems from his “Essay towards solving a
problem in the doctrine of chances” (1763)
♦ Difficult bit in general: estimating prior probabilities (easy in the
case of naïve Bayes)
●
Extension of naïve Bayes: Bayesian networks (which we'll
discuss later)
●
Algorithm for association rules is called APRIORI
●
Minsky and Papert (1969) showed that linear classifiers
have limitations, e.g. can’t learn XOR
♦ But: combinations of them can (→ multi-layer neural nets,
which we'll discuss later)

Data Mining: Practical Machine Learning Tools and Techniques (Chapter 4) 111