Chapter 3:

Supervised
Learning
CS583, Bing Liu, UIC 2
Road Map

Basic concepts

Decision tree induction

Evaluation of classifiers

Rule induction

Classification using association rules

Nave Bayesian classification

Nave Bayes for text classification

Support vector machines

K-nearest neighbor

Summary
CS583, Bing Liu, UIC 3
An example application

An emergency room in a hospital measures 17

variables (e.g., blood pressure, age, etc) of newly
admitted patients.

A decision is needed: whether to put a new patient

in an intensive-care unit.

Due to the high cost of ICU, those patients who may

survive less than a month are given higher priority.

Problem: to predict high-risk patients and

discriminate them from low-risk patients.
CS583, Bing Liu, UIC 4
Another application

A credit card company receives thousands of

applications for new cards. Each application contains
information about an applicant,

age

Marital status

annual salary

outstanding debts

credit rating

etc.

Problem: to decide whether an application should

approved, or to classify applications into two
categories, approved and not approved.
CS583, Bing Liu, UIC 5
Machine learning and our
focus

Like human learning from past experiences.

A computer does not have experiences.

A computer system learns from data, which

represent some past experiences of an application
domain.

Our focus: learn a target function that can be used

to predict the values of a discrete class attribute,
e.g., approve or not-approved, and high-risk or low
risk.

The task is commonly called: Supervised learning,

classification, or inductive learning.
CS583, Bing Liu, UIC 6

Data: A set of data records (also called

examples, instances or cases) described by

k attributes: A
1
, A
2
, A
k
.

a class: Each example is labelled with a pre-

defined class.

Goal: To learn a classification model from the

data that can be used to predict the classes
of new (future, or test) cases/instances.
The data and the goal
CS583, Bing Liu, UIC 7
An example: data (loan
application)
Approved or not
CS583, Bing Liu, UIC 8
An example: the learning task

Learn a classification model from the data

Use the model to classify future loan applications

into

Yes (approved) and

No (not approved)

What is the class for following case/instance?

CS583, Bing Liu, UIC 9
Supervised vs. unsupervised
Learning

Supervised learning: classification is seen as

supervised learning from examples.

Supervision: The data (observations,

measurements, etc.) are labeled with pre-defined
classes. It is like that a teacher gives the classes
(supervision).

Test data are classified into these classes too.

Unsupervised learning (clustering)

Class labels of the data are unknown

Given a set of data, the task is to establish the

existence of classes or clusters in the data
CS583, Bing Liu, UIC 10
Supervised learning process:
two steps

Learning (training): Learn a model using the

training data

Testing: Test the model using unseen test

data to assess the model accuracy
,
cases test of number Total
tions classifica correct of Number
Accuracy
CS583, Bing Liu, UIC 11
What do we mean by
learning?

Given

a data set D,

a task T, and

a performance measure M,
a computer system is said to learn from D to
perform the task T if after learning the
systems performance on T improves as
measured by M.

In other words, the learned model helps the

system to perform T better as compared to
no learning.
CS583, Bing Liu, UIC 12
An example

Data: Loan application data

Task: Predict whether a loan should be

approved or not.

Performance measure: accuracy.

No learning: classify all future applications (test
data) to the majority class (i.e., Yes):
Accuracy = 9/15 = 60%.

We can do better than 60% with learning.

CS583, Bing Liu, UIC 13
Fundamental assumption of
learning
Assumption: The distribution of training
examples is identical to the distribution of test
examples (including future unseen
examples).

In practice, this assumption is often violated

to certain degree.

Strong violations will clearly result in poor

classification accuracy.

To achieve good accuracy on the test data,

training examples must be sufficiently
representative of the test data.
CS583, Bing Liu, UIC 14
Road Map

Basic concepts

Decision tree induction

Evaluation of classifiers

Rule induction

Classification using association rules

Nave Bayesian classification

Nave Bayes for text classification

Support vector machines

K-nearest neighbor

Summary
CS583, Bing Liu, UIC 15
Introduction

Decision tree learning is one of the most

widely used techniques for classification.

Its classification accuracy is competitive with

other methods, and

it is very efficient.

The classification model is a tree, called

decision tree.

C4.5 by Ross Quinlan is perhaps the best

known system. It can be downloaded from
the Web.
CS583, Bing Liu, UIC 16
The loan data (reproduced)
Approved or not
CS583, Bing Liu, UIC 17
A decision tree from the loan
data

Decision nodes and leaf nodes (classes)

CS583, Bing Liu, UIC 18
Use the decision tree
No
CS583, Bing Liu, UIC 19
Is the decision tree unique?

No. Here is a simpler tree.

We want smaller tree and accurate tree.

Easy to understand and perform better.

Finding the best tree is

NP-hard.

All current tree building

algorithms are heuristic
algorithms
CS583, Bing Liu, UIC 20
From a decision tree to a set
of rules

A decision tree can

be converted to a
set of rules

Each path from the

root to a leaf is a
rule.
CS583, Bing Liu, UIC 21
Algorithm for decision tree
learning

Basic algorithm (a greedy divide-and-conquer algorithm)

Assume attributes are categorical now (continuous attributes

can be handled too)

Tree is constructed in a top-down recursive manner

At start, all the training examples are at the root

Examples are partitioned recursively based on selected

attributes

Attributes are selected on the basis of an impurity function

(e.g., information gain)

Conditions for stopping partitioning

All examples for a given node belong to the same class

There are no remaining attributes for further partitioning

majority class is the leaf

There are no examples left

CS583, Bing Liu, UIC 22
Decision tree learning
algorithm
CS583, Bing Liu, UIC 23
Choose an attribute to
partition data

The key to building a decision tree - which

attribute to choose in order to branch.

The objective is to reduce impurity or

uncertainty in data as much as possible.

A subset of data is pure if all instances belong to

the same class.

The heuristic in C4.5 is to choose the attribute

with the maximum Information Gain or Gain
Ratio based on information theory.
CS583, Bing Liu, UIC 24
The loan data (reproduced)
Approved or not
CS583, Bing Liu, UIC 25
Two possible roots, which is
better?

Fig. (B) seems to be better.

CS583, Bing Liu, UIC 26
Information theory

Information theory provides a mathematical

basis for measuring the information content.

To understand the notion of information, think

about it as providing the answer to a question,
for example, whether a coin will come up heads.

If one already has a good guess about the answer,

then the actual answer is less informative.

If one already knows that the coin is rigged so that it

will come with heads with probability 0.99, then a
message (advanced information) about the actual
outcome of a flip is worth less than it would be for a
honest coin (50-50).
CS583, Bing Liu, UIC 27
Information theory (cont )

For a fair (honest) coin, you have no

information, and you are willing to pay more
(say in terms of $) for advanced information -
less you know, the more valuable the
information.

Information theory uses this same intuition,

but instead of measuring the value for
information in dollars, it measures information
contents in bits.

One bit of information is enough to answer a

yes/no question about which one has no
idea, such as the flip of a fair coin
CS583, Bing Liu, UIC 28
Information theory: Entropy
measure

The entropy formula,

Pr(c
j
) is the probability of class c
j
in data set D

We use entropy as a measure of impurity or

disorder of data set D. (Or, a measure of
information in a tree)
, 1 ) Pr(
) Pr( log ) Pr( ) (
| |
1
| |
1
2

C
j
j
j
C
j
j
c
c c D entropy
CS583, Bing Liu, UIC 29
Entropy measure: let us get a
feeling

As the data become purer and purer, the entropy value

becomes smaller and smaller. This is useful to us!
CS583, Bing Liu, UIC 30
Information gain

Given a set of examples D, we first compute its

entropy:

If we make attribute A
i
, with v values, the root of the
current tree, this will partition D into v subsets D
1
, D
2

, D
v
. The expected entropy if A
i
is used as the
current root:

v
j
j
j
A
D entropy
D
D
D entropy
i
1
) (
| |
| |
) (
CS583, Bing Liu, UIC 31
Information gain (cont )

Information gained by selecting attribute A

i
to
branch or to partition the data is

We choose the attribute with the highest gain to

branch/split the current tree.
) ( ) ( ) , ( D entropy D entropy A D gain
i
A i

CS583, Bing Liu, UIC 32
An example
Age Yes No entropy(Di)
young 2 3 0.971
middle 3 2 0.971
old 4 1 0.722

Own_house is the best

choice for the root.
971 . 0
15
9
log
15
9
15
6
log
15
6
) (
2 2
D entropy
551 . 0
918 . 0
15
9
0
15
6

) (
15
9
) (
15
6
) (
2 1 _

+
D entropy D entropy D entropy
house Own
888 . 0
722 . 0
15
5
971 . 0
15
5
971 . 0
15
5

) (
15
5
) (
15
5
) (
15
5
) (
3 2 1

+ +
D entropy D entropy D entropy D entropy
Age
CS583, Bing Liu, UIC 33
We build the final tree

We can use information gain ratio to evaluate the

impurity as well (see the handout)
CS583, Bing Liu, UIC 34
Handling continuous
attributes

Handle continuous attribute by splitting into

two intervals (can be more) at each node.

How to find the best threshold to divide?

Use information gain or gain ratio again

Sort all the values of an continuous attribute in

increasing order {v
1
, v
2
, , v
r
},

One possible threshold between two adjacent

values v
i
and v
i+1
. Try all possible thresholds and
find the one that maximizes the gain (or gain ratio).
CS583, Bing Liu, UIC 35
An example in a continuous
space
CS583, Bing Liu, UIC 36
Avoid overfitting in
classification

Overfitting: A tree may overfit the training data

Good accuracy on training data but poor on test data

Symptoms: tree too deep and too many branches, some

may reflect anomalies due to noise or outliers

Two approaches to avoid overfitting

Pre-pruning: Halt tree construction early

Difficult to decide because we do not know what may

happen subsequently if we keep growing the tree.

Post-pruning: Remove branches or sub-trees from a

fully grown tree.

This method is commonly used. C4.5 uses a statistical

method to estimates the errors at each node for pruning.

A validation set may be used for pruning as well.

CS583, Bing Liu, UIC 37
An example
Likely to overfit the data
CS583, Bing Liu, UIC 38
Other issues in decision tree
learning

From tree to rules, and rule pruning

Handling of miss values

Handing skewed distributions

Handling attributes and classes with different

costs.

Attribute construction

Etc.
CS583, Bing Liu, UIC 39
Road Map

Basic concepts

Decision tree induction

Evaluation of classifiers

Rule induction

Classification using association rules

Nave Bayesian classification

Nave Bayes for text classification

Support vector machines

K-nearest neighbor

Summary
CS583, Bing Liu, UIC 40
Evaluating classification
methods

Predictive accuracy

Efficiency

time to construct the model

time to use the model

Robustness: handling noise and missing values

Scalability: efficiency in disk-resident databases

Interpretability:

understandable and insight provided by the model

Compactness of the model: size of the tree, or the

number of rules.
CS583, Bing Liu, UIC 41
Evaluation methods

Holdout set: The available data set D is divided into

two disjoint subsets,

the training set D

train
(for learning a model)

the test set D

test
(for testing the model)

Important: training set should not be used in testing

and the test set should not be used in learning.

Unseen test set provides a unbiased estimate of accuracy.

The test set is also called the holdout set. (the

examples in the original data set D are all labeled
with classes.)

This method is mainly used when the data set D is

large.
CS583, Bing Liu, UIC 42
Evaluation methods (cont)

n-fold cross-validation: The available data is

partitioned into n equal-size disjoint subsets.

Use each subset as the test set and combine the rest
n-1 subsets as the training set to learn a classifier.

The procedure is run n times, which give n accuracies.

The final estimated accuracy of learning is the average

of the n accuracies.

10-fold and 5-fold cross-validations are commonly

used.

This method is used when the available data is not

large.
CS583, Bing Liu, UIC 43
Evaluation methods (cont)

Leave-one-out cross-validation: This

method is used when the data set is very
small.

It is a special case of cross-validation

Each fold of the cross validation has only a

single test example and all the rest of the
data is used in training.

If the original data has m examples, this is m-

fold cross-validation
CS583, Bing Liu, UIC 44
Evaluation methods (cont)

Validation set: the available data is divided into

three subsets,

a training set,

a validation set and

a test set.

A validation set is used frequently for estimating

parameters in learning algorithms.

In such cases, the values that give the best accuracy

on the validation set are used as the final parameter
values.

Cross-validation can be used for parameter

estimating as well.
CS583, Bing Liu, UIC 45
Classification measures

Accuracy is only one measure (error = 1-accuracy).

Accuracy is not suitable in some applications.

In text mining, we may only be interested in the

documents of a particular topic, which are only a
small portion of a big document collection.

In classification involving skewed or highly

imbalanced data, e.g., network intrusion and financial
fraud detections, we are interested only in the
minority class.

High accuracy does not mean any intrusion is detected.

E.g., 1% intrusion. Achieve 99% accuracy by doing nothing.

The class of interest is commonly called the positive

class, and the rest negative classes.
CS583, Bing Liu, UIC 46
Precision and recall
measures

Used in information retrieval and text classification.

We use a confusion matrix to introduce them.

CS583, Bing Liu, UIC 47
Precision and recall
measures (cont)

Precision p is the number of correctly classified

positive examples divided by the total number of
examples that are classified as positive.

Recall r is the number of correctly classified positive

examples divided by the total number of actual
positive examples in the test set.
. .
FN TP
TP
r
FP TP
TP
p
+

CS583, Bing Liu, UIC 48

An example

This confusion matrix gives

precision p = 100% and

recall r = 1%
because we only classified one positive example correctly
and no negative examples wrongly.

Note: precision and recall only measure

classification on the positive class.
CS583, Bing Liu, UIC 49
F
1
-value (also called F
1
-score)

It is hard to compare two classifiers using two measures. F

1

score combines precision and recall into one measure

The harmonic mean of two numbers tends to be closer to the

smaller of the two.

For F
1
-value to be large, both p and r much be large.
CS583, Bing Liu, UIC 50
Another evaluation method:
Scoring and ranking

Scoring is related to classification.

We are interested in a single class (positive

class), e.g., buyers class in a marketing
database.

Instead of assigning each test instance a

definite class, scoring assigns a probability
estimate (PE) to indicate the likelihood that the
example belongs to the positive class.
CS583, Bing Liu, UIC 51
Ranking and lift analysis

After each example is given a PE score, we can

rank all examples according to their PEs.

We then divide the data into n (say 10) bins. A

lift curve can be drawn according how many
positive examples are in each bin. This is called
lift analysis.

Classification systems can be used for scoring.

Need to produce a probability estimate.

E.g., in decision trees, we can use the confidence value at

each leaf node as the score.
CS583, Bing Liu, UIC 52
An example

We want to send promotion materials to

potential customers to sell a watch.

Each package cost $0.50 to send (material

and postage).

If a watch is sold, we make $5 profit.

Suppose we have a large amount of past

data for building a predictive/classification
model. We also have a large list of potential
customers.

How many packages should we send and

who should we send to?
CS583, Bing Liu, UIC 53
An example

Assume that the test set has 10000

instances. Out of this, 500 are positive cases.

After the classifier is built, we score each test

instance. We then rank the test set, and
divide the ranked test set into 10 bins.

Each bin has 1000 test instances.

Bin 1 has 210 actual positive instances

Bin 2 has 120 actual positive instances

Bin 3 has 60 actual positive instances

Bin 10 has 5 actual positive instances

CS583, Bing Liu, UIC 54
Lift curve
210 120 60 40 22 18 12 7 6 5
42% 24% 12% 8% 4.40% 3.60% 2.40% 1.40% 1.20% 1%
42% 66% 78% 86% 90.40% 94% 96.40% 97.80% 99% 100%
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Percent of testing cases
P
e
r
c
e
n
t

o
f

t
o
t
a
l

p
o
s
i
t
i
v
e

c
a
s
e
s
lift
random
Bin 1 2 3 4 5 6 7 8 9 10
CS583, Bing Liu, UIC 55
Road Map

Basic concepts

Decision tree induction

Evaluation of classifiers

Rule induction

Classification using association rules

Nave Bayesian classification

Nave Bayes for text classification

Support vector machines

K-nearest neighbor

Summary
CS583, Bing Liu, UIC 56
Introduction

We showed that a decision tree can be

converted to a set of rules.

Can we find if-then rules directly from data

for classification?

Yes.

Rule induction systems find a sequence of

rules (also called a decision list) for
classification.

The commonly used strategy is sequential

covering.
CS583, Bing Liu, UIC 57
Sequential covering

Learn one rule at a time, sequentially.

After a rule is learned, the training examples

covered by the rule are removed.

Only the remaining data are used to find

subsequent rules.

The process repeats until some stopping

criteria are met.
Note: a rule covers an example if the example
satisfies the conditions of the rule.

We introduce two specific algorithms.

CS583, Bing Liu, UIC 58
Algorithm 1: ordered rules

The final classifier:

<r
1
, r
2
, , r
k
, default-class>
CS583, Bing Liu, UIC 59
Algorithm 2: ordered classes

Rules of the same class are together.

CS583, Bing Liu, UIC 60
Algorithm 1 vs. Algorithm 2

Differences:

Algorithm 2: Rules of the same class are found

together. The classes are ordered. Normally,
minority class rules are found first.

Algorithm 1: In each iteration, a rule of any class

may be found. Rules are ordered according to the
sequence they are found.

Use of rules: the same.

For a test instance, we try each rule sequentially.

The first rule that covers the instance classifies it.

If no rule covers it, default class is used, which is

the majority class in the data.
CS583, Bing Liu, UIC 61
Learn-one-rule-1 function

Let us consider only categorical attributes

Let attributeValuePairs contains all possible

attribute-value pairs (A
i
= a
i
) in the data.

Iteration 1: Each attribute-value is evaluated

as the condition of a rule. I.e., we compare all
such rules A
i
= a
i
c
j
and keep the best one,

Evaluation: e.g., entropy

Also store the k best rules for beam search (to

search more space). Called new candidates.
CS583, Bing Liu, UIC 62
Learn-one-rule-1 function
(cont )

In iteration m, each (m-1)-condition rule in the

new candidates set is expanded by attaching
each attribute-value pair in attributeValuePairs
as an additional condition to form candidate
rules.

These new candidate rules are then evaluated

in the same way as 1-condition rules.

Update the best rule

Update the k-best rules

The process repeats unless stopping criteria

are met.
CS583, Bing Liu, UIC 63
Learn-one-rule-1 algorithm
CS583, Bing Liu, UIC 64
Learn-one-rule-2 function

Split the data:

Pos -> GrowPos and PrunePos

Neg -> GrowNeg and PruneNeg

Grow sets are used to find a rule (BestRule),

and the Prune sets are used to prune the rule.

GrowRule works similarly as in learn-one-rule-

1, but the class is fixed in this case. Recall the
second algorithm finds all rules of a class first
(Pos) and then moves to the next class.
CS583, Bing Liu, UIC 65
Learn-one-rule-2 algorithm
CS583, Bing Liu, UIC 66
Rule evaluation in learn-one-
rule-2

Let the current partially developed rule be:

R: av
1
, .., av
k
class

where each av
j
is a condition (an attribute-value pair).

By adding a new condition av

k+1
, we obtain the rule
R+: av
1
, .., av
k
, av
k+1
class.

The evaluation function for R+ is the following

information gain criterion (which is different from the
gain function used in decision tree learning).

Rule with the best gain is kept for further extension.

,
_

+

+
0 0
0
1 1
1
1
2 2
log log ) , (
n p
p
n p
p
p R R gain
CS583, Bing Liu, UIC 67
Rule pruning in learn-one-rule-
2

Consider deleting every subset of conditions from the

BestRule, and choose the deletion that maximizes the
function:
where p (n) is the number of examples in PrunePos
(PruneNeg) covered by the current rule (after a
deletion).
n p
n p
PruneNeg PrunePos BestRule v
+

) , , (
CS583, Bing Liu, UIC 68
Discussions

Accuracy: similar to decision tree

Efficiency: Run much slower than decision tree

induction because

To generate each rule, all possible rules are tried on the

data (not really all, but still a lot).

When the data is large and/or the number of attribute-value

pairs are large. It may run very slowly.

Rule interpretability: Can be a problem because each

rule is found after data covered by previous rules are
removed. Thus, each rule may not be treated as
independent of other rules.
CS583, Bing Liu, UIC 69
Road Map

Basic concepts

Decision tree induction

Evaluation of classifiers

Rule induction

Classification using association rules

Nave Bayesian classification

Nave Bayes for text classification

Support vector machines

K-nearest neighbor

Summary
CS583, Bing Liu, UIC 70
Association rules for
classification

Classification: mine a small set of rules

existing in the data to form a classifier or
predictor.

It has a target attribute: Class attribute

Association rules: have no fixed target, but

we can fix a target.

Class association rules (CAR): has a target

class attribute. E.g.,
Own_house = true Class =Yes [sup=6/15, conf=6/6]

CARs can obviously be used for classification.

CS583, Bing Liu, UIC 71
Decision tree vs. CARs

The decision tree below generates the following 3 rules.

Own_house = true Class =Yes [sup=6/15, conf=6/6]
Own_house = false, Has_job = true Class=Yes [sup=5/15, conf=5/5]
Own_house = false, Has_job = false Class=No [sup=4/15, conf=4/4]

But there are many other

rules that are not found by
the decision tree
CS583, Bing Liu, UIC 72
There are many more rules

CAR mining finds all of

them.

In many cases, rules

not in the decision tree
(or a rule list) may
perform classification
better.

Such rules may also be

actionable in practice
CS583, Bing Liu, UIC 73
Decision tree vs. CARs (cont
)

Association mining require discrete attributes.

Decision tree learning uses both discrete and
continuous attributes.

CAR mining requires continuous attributes

discretized. There are several such algorithms.

Decision tree is not constrained by minsup or

minconf, and thus is able to find rules with
very low support. Of course, such rules may
be pruned due to the possible overfitting.
CS583, Bing Liu, UIC 74
Considerations in CAR
mining

Multiple minimum class supports

Deal with imbalanced class distribution, e.g.,

some class is rare, 98% negative and 2%
positive.

We can set the minsup(positive) = 0.2% and

minsup(negative) = 2%.

If we are not interested in classification of

negative class, we may not want to generate
rules for negative class. We can set
minsup(negative)=100% or more.

Rule pruning may be performed.

CS583, Bing Liu, UIC 75
Building classifiers

There are many ways to build classifiers using

CARs. Several existing systems available.

Simplest: After CARs are mined, do nothing.

For each test case, we simply choose the most

confident rule that covers the test case to classify it.
Microsoft SQL Server has a similar method.

Or, using a combination of rules.

Another method (used in the CBA system) is

similar to sequential covering.

Choose a set of rules to cover the training data.

CS583, Bing Liu, UIC 76
Rules are sorted first
Definition: Given two rules, r
i
and r
j
, r
i
r
j
(also
called ri precedes r
j
or r
i
has a higher
precedence than r
j
) if

the confidence of r
i
is greater than that of r
j
, or

their confidences are the same, but the support of

r
i
is greater than that of r
j
, or

both the confidences and supports of r

i
and r
j
are
the same, but r
i
is generated earlier than r
j
.
A CBA classifier L is of the form:
L = <r
1
, r
2
, , r
k
, default-class>
CS583, Bing Liu, UIC 77
Classifier building using CARs

This algorithm is very inefficient

CBA has very efficient algorithm that scans the

data at most two times (quite involved).
CS583, Bing Liu, UIC 78
Road Map

Basic concepts

Decision tree induction

Evaluation of classifiers

Rule induction

Classification using association rules

Nave Bayesian classification

Nave Bayes for text classification

Support vector machines

K-nearest neighbor

Summary
CS583, Bing Liu, UIC 79
Bayesian classification

Probabilistic view: Supervised learning can naturally

be studied from a probabilistic point of view.

Let A
1
through A
k
be attributes with discrete values.
The class is C.

Given a test example d with observed attribute values

a
1
through a
k
.

Classification is basically to compute the following

posteriori probability. The prediction is the class c
j

such that
is maximal
CS583, Bing Liu, UIC 80
Apply Bayes Rule

Pr(C=c
j
) is the class prior probability: easy to
estimate from the training data.

| |
1
| | | | 1 1
| | | | 1 1
| | | | 1 1
| | | | 1 1
| | | | 1 1
) Pr( ) | ,..., Pr(
) Pr( ) | ,..., Pr(
) ,..., Pr(
) Pr( ) | ,..., Pr(
) ,..., | Pr(
C
r
r r A A
j j A A
A A
j j A A
A A j
c C c C a A a A
c C c C a A a A
a A a A
c C c C a A a A
a A a A c C
CS583, Bing Liu, UIC 81
Computing probabilities

The denominator P(A

1
=a
1
,...,A
k
=a
k
) is irrelevant
for decision making since it is the same for
every class.

We only need P(A

1
=a
1
,...,A
k
=a
k
| C=c
i
), which
can be written as
Pr(A
1
=a
1
|A
2
=a
2
,...,A
k
=a
k
,

C=c
j
)* Pr(A
2
=a
2
,...,A
k
=a
k
|C=c
j
)

Recursively, the second factor above can be

written in the same way, and so on.

Now an assumption is needed.

CS583, Bing Liu, UIC 82
Conditional independence
assumption

All attributes are conditionally independent

given the class C = c
j
.

Formally, we assume,
Pr(A
1
=a
1
| A
2
=a
2
, ..., A
|A|
=a
|A|
, C=c
j
) = Pr(A
1
=a
1
| C=c
j
)
and so on for A
2
through A
|A|
. I.e.,

| |
1
| | | | 1 1
) | Pr( ) | ,..., Pr(
A
i
j i i i A A
c C a A c C a A a A
CS583, Bing Liu, UIC 83
Final nave Bayesian classifier

We are done!

How do we estimate P(A

i
= a
i
|

C=c
j
)? Easy!.

| |
1
| |
1
| |
1
| | | | 1 1
) | Pr( ) Pr(
) | Pr( ) Pr(
) ,..., | Pr(
C
r
A
i
r i i r
A
i
j i i j
A A j
c C a A c C
c C a A c C
a A a A c C
CS583, Bing Liu, UIC 84
Classify a test instance

If we only need a decision on the most

probable class for the test instance, we only
need the numerator as its denominator is the
same for every class.

Thus, given a test example, we compute the

following to decide the most probable class
for the test instance

| |
1
) | Pr( ) Pr( max arg
A
i
j i i j
c
c C a A c c
j
CS583, Bing Liu, UIC 85
An example

Compute all probabilities

required for classification
CS583, Bing Liu, UIC 86
An Example (cont )

For C = t, we have

For class C = f, we have

C = t is more probable. t is the final class.

25
2
5
2
5
2
2
1
) | Pr( ) Pr(
2
1


j
j j
t C a A t C
25
1
5
2
5
1
2
1
) | Pr( ) Pr(
2
1


j
j j
f C a A f C
CS583, Bing Liu, UIC 87
Additional issues

Numeric attributes: Nave Bayesian

learning assumes that all attributes are
categorical. Numeric attributes need to be
discretized.

Zero counts: An particular attribute value

never occurs together with a class in the
training set. We need smoothing.

Missing values: Ignored

i j
ij
j i i
n n
n
c C a A

+
+
) | Pr(
CS583, Bing Liu, UIC 88
On nave Bayesian classifier

Advantages:

Easy to implement

Very efficient

Good results obtained in many applications

Disadvantages

Assumption: class conditional independence,

therefore loss of accuracy when the assumption
is seriously violated (those highly correlated data
sets)
CS583, Bing Liu, UIC 89
Road Map

Basic concepts

Decision tree induction

Evaluation of classifiers

Rule induction

Classification using association rules

Nave Bayesian classification

Nave Bayes for text classification

Support vector machines

K-nearest neighbor

Summary
CS583, Bing Liu, UIC 90
Text
classification/categorization

Due to the rapid growth of online documents in

organizations and on the Web, automated document
classification has become an important problem.

Techniques discussed previously can be applied to

text classification, but they are not as effective as the
next three methods.

We first study a nave Bayesian method specifically

formulated for texts, which makes use of some text
specific features.

However, the ideas are similar to the preceding

method.
CS583, Bing Liu, UIC 91
Probabilistic framework

Generative model: Each document is

generated by a parametric distribution
governed by a set of hidden parameters.

The generative model makes two

assumptions

The data (or the text documents) are generated by

a mixture model,

There is one-to-one correspondence between

mixture components and document classes.
CS583, Bing Liu, UIC 92
Mixture model

A mixture model models the data with a

number of statistical distributions.

Intuitively, each distribution corresponds to a data

cluster and the parameters of the distribution
provide a description of the corresponding cluster.

Each distribution in a mixture model is also

called a mixture component.

The distribution/component can be of any

kind
CS583, Bing Liu, UIC 93
An example

The figure shows a plot of the probability

density function of a 1-dimensional data set
(with two classes) generated by

a mixture of two Gaussian distributions,

one per class, whose parameters (denoted by

i
) are
the mean (
i
) and the standard deviation (
i
), i.e.,
i
=
(
i
,
i
).
class 1 class 2
CS583, Bing Liu, UIC 94
Mixture model (cont )

Let the number of mixture components (or

distributions) in a mixture model be K.

Let the jth distribution have the parameters

j
.

Let be the set of parameters of all

components, = {
1
,
2
, ,
K
,
1
,
2
, ,

K
}, where
j
is the mixture weight (or mixture
probability) of the mixture component j and
j

is the parameters of component j.

How does the model generate documents?

CS583, Bing Liu, UIC 95
Document generation

Due to one-to-one correspondence, each class

corresponds to a mixture component. The mixture
weights are class prior probabilities, i.e.,
j
= Pr(c
j
|).

The mixture model generates each document d

i
by:

first selecting a mixture component (or class) according to

class prior probabilities (i.e., mixture weights),
j
= Pr(c
j
|).

then having this selected mixture component (c

j
) generate a
document d
i
according to its parameters, with distribution
Pr(d
i
|c
j
; ) or more precisely Pr(d
i
|c
j
;
j
).
) ; | Pr( ) | Pr( ) | Pr(
| |
1

C
j
j i j i
c d c d (23)
CS583, Bing Liu, UIC 96
Model text documents

The nave Bayesian classification treats each

document as a bag of words. The
generative model makes the following further
assumptions:

Words of a document are generated independently

of context given the class label. The familiar nave
Bayes assumption used before.

The probability of a word is independent of its

position in the document. The document length is
chosen independent of its class.
CS583, Bing Liu, UIC 97
Multinomial distribution

With the assumptions, each document can be

regarded as generated by a multinomial
distribution.

In order words, each document is drawn from

a multinomial distribution of words with as
many independent trials as the length of the
document.

The words are from a given vocabulary V =

{w
1
, w
2
, , w
|V|
}.
CS583, Bing Liu, UIC 98
Use probability function of
multinomial distribution
where N
ti
is the number of times that word w
t

occurs in document d
i
and

| |
1
!
) ; | Pr(
|! | |) Pr(| ) ; | Pr(
V
t
ti
ti
N
j t
i i j i
N
c w
d d c d
| |
| |
1
i
V
t
it d N

. 1 ) ; | Pr(
| |
1

V
t
j t c w
(24)
(25)
CS583, Bing Liu, UIC 99
Parameter estimation

The parameters are estimated based on empirical

counts.

In order to handle 0 counts for infrequent occurring

words that do not appear in the training set, but may
appear in the test set, we need to smooth the
probability. Lidstone smoothing, 0 1
.
) | Pr(
) | Pr(
)

; | Pr(
| |
1
| |
1
| |
1

V
s
D
i
i j si
D
i
i j ti
j t
d c N
d c N
c w
.
) | Pr( | |
) | Pr(
)

; | Pr(
| |
1
| |
1
| |
1

+
+

V
s
D
i
i j si
D
i
i j ti
j t
d c N V
d c N
c w

(26)
(27)
CS583, Bing Liu, UIC 100
Parameter estimation (cont
)

Class prior probabilities, which are mixture

weights
j
, can be easily estimated using
training data
| |
) | Pr(
)

| Pr(
| |
1
D
d c
c
D
i
i j
j

(28)
CS583, Bing Liu, UIC 101
Classification

Given a test document d

i,
from Eq. (23) (27) and (28)

| |
1
| |
1
,
| |
1
,
)

; | Pr( ) Pr(
)

; | Pr( )

| Pr(

)

| Pr(
)

; | Pr( )

| Pr(
)

; | Pr(
C
r
d
k
r k d
d
k
k d
i
i
r
i
j
i
j
i
j i j
i j
c w c
c w c
d
c d c
d c
CS583, Bing Liu, UIC 102
Discussions

Most assumptions made by nave Bayesian

learning are violated to some degree in
practice.

Despite such violations, researchers have

shown that nave Bayesian learning produces
very accurate models.

The main problem is the mixture model

assumption. When this assumption is seriously
violated, the classification performance can be
poor.

Nave Bayesian learning is extremely

efficient.
CS583, Bing Liu, UIC 103
Road Map

Basic concepts

Decision tree induction

Evaluation of classifiers

Rule induction

Classification using association rules

Nave Bayesian classification

Nave Bayes for text classification

Support vector machines

K-nearest neighbor

Summary
CS583, Bing Liu, UIC 104
Introduction

Support vector machines were invented by V. Vapnik

and his co-workers in 1970s in Russia and became
known to the West in 1992.

SVMs are linear classifiers that find a hyperplane to

separate two class of data, positive and negative.

Kernel functions are used for nonlinear separation.

SVM not only has a rigorous theoretical foundation,

but also performs classification more accurately than
most other methods in applications, especially for
high dimensional data.

It is perhaps the best classifier for text classification.

CS583, Bing Liu, UIC 105
Basic concepts

Let the set of training examples D be

{(x
1
, y
1
), (x
2
, y
2
), , (x
r
, y
r
)},
where x
i
= (x
1
, x
2
, , x
n
) is an input vector in a real-
valued space X R
n
and y
i
is its class label (output
value), y
i
{1, -1}.
1: positive class and -1: negative class.

SVM finds a linear function of the form (w: weight

vector)
f(x) = w x + b

'

< +
+

0 1
0 1
b if
b if
y
i
i
i
x w
x w
CS583, Bing Liu, UIC 106
The hyperplane

The hyperplane that separates positive and negative

training data is
w x + b = 0

It is also called the decision boundary (surface).

So many possible hyperplanes, which one to choose?

CS583, Bing Liu, UIC 107
Maximal margin hyperplane

SVM looks for the separating hyperplane with the largest

margin.

Machine learning theory says this hyperplane minimizes the

error bound
CS583, Bing Liu, UIC 108
Linear SVM: separable case

Assume the data are linearly separable.

Consider a positive data point (x

+
, 1) and a negative
(x
-
, -1) that are closest to the hyperplane
<w x> + b = 0.

We define two parallel hyperplanes, H

+
and H
-
, that
pass through x
+
and x
-
respectively. H
+
and H
-
are
also parallel to <w x> + b = 0.
CS583, Bing Liu, UIC 109
Compute the margin

Now let us compute the distance between the two

margin hyperplanes H
+
and H
-
. Their distance is the
margin (d
+
+ d

in the figure).

Recall from vector space in algebra that the

(perpendicular) distance from a point x
i
to the
hyperplane w x + b = 0 is:
where ||w|| is the norm of w,
|| ||
| |
w
x w b
i
+
2 2
2
2
1
... || ||
n
w w w + + + > < w w w
(36)
(37)
CS583, Bing Liu, UIC 110
Compute the margin (cont )

Let us compute d
+
.

Instead of computing the distance from x

+
to the
separating hyperplane w x + b = 0, we pick up
any point x
s
on w x + b = 0 and compute the
distance from x
s
to w x
+
+ b = 1 by applying the
distance Eq. (36) and noticing w x
s
+ b = 0,
|| ||
1
|| ||
| 1 |
w w
x w
s

+
b
d
|| ||
2
w
+
+
d d margin
(38)
(39)
CS583, Bing Liu, UIC 111
A optimization problem!
Definition (Linear SVM: separable case): Given a set of
linearly separable training examples,
D = {(x
1
, y
1
), (x
2
, y
2
), , (x
r
, y
r
)}
Learning is to solve the following constrained minimization
problem,
summarizes
w x
i
+ b 1 for y
i
= 1
w x
i
+ b -1for y
i
= -1.
r i b y
i i
..., 2, 1, , 1 ) ( : Subject to
2
: Minimize
+

x w
w w
r i b y
i i
..., 2, 1, , 1 ( + x w
(40)
CS583, Bing Liu, UIC 112
Solve the constrained
minimization
Standard Lagrangian method
where
i
0 are the Lagrange multipliers.

Optimization theory says that an optimal

solution to (41) must satisfy certain
conditions, called Kuhn-Tucker conditions,
which are necessary (but not sufficient)

Kuhn-Tucker conditions play a central role in

constrained optimization.
] 1 ) ( [
2
1
1
+

b y L
i
r
i
i i P
x w w w
(41)
CS583, Bing Liu, UIC 113
Kuhn-Tucker conditions

Eq. (50) is the original set of constraints.

The complementarity condition (52) shows that only those

data points on the margin hyperplanes (i.e., H
+
and H
-
) can
have
i
> 0 since for them y
i
(w x
i
+ b) 1 = 0.

These points are called the support vectors, All the other
parameters
i
= 0.
CS583, Bing Liu, UIC 114
Solve the problem

In general, Kuhn-Tucker conditions are necessary for

an optimal solution, but not sufficient.

However, for our minimization problem with a convex

objective function and linear constraints, the Kuhn-
Tucker conditions are both necessary and sufficient
for an optimal solution.

Solving the optimization problem is still a difficult task

due to the inequality constraints.

However, the Lagrangian treatment of the convex

optimization problem leads to an alternative dual
formulation of the problem, which is easier to solve
than the original problem (called the primal).
CS583, Bing Liu, UIC 115
Dual formulation

From primal to a dual: Setting to zero the

partial derivatives of the Lagrangian (41) with
respect to the primal variables (i.e., w and
b), and substituting the resulting relations
back into the Lagrangian.

I.e., substitute (48) and (49), into the original

Lagrangian (41) to eliminate the primal variables
(55)
,
2
1
1 , 1



j i
r
j i
j i j i
r
i
i D
y y L x x
CS583, Bing Liu, UIC 116
Dual optimization prolem

This dual formulation is called the Wolfe dual.

For the convex objective function and linear constraints of

the primal, it has the property that the maximum of L
D

occurs at the same values of w, b and
i
, as the minimum
of L
P
(the primal).

Solving (56) requires numerical techniques and clever

strategies, which are beyond our scope.
CS583, Bing Liu, UIC 117
The final decision boundary

After solving (56), we obtain the values for

i
, which
are used to compute the weight vector w and the
bias b using Equations (48) and (52) respectively.

The decision boundary

Testing: Use (57). Given a test instance z,

If (58) returns 1, then the test instance z is classified

as positive; otherwise, it is classified as negative.
0 + +

b y b
sv i
i i i
x x x w (57)

,
_

+ +

sv i
i i i
b y sign b sign z x z w ) ( (58)
CS583, Bing Liu, UIC 118
Linear SVM: Non-separable
case

Linear separable case is the ideal situation.

Real-life data may have noise or errors.

Class label incorrect or randomness in the application

domain.

Recall in the separable case, the problem was

With noisy data, the constraints may not be

satisfied. Then, no solution!
r i b y
i i
..., 2, 1, , 1 ) ( : Subject to
2
: Minimize
+

x w
w w
CS583, Bing Liu, UIC 119
Relax the constraints

To allow errors in data, we relax the margin

constraints by introducing slack variables,
i

( 0) as follows:
w x
i
+ b 1
i
for y
i
= 1
w x
i
+ b 1 +
i
for y
i
= -1.

The new constraints:

Subject to: y
i
(w x
i
+ b) 1
i
, i =1, ,
r,

i
0, i =1, 2, , r.
CS583, Bing Liu, UIC 120
Geometric interpretation

Two error data points x

a
and x
b
(circled) in wrong
regions
CS583, Bing Liu, UIC 121
Penalize errors in objective
function

We need to penalize the errors in the

objective function.

A natural way of doing it is to assign an extra

cost for errors to change the objective
function to

k = 1 is commonly used, which has the

advantage that neither
i
nor its Lagrangian
multipliers appear in the dual formulation.

+

r
i
k
i
C
1
) (
2
: Minimize
w w
(60)
CS583, Bing Liu, UIC 122
New optimization problem

This formulation is called the soft-margin

SVM. The primal Lagrangian is
where
i
,
i
0 are the Lagrange multipliers
r i
r i b y
C
i
i i i
r
i
i
..., 2, 1, , 0
..., 2, 1, , 1 ) ( : Subject to
2
: Minimize
1

+
+

x w
w w
(61)


+ + +
r
i
i i i i
r
i
i i
r
i
i P
b y C L
1 1 1
] 1 ) ( [
2
1
x w w w
(62)
CS583, Bing Liu, UIC 123
Kuhn-Tucker conditions
CS583, Bing Liu, UIC 124
From primal to dual

As the linear separable case, we transform

the primal to a dual by setting to zero the
partial derivatives of the Lagrangian (62) with
respect to the primal variables (i.e., w, b
and
i
), and substituting the resulting relations
back into the Lagrangian.

Ie.., we substitute Equations (63), (64) and

(65) into the primal Lagrangian (62).

From Equation (65), C

i

i
= 0, we can
deduce that
i
C because
i
0.
CS583, Bing Liu, UIC 125
Dual

The dual of (61) is

Interestingly,
i
and its Lagrange multipliers
i
are not
in the dual. The objective function is identical to that
for the separable case.

The only difference is the constraint

i
C.
CS583, Bing Liu, UIC 126
Find primal variable values

The dual problem (72) can be solved numerically.

The resulting
i
values are then used to compute w
and b. w is computed using Equation (63) and b is
computed using the Kuhn-Tucker complementarity
conditions (70) and (71).

Since no values for

i
, we need to get around it.

From Equations (65), (70) and (71), we observe that if 0 <

i

< C then both
i
= 0 and y
i
w x
i
+ b 1 +
i
= 0. Thus, we
can use any training data point for which 0 <
i
< C and
Equation (69) (with
i
= 0) to compute b.
. 0
1
1

j
r
i
i i i
i
y
y
b x x
(73)
CS583, Bing Liu, UIC 127
(65), (70) and (71) in fact tell
us more

(74) shows a very important property of SVM.

The solution is sparse in

i
. Many training data points are
outside the margin area and their
i
s in the solution are 0.

Only those data points that are on the margin (i.e., y

i
(w x
i

+ b) = 1, which are support vectors in the separable case),
inside the margin (i.e.,
i
= C and y
i
(w x
i
+ b) < 1), or
errors are non-zero.

Without this sparsity property, SVM would not be practical for

large data sets.
CS583, Bing Liu, UIC 128
The final decision boundary

The final decision boundary is (we note that many

i
s
are 0)

The decision rule for classification (testing) is the

same as the separable case, i.e.,
sign(w x + b).

Finally, we also need determine the parameter C in

the objective function. It is normally chosen through
the use of a validation set or cross-validation.
0
1
+ +

b y b
r
i
i i i
x x x w
(75)
CS583, Bing Liu, UIC 129
How to deal with nonlinear
separation?

The SVM formulations require linear separation.

Real-life data sets may need nonlinear separation.

To deal with nonlinear separation, the same

formulation and techniques as for the linear case are
still used.

We only transform the input data into another space

(usually of a much higher dimension) so that

a linear decision boundary can separate positive and

negative examples in the transformed space,

The transformed space is called the feature space.

The original data space is called the input space.
CS583, Bing Liu, UIC 130
Space transformation

The basic idea is to map the data in the input

space X to a feature space F via a nonlinear
mapping ,

After the mapping, the original training data

set {(x
1
, y
1
), (x
2
, y
2
), , (x
r
, y
r
)} becomes:
{( (x
1
), y
1
), ( (x
2
), y
2
), , ( (x
r
), y
r
)}
) (
:
x x

F X
(76)
(77)
CS583, Bing Liu, UIC 131
Geometric interpretation

In this example, the transformed space is

also 2-D. But usually, the number of
dimensions in the feature space is much
higher than that in the input space
CS583, Bing Liu, UIC 132
Optimization problem in (61)
becomes
CS583, Bing Liu, UIC 133
An example space
transformation

Suppose our input space is 2-dimensional,

and we choose the following transformation
(mapping) from 2-D to 3-D:

The training example ((2, 3), -1) in the input

space is transformed to the following in the
feature space:
((4, 9, 8.5), -1)
) 2 , , ( ) , (
2 1
2
2
2
1 2 1
x x x x x x
CS583, Bing Liu, UIC 134
Problem with explicit
transformation

The potential problem with this explicit data

transformation and then applying the linear SVM is
that it may suffer from the curse of dimensionality.

The number of dimensions in the feature space can

be huge with some useful transformations even with
reasonable numbers of attributes in the input space.

This makes it computationally infeasible to handle.

Fortunately, explicit transformation is not needed.

CS583, Bing Liu, UIC 135
Kernel functions

We notice that in the dual formulation both

the construction of the optimal hyperplane (79) in F and

the evaluation of the corresponding decision function (80)

only require dot products (x) (z) and never the
mapped vector (x) in its explicit form. This is a crucial
point.

Thus, if we have a way to compute the dot product

(x) (z) using the input vectors x and z
directly,

no need to know the feature vector (x) or even itself.

In SVM, this is done through the use of kernel

functions, denoted by K,
K(x, z) = (x) (z)
(82)
CS583, Bing Liu, UIC 136
An example kernel function

Polynomial kernel
K(x, z) = x z
d

Let us compute the kernel with degree d = 2 in a 2-

dimensional space: x = (x
1
, x
2
) and z = (z
1
, z
2
).

This shows that the kernel x z

2
is a dot product
in a transformed feature space
(83)
, ) ( ) (
) 2 ( ) 2 (
2
) (
2 2 2 2
2 2 2 2
2 2
1
2 2
1 1
2 2
1
2 2
1 1
2
1
2
1
2
1 1
2


+ +
+
z x

z x

z z , z , z x x , x , x
z x z x z x z x
z x z x
(84)
CS583, Bing Liu, UIC 137
Kernel trick

The derivation in (84) is only for illustration

purposes.

We do not need to find the mapping function.

We can simply apply the kernel function

directly by

replace all the dot products (x) (z) in (79)

and (80) with the kernel function K(x, z) (e.g., the
polynomial kernel x z
d
in (83)).

This strategy is called the kernel trick.

CS583, Bing Liu, UIC 138
Is it a kernel function?

The question is: how do we know whether a

function is a kernel without performing the
derivation such as that in (84)? I.e,

How do we know that a kernel function is indeed a

dot product in some feature space?

This question is answered by a theorem

called the Mercers theorem, which we will
not discuss here.
CS583, Bing Liu, UIC 139
Commonly used kernels

It is clear that the idea of kernel generalizes the dot

product in the input space. This dot product is also a
kernel with the feature map being the identity
CS583, Bing Liu, UIC 140
Some other issues in SVM

SVM works only in a real-valued space. For a

categorical attribute, we need to convert its
categorical values to numeric values.

SVM does only two-class classification. For multi-

class problems, some strategies can be applied, e.g.,
one-against-rest, and error-correcting output coding.

The hyperplane produced by SVM is hard to

understand by human users. The matter is made
worse by kernels. Thus, SVM is commonly used in
applications that do not required human
understanding.
CS583, Bing Liu, UIC 141
Road Map

Basic concepts

Decision tree induction

Evaluation of classifiers

Rule induction

Classification using association rules

Nave Bayesian classification

Nave Bayes for text classification

Support vector machines

K-nearest neighbor

Summary
CS583, Bing Liu, UIC 142
k-Nearest Neighbor Classification
(kNN)

Unlike all the previous learning methods, kNN

does not build model from the training data.

To classify a test instance d, define k-

neighborhood P as k nearest neighbors of d

Count number n of training instances in P that

belong to class c
j

Estimate Pr(c
j
|d) as n/k

No training is needed. Classification time is

linear in training set size for each test case.
CS583, Bing Liu, UIC 143
kNNAlgorithm

k is usually chosen empirically via a validation

set or cross-validation by trying a range of k
values.

Distance function is crucial, but depends on

applications.
CS583, Bing Liu, UIC 144
Example: k=6 (6NN)
Government
Science
Arts
A new point
Pr(science| )
?
CS583, Bing Liu, UIC 145
Discussions

kNN can deal with complex and arbitrary

decision boundaries.

Despite its simplicity, researchers have

shown that the classification accuracy of kNN
can be quite strong and in many cases as
accurate as those elaborated methods.

kNN is slow at the classification time

kNN does not produce an understandable

model
CS583, Bing Liu, UIC 146
Road Map

Basic concepts

Decision tree induction

Evaluation of classifiers

Rule induction

Classification using association rules

Nave Bayesian classification

Nave Bayes for text classification

Support vector machines

K-nearest neighbor

Summary
CS583, Bing Liu, UIC 147
Summary

Applications of supervised learning are in almost any

field or domain.

We studied 8 classification techniques.

There are still many other methods, e.g.,

Bayesian networks

Neural networks

Genetic algorithms

Fuzzy classification
This large number of methods also show the importance of
classification and its wide applicability.

It remains to be an active research area.