Scribd
Upload
49 views

Lecture 8 PDF

Uploaded by

Sachin singh
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
49 views

Lecture 8 PDF

Uploaded by

Sachin singh
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 23

Decision Trees

Dr. Jainendra Shukla


[email protected]
B-403, R & D Block
Computer Science and Engineering
Feeling Hungry
Examples

● Approve a credit card applications or not


● Approve a loan applications or not
● If a customer will take up a product or not
● If a transaction is fraudulent or not
● If a customer will close a telephone connection
Playing Tennis
Playing Tennis
Decision Tree Learning

A Decision tree for


F: <Outlook, Humidity, Wind, Temp> PlayTennis?

● Each internal node: test one attribute Xi


● Each branch from a node: selects one value for Xi
● Each leaf node: predict Y (or P(Y|X ∈ leaf))
Problem Setting
● Set of possible instances X
○ each instance x in X is a feature vector
○ e.g., <Humidity=low, Wind=weak, Outlook=rain,
Temp=hot>
● Unknown target function f : X->Y
○ Y is discrete valued
● Set of function hypotheses H={ h | h : X->Y }
○ Each hypothesis h is a decision tree
○ Trees sorts x to leaf, which assigns y
Input:
• Training examples {<x(i),y(i)>} of unknown target function f
Output:
• Hypothesis h ∈ H that best approximates target function f
Top Down Induction of Decision Trees
Node =Root
Main loop:
1. A <— the "best" decision attribute for next node
2. Assign A as decision attribute for node
3. For each value of A, create new descendant of node
4. Sort training examples to leaf nodes
5. If training examples perfectly classified,
a. Then STOP
b. Else iterate over new leaf nodes
Entropy

● Which attribute is best?


● The entropy -> information content
a. More uncertainty -> More entropy [Predictability?]

Result Prob Result Prob


H 0.5 H 0.75
T 0.5 T 0.25
Numerical: H(Y) = ?
Impurity

● Purity (or impurity) is homogeneity (or heterogeneity) of the


data.

● Entropy measures the impurity of the training sample S.


Conditional Entropy

H(Y|X1) = ?
Information Gain

● Information Gain is the expected reduction in entropy after


splitting

○ IG(X) > 0; split is preferred


Selecting the next attribute: H, W?
Selecting the next attribute: H, W?
Selecting the next attribute: Humidity

● IG(Humidity) = H(PlayTennis) - H(PlayTennis|Humidity)


● H(PlayTennis)
= -(9/14)*log2(9/14) -(5/14)*log2(5/14) = 0.940
● H(PlayTennis|Humidity)
=
-P(High)*[P(PlayTennis|High)*log2P(PlayTennis|High) +
P(~PlayTennis|High)*log2P(~PlayTennis|High)]
-
P(Normal)*[P(PlayTennis|Normal)*log2P(PlayTennis|Normal
) + P(~PlayTennis|Normal)*log2P(~PlayTennis|Normal)]
Selecting the next attribute: Humidity
Selecting the next attribute: Wind
Selecting the next attribute: H, W?

● IG(Humidity) = 0.151
● IG(Humidity) = 0.048
○ It is better to split on humidity rather than wind as
humidity has a higher information gain.
Gini Impurity
Gini Impurity

● Gini = 0.46
References
1. Constructing optimal binary decision trees is NP-complete."
Information Processing Letters 5.1 (1976): 15-17.
2. Entropy: https://www.cs.utexas.edu/~byoung/cs361/lecture32.pdf
3. http://www.cs.cmu.edu/~tom/10701_sp11/slides/DTreesAndOverfittin
g-1-11-2011_final.pdf
4. https://www.ke.tu-darmstadt.de/lehre/archiv/ws0809/mldm/dt.pdf

You might also like