Decision Tree ID3 Algorithm

|Machine Learning
AshirbadPradhan · Follow
6 min read · Jun 11, 2023

75

In this blog ,we understand Decision Tree ID3 algorithm in details with
example sample dataset.
Decision Tree

A Decision Tree is a supervised Machine learning algorithms used for

both regression and classification problem statement. It uses the tree
representation to solve a problem in which each node represents an
attribute, each link represents a decision rule and each leaf represents an
outcome( categorical or continuous value).

Decision Tree Terminologies

Root Node- It is the topmost node in the tree, which represent the complete
dataset. Also we can say it is the starting point of the decision-making
process.

Decision/Internal Node- Decision nodes are nothing but the result in the
splitting of data into multiple data segments and main goal is to have the
children nodes with maximum homogeneity or purity( means all of the
same kind).

Leaf/Terminal Node- This node represent the data section having highest
homogeneity (means all of the same kind).
Entropy-It is used for checking the impurity or uncertainty present in the
data. Entropy is used to evaluate the quality of a split. When entropy is zero
the sample is completely homogeneous, meaning that each instance belongs
to the same class and entropy is one when the sample is equally divided
between different classes.

Formula of Entropy -

Information Gain- Information gain indicates how much information a

particular feature/ variable give us about the final outcome.
Formula of information gain-

ID3 [Iterative Dichotomiser3]

(It is the most popular algorithms used to constructing trees.)

ID3 stands for Iterative Dichotomizer3 and is named such because the
algorithm iteratively(repeatedly) dichotomizes(divides) features into two or
more groups at each step.ID3 is an algorithm invented by Ross Quinlan used
to generate a decision tree from a dataset and is the most popular algorithms
used to constructing trees.

ID3 is the core algorithm for building a decision tree .It employs a top-down
greedy search through the space of all possible branches with no
backtracking. This algorithm uses information gain and entropy to construct
a classification decision tree.

Characteristics of ID3 Algorithm

Major Characteristics of the ID3 Algorithm are listed below:

ID3 can overfit the training data (to avoid overfitting, smaller decision
trees should be preferred over larger ones).

This algorithm usually produces small trees, but it does not always
produce the smallest possible tree.

ID3 is harder to use on continuous data (if the values of any given
attribute is continuous, then there are many more places to split the data
 on this attribute, and searching for the best value to split by can be time-
consuming).

Advantages and Disadvantages of ID3 Algorithm

Advantages
Inexpensive to construct

Extremely fast at classifying unknown records Easy to interpret for small-

sized trees.

Robust to noise (especially when methods to avoid over-fitting are

employed).

Can easily handle redundant or irrelevant attributes (unless the attributes

are interacting).

Disadvantages
The space of possible decision trees is exponentially large. Greedy
approaches are often unable to find the best tree.

Does not take into account interactions between attributes.

Each decision boundary involves only a single attribute.

Steps to making Decision Tree

a) Take the Entire dataset as an input.

b) Calculate the Entropy of the target variable, As well as the predictor

attributes

c) Calculate the information gain of all attributes.

d) Choose the attribute with the highest information gain as the Root Node

e) Repeat the same procedure on every branch until the decision node of
each branch is finalized.

Example-
Forecast whether the match will be played or not according to the weather
condition. Here we can see the table-

First we are going to calculate the entropy for “Decision” attribute which is a
target variable and also calculate the entropy for independent attributes like
“Outlook”, “Temp.” , “Humidity” , “Wind” .
Repeat the same procedure on every branch until the decision node of each
branch is finalized.
Outlook=Sunny and temp=Hot.

Outlook=Sunny and temp=Mild

Outlook=Sunny and Temp=Cool

Outlook=Sunny |Humidity=High

Outlook=Sunny | Humidity=Normal
Outlook=Strong | Wind=Strong

Outlook=Sunny | Wind=Weak
Here decision will always be yes if outlook were overcast. So no need of
calculating entropy and information gain.
Outlook=Rain | Temp=Mild

Outlook=Rain |Temp=Cool

Outlook=Rain | Humidity=High

Outlook=Rain | Humidity=Normal
Search Write Sign up Sign in

Outlook=Rain | Wind=Strong

Outlook=Rain | Wind=Weak
Here we can see that information gain is high for (Outlook=Rain | Wind) , so
it will be the decision node after Rain.
Decision Tree construction is over. Here we learned that how a decision tree
is created in backend using this algorithm.

Conclusion
We covered the process of the ID3 algorithm in detail and saw how easy it
was to create a Decision Tree using this algorithm by using only two metrics
i.e. Entropy and Information Gain.

Hope you liked it , fellas !

Thanks for reading .

Data Science Machine Learning Decision Tree Data Scientist Data Analyst

Written by AshirbadPradhan Follow

13 Followers

More from AshirbadPradhan