HAZARA UNIVERSITY MANSEHRA

Starting with the Name of ALLAH Almighty, Who’s

Name: SAAD IQBAL

Roll no: 301-211073

Department: Information Technology

Section: BSCS-B

Semester: 6th

Subject Artificial Intelligence

Submitted to Miss Muneeba Darwaish

Assignment 1
Question 1

Explain ID3 Algorithm with the help of appropriate example; calculate the entropy, information gain
and other factors on decision

Introduction:

The ID3 (Iterative Dichotomiser 3) algorithm is a decision tree learning algorithm used for building
decision trees from a given dataset. It is a greedy algorithm that constructs the decision tree in a
top-down, recursive manner by selecting the most informative attribute at each node to split the
data.

Example:

Let's consider the following dataset for playing tennis based on several attributes:

Day Outlook Temperature Humidity Wind Play Tennis

1 Sunny Hot High Weak No

2 Sunny Hot High Strong No

3 Overcast Hot High Weak Yes

4 Rain Mild High Weak Yes

5 Rain Cool Normal Weak Yes

6 Rain Cool Normal Strong No

7 Overcast Cool Normal Strong Yes

8 Sunny Mild High Weak No

9 Sunny Cool Normal Weak Yes

10 Rain Mild Normal Weak Yes

11 Sunny Mild Normal Strong Yes

12 Overcast Mild High Strong Yes

13 Overcast Hot Normal Weak Yes

14 Rain Mild High Strong No

Entropy Calculation:

Entropy is a measure of the impurity or uncertainty in the dataset. It is calculated as follows:

Entropy(S) = -Σ(p(i) * log2(p(i)))

Where:

• S is the current dataset

• p(i) is the proportion of instances belonging to class i

For the given dataset, the initial entropy is calculated as:

Entropy(S) = -((9/14) * log2(9/14) + (5/14) * log2(5/14)) Entropy(S) = 0.940

Information Gain Calculation:

Information gain is a measure of how much entropy (or impurity) is reduced by splitting the dataset
on a particular attribute. The attribute with the highest information gain is chosen as the root node
of the decision tree.

Information Gain (S, A) = Entropy(S) - Σ((|Sv| / |S|) * Entropy(Sv))

Where:
 • S is the current dataset

• A is the attribute for which information gain is being calculated

• Sv is the subset of S for which attribute A has the value v

• |Sv| is the number of instances in Sv

• |S| is the total number of instances in S

For example, let's calculate the information gain for the "Outlook" attribute:

Outlook = Sunny Entropy(Sv) = -((2/5) * log2(2/5) + (3/5) * log2(3/5)) = 0.971

Outlook = Overcast Entropy(Sv) = 0 (all instances belong to the same class)

Outlook = Rain Entropy(Sv) = -((2/5) * log2(2/5) + (3/5) * log2(3/5)) = 0.971

Information Gain(S, Outlook) = 0.940 - ((5/14) * 0.971 + (4/14) * 0 + (5/14) * 0.971) Information
Gain(S, Outlook) = 0.246

Similarly, the information gain can be calculated for other attributes, and the attribute with the
highest information gain ("Outlook" in this case) is chosen as the root node of the decision tree.

Decision Tree Construction:

The ID3 algorithm recursively constructs the decision tree by selecting the attribute with the
highest information gain at each node until all instances belong to the same class or there are no
more attributes left to split.

The decision tree for the given dataset would be as follows:

Outlook

/ \

/ \

Sunny Overcast

/ \ \

Humidity Play Tennis

/ \ = Yes
High Normal

\ /

\ /

No Yes

In this decision tree, the root node is "Outlook." If the outlook is sunny, the next attribute to consider
is "Humidity." If the humidity is high, the decision is "No" for playing tennis; otherwise, the decision
is "Yes." If the outlook is overcast, the decision is always "Yes" for playing tennis.

Question 2

Explanation of Directed Acyclic Graph (DAG) with Burglary Alarm Example

Introduction:

A Directed Acyclic Graph (DAG) is a directed graph with no cycles. In other words, it is a graph
where you cannot start from any node and traverse through the edges to return to the same node.
DAGs are commonly used to represent probabilistic models, such as Bayesian Networks.

Burglary Alarm Example:

Consider the following scenario: You have a burglar alarm installed in your house, and it can be
triggered by a burglar breaking in or by small earthquakes that are common in your area.
Additionally, you have two neighbors, John and Mary, who can potentially call you if they hear the
alarm.

We can represent this scenario using a DAG, where the nodes represent the random variables, and
the edges represent the causal relationships between them.

Burglar

/ \

/ \

/ \

Earthquake Alarm

\ / \
 \ / \

\ / \

Call Call

John MaryIn this DAG:

• The "Burglar" and "Earthquake" nodes represent the potential causes of the alarm.

• The "Alarm" node represents the event of the alarm being triggered.

• The "Call John" and "Call Mary" nodes represent the events of John and Mary calling you
after hearing the alarm.

The edges in the DAG represent the causal relationships between the variables:

• A burglar can directly cause the alarm to trigger.

• An earthquake can directly cause the alarm to trigger.

• The alarm can directly cause John and Mary to call you.

However, there are no direct edges between "Burglar" and "Call John" or "Call Mary," as the burglar
does not directly cause them to call; it is the alarm that triggers the call.

This DAG captures the conditional independence relationships between the variables. For example,
if we know the state of the "Alarm" node, the "Burglar" and "Earthquake" nodes become
independent of the "Call John" and "Call Mary" nodes. This property allows us to simplify the
calculations and make efficient inferences in probabilistic models.

DAGs are widely used in various domains, including machine learning, artificial intelligence, and
decision support systems, to represent and reason about complex probabilistic models.