Machine Learning Lab (IT-804)

Experiment No: 3
Objective: Write a program to demonstrate the working of the decision tree based ID3
algorithm. Use an appropriate data set for building the decision tree and apply this knowledge to
classify a new sample.
Explanation:
ID3 algorithm, stands for Iterative Dichotomiser 3, is a classification algorithm that follows a
greedy approach of building a decision tree by selecting a best attribute that yields maximum
Information Gain (IG) or minimum Entropy (H).
ID3 Steps:
 Calculate the Information Gain of each feature.
 Considering that all rows don’t belong to the same class, split the dataset S into subsets
using the feature for which the Information Gain is maximum.
 Make a decision tree node using the feature with the maximum Information gain.
 If all rows belong to the same class, make the current node as a leaf node with the class as
its label.
 Repeat for the remaining features until we run out of all features, or the decision tree has
all leaf nodes.
Entropy: Entropy, also called as Shannon Entropy is denoted by Entropy(S) for a finite set S, is
the measure of the amount of impurity or uncertainty or randomness in data.

If the target attribute can take on k different values, then the entropy of S relative to this k-wise
classification is defined as
k

Entropy ( S )=−∑ pim logn pi m

i=1

Where pi is the proportion of S belonging to class i.

Information Gain: Information gain is also called as Kullback-Leibler divergence denoted by
Gain(S, A) for a set S is the effective change in entropy after deciding on a particular attribute A.
It measures the relative change in entropy with respect to the independent variables. In other
words it is a measure of the effectiveness of an attribute in classifying the training data.
The information gain, Gain(S, A) of an attribute A
∣Sv∣
Gain( S , A )= Entropy( S )− ∑ Entropy ( S )
v ∈ Values( A ) ∣S∣ v
Machine Learning Lab (IT-804)

Program:
import math
import csv

def load_csv(filename):
lines=csv.reader(open(filename,"r"));
dataset = list(lines)
headers = dataset.pop(0)
return dataset,headers

class Node:
def init (self,attribute):
self.attribute=attribute
self.children=[ ]
self.answer=""
def subtables(data,col,delete):
dic={}
coldata=[row[col] for row in data]
attr=list(set(coldata))

counts=[0]*len(attr)
r=len(data)
c=len(data[0])
for x in range(len(attr)):
for y in range(r):
if data[y][col]==attr[x]:
counts[x]+=1
for x in range(len(attr)):
dic[attr[x]]=[[0 for i in range(c)] for j in range(counts[x])]
pos=0
for y in range(r):
if data[y][col]==attr[x]:
if delete:
del data[y][col]
dic[attr[x]][pos]=data[y]
Machine Learning Lab (IT-804)

pos+=1
return attr,dic

def entropy(S):
attr=list(set(S))
if len(attr)==1:
return 0

counts=[0,0]
for i in range(2):
counts[i]=sum([1 for x in S if attr[i]==x])/(len(S)*1.0)

sums=0
for cnt in counts:
sums+=-1*cnt*math.log(cnt,2)
return sums

def compute_gain(data,col):
attr,dic = subtables(data,col,delete=False)

total_size=len(data)
entropies=[0]*len(attr)
ratio=[0]*len(attr)

total_entropy=entropy([row[-1] for row in data])

for x in range(len(attr)):
ratio[x]=len(dic[attr[x]])/(total_size*1.0)
entropies[x]=entropy([row[-1] for row in dic[attr[x]]])
total_entropy-=ratio[x]*entropies[x]
return total_entropy

def build_tree(data,features):
lastcol=[row[-1] for row in data]
if(len(set(lastcol)))==1:
node=Node("")
Machine Learning Lab (IT-804)

node.answer=lastcol[0]
return node
n=len(data[0])-1
gains=[0]*n
for col in range(n):
gains[col]=compute_gain(data,col)
split=gains.index(max(gains))
node=Node(features[split])
fea = features[:split]+features[split+1:]

attr,dic=subtables(data,split,delete=True)
for x in range(len(attr)):
child=build_tree(dic[attr[x]],fea)
node.children.append((attr[x],child))
return node

def print_tree(node,level):
if node.answer!="":
print(" "*level,node.answer)
return
print(" "*level,node.attribute)
for value,n in node.children:
print(" "*(level+1),value)
print_tree(n,level+2)

def classify(node,x_test,features):
if node.answer!="":
print(node.answer)
return
pos=features.index(node.attribute)
for value, n in node.children:
if x_test[pos]==value:
classify(n,x_test,features)

'''Main program'''
Machine Learning Lab (IT-804)

dataset,features=load_csv("id3.csv")
node1=build_tree(dataset,features)

print("The decision tree for the dataset using ID3 algorithm is")
print_tree(node1,0)
testdata,features=load_csv("id3_test.csv")
for xtest in testdata:
print("The test instance:",xtest)
print("The label for test instance:",end=" ")
classify(node1,xtest,features
Data set:
id3_test.csv
Day Outlook Temp. Humidity Wind PayTennis
1 Sunny Hot High Weak No
2 Sunny Hot High Strong No
3 Overcast Hot High Weak Yes
4 Rain Mild High Weak Yes
5 Rain Cool Normal Weak Yes
6 Rain Cool Normal Strong No
7 Overcast Cool Normal Strong Yes
8 Sunny Mild High Weak No
9 Sunny Cool Normal Weak Yes
10 Rain Mild Normal Weak Yes
11 Sunny Mild Normal Strong Yes
12 Overcast Mild High Strong Yes
13 Overcast Hot Normal Weak Yes
14 Rain Mild High Strong No

Output:
outlook
overcast b'yes'
rain
wind
b'strong' b'no' b'weak' b'yes'
sunny
Machine Learning Lab (IT-804)

humidity b'high' b'no'

b'normal' b'yes
The test instance: [‘rain’, ‘cool’, ‘normal’, ‘strong’]
The label for test instance: no
The test instance: [‘sunny’, ‘mild’, ‘normal’, ‘strong’]
The label for test instance: yes
Machine Learning Lab (IT-804)
Experiment No: 4
Objective: Build an Artificial Neural Network by implementing the Backpropagation
algorithm and test the same using appropriate data sets.
Explanation:
BACKPROPAGATION (training_example, ƞ, nin, nout, nhidden )
→→ →
Each training example is a pair of the form ( x , t ) , where ( x) is the vector of network input
→
values, ( t ) and is the vector of target network output values.
η is the learning rate (e.g., .05). ni, is the number of network inputs, nhidden the number of units in
the hidden layer, and nout the number of output units.
The input from unit i into unit j is denoted xji, and the weight from unit i to unit j is denoted wji
 Create a feed-forward network with ni inputs, nhidden hidden units, and nout output units.
 Initialize all network weights to small random numbers
 Until the termination condition is met, Do
→→
o For each ( x , t ) in training examples, Do
Propagate the input forward through the network:
→
1. Input the instance x , to the network and compute the output ou of every unit
u in the network.
Propagate the errors backward through the network:

Training Examples:
Example Sleep Study Expected % in
Exams
1 2 9 92
2 1 5 86
3 3 6 89
Normalize the input:
Example Sleep Study Expected % in
Exams
1 2/3 = 0.66666667 9/9 = 1 0.92
2 1/3 = 0.33333333 5/9 = 0.55555556 0.86
3 3/3 = 1 6/9 = 0.66666667 0.89
Machine Learning Lab (IT-804)

Program:
import numpy as np
X = np.array(([2, 9], [1, 5], [3, 6]), dtype=float)
y = np.array(([92], [86], [89]), dtype=float)
X = X/np.amax(X,axis=0) # maximum of X array longitudinally y = y/100

#Sigmoid Function
def sigmoid (x):
return 1/(1 + np.exp(-x))

#Derivative of Sigmoid Function def

derivatives_sigmoid(x):
Machine Learning Lab (IT-804)

return x * (1 - x)

#Variable initialization
epoch=5000 #Setting training iterations
lr=0.1 #Setting learning rate
inputlayer_neurons = 2 #number of features in data set
hiddenlayer_neurons = 3 #number of hidden layers neurons
output_neurons = 1 #number of neurons at output layer

#weight and bias initialization

wh=np.random.uniform(size=(inputlayer_neurons,hiddenlayer_neurons)) #weight of the link from input
node to hidden node
bh=np.random.uniform(size=(1,hiddenlayer_neurons)) # bias of the link from input node to hidden node
wout=np.random.uniform(size=(hiddenlayer_neurons,output_neurons)) #weight of the link from hidden
node to output node
bout=np.random.uniform(size=(1,output_neurons)) #bias of the link from hidden node to output node

#draws a random range of numbers uniformly of dim x*y

for i in range(epoch):
#Forward Propogation
hinp1=np.dot(X,wh)
hinp=hinp1 + bh
hlayer_act = sigmoid(hinp)
outinp1=np.dot(hlayer_act,wout)
outinp= outinp1+ bout
output = sigmoid(outinp)

#Backpropagation
EO = y-output
outgrad = derivatives_sigmoid(output)
d_output = EO* outgrad
EH = d_output.dot(wout.T)

#how much hidden layer weights contributed to error

hiddengrad = derivatives_sigmoid(hlayer_act)
d_hiddenlayer = EH * hiddengrad

# dotproduct of nextlayererror and currentlayerop

 Machine Learning Lab (IT804) Jan-Jun 2021

wout += hlayer_act.T.dot(d_output) *lr

wh += X.T.dot(d_hiddenlayer) *lr
print("Input: \n" + str(X))
print("Actual Output: \n" + str(y))
print("Predicted Output: \n" ,output)

Data Set:

Output:
Input:
[[ 0.66666667 1. ]
[ 0.33333333 0.55555556]
[ 1. 0.66666667]]
Actual Output:
[[ 0.92]
[ 0.86]
[ 0.89]]
Predicted Output:
[[ 0.89559591]
[ 0.88142069]
[ 0.8928407 ]]

1
Laboratory File