Scribd
Upload
15 views

ML Lab 10 - Ensemble Learning

This document provides an introduction and examples of ensemble learning techniques for classification tasks using the R programming language. It loads data on car seat sales, splits it into training and test sets, and trains three models: a single decision tree (Model 0), a bagged decision tree ensemble (Model 1), and a random forest (Model 2). It evaluates the models on both training and test data and compares their performance, finding that the random forest performs best with an AUC of 0.9021 on test data. The document provides exercises to apply boosting and analyze results compared to bagging, and questions about determining the number of base classifiers and calculating ensemble error.

Uploaded by

PRIYANSH AGGARWAL
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
15 views

ML Lab 10 - Ensemble Learning

This document provides an introduction and examples of ensemble learning techniques for classification tasks using the R programming language. It loads data on car seat sales, splits it into training and test sets, and trains three models: a single decision tree (Model 0), a bagged decision tree ensemble (Model 1), and a random forest (Model 2). It evaluates the models on both training and test data and compares their performance, finding that the random forest performs best with an AUC of 0.9021 on test data. The document provides exercises to apply boosting and analyze results compared to bagging, and questions about determining the number of base classifiers and calculating ensemble error.

Uploaded by

PRIYANSH AGGARWAL
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

Lab Sheet - 10

Ensemble Learning
Machine Learning
BITS F464
I Semester 2023-24

Ensemble Learning is a type of supervised learning technique in which the basic idea is to
generate multiple models on a training dataset and then simply combining (average) their
output to generate a strong model which outperforms all the individual base classifiers.

Load the libraries


library(psych) #for general functions
library(ggplot2) #for data visualization
# library(devtools)
library(caret) #for training and cross validation (also calls other model li
baries)
## Warning: Installed Rcpp (0.12.13) different from Rcpp used to build dplyr
(0.12.11).
## Please reinstall dplyr to avoid random crashes or undefined behavior.
library(rpart) #for trees
library(rpart.plot) # Enhanced tree plots
library(RColorBrewer) # Color selection for fancy tree plot
library(party) # Alternative decision tree algorithm
library(partykit) # Convert rpart object to BinaryTree
library(pROC) #for ROC curves
library(ISLR) #for the Carseat Data

Introduction to Data
Reading in the CarSeats Data exploration data set. This is a simulated data set containing sales of
child car seats at 400 different stores. Sales can be predicted by 10 other variables.

#loading the data


data("Carseats")

Our outcome of interest will be a binary version of Sales: Unit sales (in thousands) at each
location.
(Note again that there is no id variable. This is convenient for some tasks.)
Descriptives

#sample descriptives
describe(Carseats)
#histogram of outcome
ggplot(data=Carseats, aes(x=Sales)) +
geom_histogram(binwidth=1, boundary=.5, fill="white", color="black") +
geom_vline(xintercept = 8, color="red", size=2) +
labs(x = "Sales")

For convenience of didactic illustration we create a new variable HighSales that is binary, “No”
if Sales <= 8, and “Yes” otherwise.

#creating new binary variable


Carseats$HighSales=ifelse(Carseats$Sales<=8,"No","Yes")

Some Data cleanup

#remove old variable


Carseats$Sales <- NULL
#convert a factor variable into a numeric variable
Carseats$ShelveLoc <- as.numeric(Carseats$ShelveLoc)

Splitting the data into training and test sets


We split the data - half for Training, half for Testing

#random sample half the rows


halfsample = sample(dim(Carseats)[1], dim(Carseats)[1]/2) # half of sample
#create training and test data sets
Carseats.train = Carseats[halfsample, ]
Carseats.test = Carseats[-halfsample, ]

We will use these to evaluate a variety of different classification algorithms: Random Forests,
CForests,

Model 0: A Single Classification Tree


We first optimize fit of a classification tree. Our objective with the cross-validation is to optmize
the size of the tree - tuning the complexity parameter.
train.tree <- train(as.factor(HighSales) ~ .,
data=Carseats.train,
method="ctree",
trControl=cvcontrol,
tuneLength = 10)
train.tree

We see how the accruacy is maximized at a relatively less complex tree.


Look at the final tree

# plot tree
plot(train.tree$finalModel,
main="Regression Tree for Carseat High Sales")

To evalaute the accuracy of the tree we can look at the confusion matrix for the Training data.

#obtaining class predictions


tree.classTrain <- predict(train.tree,
type="raw")
head(tree.classTrain)
#computing confusion matrix
confusionMatrix(Carseats.train$HighSales,tree.classTrain)

Some Errors. But the model was learned.


More interesting is the confusion matrix when applied to the Test data.

#obtaining class predictions

tree.classTest <- predict(train.tree,


newdata = Carseats.test,
type="raw")
head(tree.classTest)
#computing confusion matrix
confusionMatrix(Carseats.test$HighSales,tree.classTest)

Accuracy of 0.71
When evaluating classification models, a few other functions may be useful. For example,
the pROC package provides convenience for calculating confusion matrices, the associcated
measures of sensitivity and specificity, and for obtaining and plotting ROC curves. We can also
look at the ROC curve by extracting probabilites of “Yes”.

#Obtaining predicted probabilites for Test data


tree.probs=predict(train.tree,
newdata=Carseats.test,
type="prob")
head(tree.probs)
#Calculate ROC curve
rocCurve.tree <- roc(Carseats.test$HighSales,tree.probs[,"Yes"])
#plot the ROC curve
plot(rocCurve.tree,col=c(4))
calculate the area under curve (bigger is better)
auc(rocCurve.tree)

Model 1: Bagging of ctrees


Training the model using treebag
We first optimize fit of a classification tree. Our objective with the cross-validation is to optmize
the size of the tree - tuning the complexity parameter.

#Using treebag

train.bagg <- train(as.factor(HighSales) ~ ., data=Carseats.train,method="tre


ebag",trControl=cvcontrol,importance=TRUE)
train.bagg
plot(varImp(train.bagg))

Not yet sure how to parse mode details from the output in order to look at the collection of trees.
Look at the collection of final trees
To evaluate the accuracy of the Bagged Trees we can look at the confusion matrix for the
Training data.

#obtaining class predictions


bagg.classTrain <- predict(train.bagg, type="raw")
head(bagg.classTrain)
#computing confusion matrix
confusionMatrix(Carseats.train$HighSales,bagg.classTrain)

The accuracy is perfect!


More interesting is the confusion matrix when applied to the Test data.

#obtaining class predictions


bagg.classTest <- predict(train.bagg,
newdata = Carseats.test,
type="raw")
head(bagg.classTest)
#computing confusion matrix
confusionMatrix(Carseats.test$HighSales,bagg.classTest)

Accuracy of 0.76
We can also look at the ROC curve by extracting probabilites of “Yes”.

#Obtaining predicted probabilites for Test data


bagg.probs=predict(train.bagg,
newdata=Carseats.test, type="prob")

head(bagg.probs)
#Calculate ROC curve
rocCurve.bagg <- roc(Carseats.test$HighSales,bagg.probs[,"Yes"])
#plot the ROC curve
plot(rocCurve.bagg,col=c(6))

#calculate the area under curve (bigger is better)


auc(rocCurve.bagg)
## Area under the curve: 0.8904

Model 2: Random Forest for classification trees


Training the model using random forest

train.rf <- train(as.factor(HighSales) ~ .,


data=Carseats.train,
method="rf",
trControl=cvcontrol,
#tuneLength = 3,
importance=TRUE)
train.rf

We can look at the confusion matrix for the Training data.

#obtaining class predictions


rf.classTrain <- predict(train.rf, type="raw")
head(rf.classTrain)
#computing confusion matrix
confusionMatrix(Carseats.train$HighSales,rf.classTrain)

No Errors. That is good - the model was learned well.


More interesting is the confusion matrix when applied to the Test data.

#obtaining class predictions


rf.classTest <- predict(train.rf, newdata = Carseats.test,
type="raw")
head(rf.classTest)
#computing confusion matrix
confusionMatrix(Carseats.test$HighSales,rf.classTest)

Accuracy of 0.78. An improvement over Bagging only


We can also look at the ROC curve by extracting probabilites of “Yes”.

#Obtaining predicted probabilites for Test data


rf.probs=predict(train.rf,newdata=Carseats.test,type="prob")
head(rf.probs)
#Calculate ROC curve
rocCurve.rf <- roc(Carseats.test$HighSales,rf.probs[,"Yes"])
#plot the ROC curve
plot(rocCurve.rf,col=c(1))
#calculate the area under curve (bigger is better)
auc(rocCurve.rf)
## Area under the curve: 0.9021
Model Comparisons
We can examine how the models do by looking at the ROC curves.

plot(rocCurve.tree,col=c(4))
plot(rocCurve.bagg,add=TRUE,col=c(6)) # color magenta is bagg
plot(rocCurve.rf,add=TRUE,col=c(1)) # color black is rf

A good demonstration of Decision Tree, Random Forest, boosting and bagging algorithm can be
found here:
See …
https://machinelearningmastery.com/machine-learning-ensembles-with-r/
https://www.analyticsvidhya.com/blog/2017/02/introduction-to-ensembling-along-with-
implementation-in-r/

Exercise
1. Apply boosting model on the same dataset and compare it with bagging ensemble. Also,
try the Adaboost function and analyze their result.
2. How to determine the number of base classifiers to be used in an ensemble?
3. How to find the error of an ensemble when the error rates of base classifiers are
different?

You might also like