MATRUSRI

ENGINEERING COLLEGE

MATRUSRI ENGINEERING COLLEGE

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

SUBJECT NAME: Machine Learning

FACULTY NAME: Mrs.J.Samatha

 MATRUSRI
ENGINEERING COLLEGE

MACHINE LEARNING
COURSE OBJECTIVES:
•To explore the supervised learning paradigms of machine learning
•To explore the unsupervised learning paradigms of machine learning
•To evaluate various machine learning algorithms and techniques
•To explore deep learning technique and various feature extraction
strategies.
•To explore recent trends in machine learning methods for IOT
applications.
COURSE OUTCOMES:
•Extract features and apply supervised learning paradigms.
•Illustrate several clustering algorithms on the give data set.
•Compare and contrast various machine learning algorithms and
get insight of when to apply particular machine learning approach
•Apply basic deep learning algorithms and feature extraction
strategies.
•Get familiarized with advanced topics of machine learning.
 MATRUSRI
ENGINEERING COLLEGE

MODULE-I

Evaluating machine learning algorithms and model selection

OUTCOMES:
• Able to evaluate different machine learning algorithms and select a
model
 MATRUSRI
ENGINEERING COLLEGE

Evaluation of Learning Algorithms

Evaluating the performance of learning systems is important because:
-Learning are usually designed to predict the class of future unlabeled
data points

Typical choices for performance evaluation:

-Error
-Accuracy
-Precision/recall

Typical choices for sampling methods:

-Hold out method or Training/test set
-K fold cross validation
-Leave one out cross validation
-Boot strap method
 MATRUSRI
ENGINEERING COLLEGE

Hold out Method(Training/Validation/Test set)

MATRUSRI
ENGINEERING COLLEGE
 MATRUSRI
ENGINEERING COLLEGE

K fold Cross Validation

1.Split the data into k equal subsets
2.Perform k rounds of learning, on each round
-1/k of the data is held out as a test set and
-The remaining examples are used as training data
3. Compute the average test set score of the k rounds
 MATRUSRI
ENGINEERING COLLEGE

Boot strap method

•In statistics, the term “bootstrap sampling”, the “bootstrap” or
“bootstrapping” for short, refers to process of “random sampling with
replacement”.

• repeated sampling from data with replacement and repeated

estimation

•Subsample will have same number of observations

•Same observation can be selected many times

•Probability of selecting each observation is same

 MATRUSRI
ENGINEERING COLLEGE

Leave one out Cross Validation

•Specific case of K fold CV.
•K=N

•ADV:
•Good way to validate
•Disadv:
•High computation time
 MATRUSRI
ENGINEERING COLLEGE

Evaluating Predictions
Suppose we want to make a prediction of a value for a target feature
on example x:

-Y is observed value of target value of target feature on example x

-Y’ is the predicted value of target feature on example x

-How is the error measured?

 MATRUSRI
ENGINEERING COLLEGE

Terminology related to Evaluation of Algorithms

Supervised Supervised learning teaches a model from labeled training data and helps you to
learning make predictions about unseen or future data. During the training, you give the
algorithm a dataset that contains correct answers (label y). Then, you validate the
model accuracy with a test data set with correct answers. A data set must be split into
training and test sets.

Classification With classification, you're trying to predict one of a small number of discrete-valued
outputs. For example, you might try to predict whether your label is binary (binary
classification) or categorical (multiclass classification).

Regression In regression, the goal of the learning problem is to predict continuous value output.

Ranking Order items according to some criterion. Ex: web search : returning web pages
relevant to a search query. Many other similar ranking problems arise in the context
of the design of information extraction of natural language processing systems.
Unsupervised Given a data set, try to find tendencies in the data by using techniques like clustering.
learning

Feature Feature is an attribute that is used as input for the model to train. Other names
include dimension or column.
 MATRUSRI
ENGINEERING COLLEGE

Terminology related to Evaluation of Algorithms

Bias Bias is the expected difference between the parameters of a
model that perfectly fits your data and the parameters that your
algorithm learned. The sample error is a poor estimator of true
error.
Variance Variance is how much the algorithm is impacted by the training
data and how much the parameters change with new training
data. The smaller the test set, the greater the expected variance.
Underfitting The model is too simple to capture the patterns within the data.
The model performs poorly on data that it was trained on and on
unseen data. High bias, low variance. High training error and
high test error.
Overfitting The model is too complicated or too specific, capturing trends
that don't generalize. The model accurately predicts data that it
was trained on but doesn't accurately predict unseen data. Low
bias, high variance. Low training error and high test error.
Bias- Bias-Variance Trade-off refers to finding a model with the right
 MATRUSRI
ENGINEERING COLLEGE

Bias
Bias is the difference between our actual and predicted values. Bias is the
simple assumptions that our model makes about our data to be able to
predict new data.
 MATRUSRI
ENGINEERING COLLEGE

Bias
When the Bias is high, assumptions made by our model are too
basic, the model can’t capture the important features of our data. This
means that our model hasn’t captured patterns in the training data and
hence cannot perform well on the testing data too. If this is the case, our
model cannot perform on new data and cannot be sent into production.
This instance, where the model cannot find patterns in our
training set and hence fails for both seen and unseen data, is called
Underfitting.
 MATRUSRI
ENGINEERING COLLEGE

Variance
We can define variance as the model’s sensitivity to fluctuations in the
data. Our model may learn from noise. This will cause our model to
consider trivial features as important.

we can see that our model has learned extremely well for our training
data, which has taught it to identify cats. But when given new data, such
as the picture of a fox, our model predicts it as a cat, as that is what it has
learned. This happens when the Variance is high, our model will capture
all the features of the data given to it, including the noise, will tune itself
to the data, and predict it very well but when given new data, it cannot
predict on it as it is too specific to training data.
 MATRUSRI
ENGINEERING COLLEGE

Variance
Hence, our model will perform really well on testing data and get high
accuracy but will fail to perform on new, unseen data. New data may not
have the exact same features and the model won’t be able to predict it
very well. This is called Overfitting.

Over-fitted model where we see model performance on, a) training data

b) new data
 MATRUSRI
ENGINEERING COLLEGE

Bias & Variance

 MATRUSRI
ENGINEERING COLLEGE

Evaluation Metrics
REGRESSION:
•Mean absolute error
•Mean squares error
•Root Mean Squared Error(RMSE)
•Mean absolute percentage error

CLASSIFICATION:
•Confusion matrix
 MATRUSRI
ENGINEERING COLLEGE

Mean Absolute Error

We will calculate the residual for every data point, taking only the
absolute value of each so that negative and positive residuals do not
cancel out. We then take the average of all these residuals.
 MATRUSRI
ENGINEERING COLLEGE

Mean Absolute Error

•Take the absolute difference between Y and Ŷ for each of the
available observations: ⎮Yᵢ-Ŷᵢ⎮ where i ϵ [1, the total number of
points in the dataset].
•Sum each absolute difference to get a total error: Σ⎮Yᵢ-Ŷᵢ⎮
•Divide the sum by a total number of observations to get a mean
error value: Σ⎮Yᵢ-Ŷᵢ⎮ / n

MAE = Σ⎮Yᵢ-Ŷᵢ⎮ / n
 MATRUSRI
ENGINEERING COLLEGE

Mean square error

•Take the difference between Y and Ŷ for each of the available

observations: Yᵢ-Ŷᵢ
•Square each of the difference value : (Yᵢ-Ŷᵢ)²
•Sum Squared values: Σ (Yᵢ-Ŷᵢ)² where i ϵ [1, the total number of
points in the dataset]
•Divide by the total number of observations: Σ (Yᵢ-Ŷᵢ)² / n
 MATRUSRI
ENGINEERING COLLEGE

Root Mean squared error

As RMSE is clear by the name itself, that it is a simple square root of mean
squared error.
 MATRUSRI
ENGINEERING COLLEGE

Mean absolute percentage error

MAPE will be lower when the prediction is lower than the actual
compared to a prediction that is higher by the same amount.
 MATRUSRI
ENGINEERING COLLEGE

Confusion matrix for 2-class problems

 MATRUSRI
ENGINEERING COLLEGE

Confusion Matrix
Basic terminology
•True Positives (TP): we correctly predicted that they do have diabetes
•True Negatives (TN): we correctly predicted that they don't have
diabetes
•False Positives (FP): we incorrectly predicted that they do have diabetes
•False Negatives (FN): we incorrectly predicted that they don't have
diabetes
 MATRUSRI
ENGINEERING COLLEGE

Other accuracy metrics

 MATRUSRI
ENGINEERING COLLEGE

Other accuracy metrics

•Precision tells us how many of the correctly predicted cases actually turned out
to be positive.
•Recall tells us how many of the actual positive cases we were able to predict
correctly with our model.
 MATRUSRI
ENGINEERING COLLEGE

Other accuracy metrics

Recall and sensitivity both are same.

 MATRUSRI
ENGINEERING COLLEGE

Other measures of performance

Using the data in the confusion matrix of a classifier of
two-class dataset, several measures of performance
have been defined.
Accuracy = (TP + TN)/( TP + TN + FP + FN )
Error rate = 1− Accuracy
Sensitivity = TP/( TP + FN)
Precision = TP/( TP + FP)
Specificity = TN /(TN + FP)
F-measure = (2 × TP)/( 2 × TP + FP + FN)
 MATRUSRI
ENGINEERING COLLEGE

Suppose we had a classification dataset with 1000 data points.

We fit a classifier on it and get the below confusion matrix

True Positive (TP) = 560; meaning 560 positive class data points were
correctly classified by the model
True Negative (TN) = 330; meaning 330 negative class data points were
correctly classified by the model
False Positive (FP) = 60; meaning 60 negative class data points were
incorrectly classified as belonging to the positive class by the model
False Negative (FN) = 50; meaning 50 positive class data points were
incorrectly classified as belonging to the negative class by the model
 MATRUSRI
ENGINEERING COLLEGE

Example

The total outcome values are:

TP = 30, TN = 930, FP = 30, FN = 10
So, the accuracy for our model turns out to be:
 MATRUSRI
ENGINEERING COLLEGE

Precision vs. Recall

Precision tells us how many of the correctly predicted cases actually
turned out to be positive.

Recall tells us how many of the actual positive cases we were able to
predict correctly with our model.
 MATRUSRI
ENGINEERING COLLEGE

Practice problem-1
Suppose a computer program for recognizing dogs in photographs
identifies eight dogs in a picture containing 12 dogs and some cats. Of the
eight dogs identified, five actually are dogs while the rest are cats.
Compute the precision and recall of the computer program.
 MATRUSRI
ENGINEERING COLLEGE

Practice problem-2
Let there be 10 balls (6 white and 4 red balls) in a box and let it be
required to pick up the red balls from them. Suppose we pick up 7 balls as
the red balls of which only 2 are actually red balls. What are the values of
precision and recall in picking red ball?
 MATRUSRI
ENGINEERING COLLEGE

Practice Problem-3
A database contains 80 records on a particular topic of which 55 are
relevant to a certain investigation. A search was conducted on that topic
and 50 records were retrieved. Of the 50 records retrieved, 40 were
relevant. Construct the confusion matrix for the search and calculate
the precision and recall scores for the search.
SOLUTION:
Each record may be assigned a class label “relevant" or “not relevant”. All
the 80 records were tested for relevance. The test classified 50 records as
“relevant”. But only 40 of them were actually relevant. Hence we have
the following confusion matrix for the search:
MATRUSRI
ENGINEERING COLLEGE
 MATRUSRI
ENGINEERING COLLEGE

Suppose we have a test dataset of 10 records with expected

outcomes and a set of predictions from our classification algorithm.

Compute the accuracy, precision, sensitivity

and specificity of the data.
 MATRUSRI
ENGINEERING COLLEGE

Sample problem
Suppose 10000 patients get tested for flu; out
of them, 9000 are actually healthy and 1000
are actually sick. For the sick people, a test
was positive for 620 and negative for 380. For
the healthy people, the same test was
positive for 180 and negative for 8820.
Construct a confusion matrix for the data and
compute the accuracy, precision and recall for
the data.
 MATRUSRI
ENGINEERING COLLEGE

Receiver Operating Characteristic (ROC)

•The acronym ROC stands for Receiver Operating
Characteristic, a terminology coming from signal
detection theory.

•The ROC curve was first developed by electrical

engineers and radar engineers during World War II
for detecting enemy objects in battlefields.

•They are now increasingly used in machine learning

and data mining research.
 MATRUSRI
ENGINEERING COLLEGE

TPR and FPR

Let a binary classifier classify a collection of test data.

TP = Number of true positives

TN = Number of true negatives
FP = Number of false positives
FN = Number of false negatives

TPR = True Positive Rate = TP/( TP + FN )= Fraction of

positive examples correctly classified = Sensitivity

FPR = False Positive Rate = FP /(FP + TN) = Fraction of

negative examples incorrectly classified = 1 −
Specificity
 MATRUSRI
ENGINEERING COLLEGE

ROC space
•We plot the values of FPR along the horizontal axis
(that is , x-axis) and the values of TPR along the
vertical axis (that is, y-axis) in a plane.

•For each classifier, there is a unique point in this

plane with coordinates (FPR,TPR).

•The ROC space is the part of the plane whose points

correspond to (FPR,TPR).

•Each prediction result or instance of a confusion

matrix represents one point in the ROC space.
 MATRUSRI
ENGINEERING COLLEGE

ROC space
The position of the point (FPR,TPR) in the ROC
space gives an indication of the performance
of the classifier.

For example, let us consider some special

points in the space

One step higher for positive examples and

one step right for negative examples
MATRUSRI
ENGINEERING COLLEGE
 MATRUSRI
ENGINEERING COLLEGE

Special points in ROC space

•The left bottom corner point (0, 0):
•Always negative prediction
•A classifier which produces this point in the ROC
space never classifies an example as positive,
neither rightly nor wrongly, because for this point
TP = 0 and FP = 0.
•It always makes negative predictions.
•All positive instances are wrongly predicted and
all negative instances are correctly predicted.
•It commits no false positive errors.
 MATRUSRI
ENGINEERING COLLEGE

Special points in ROC space

•The right top corner point (1, 1):
•Always positive prediction
•A classifier which produces this point in the ROC
space always classifies an example as positive
because for this point FN = 0 and TN = 0.
•All positive instances are correctly predicted and
all negative instances are wrongly predicted.
• It commits no false negative errors.
 MATRUSRI
ENGINEERING COLLEGE

Special points in ROC space

•The left top corner point (0, 1):
•Perfect prediction
•A classifier which produces this point in the ROC
space may be thought as a perfect classifier.
•It produces no false positives and no false
negatives
 MATRUSRI
ENGINEERING COLLEGE

Special points in ROC space

• Points along the diagonal:
• Random performance

• Consider a classifier where the class labels

are randomly guessed, say by flipping a

coin.
• Then, the corresponding points in the ROC

space will be lying very near the diagonal

line joining the points (0, 0) and (1, 1).
 MATRUSRI
ENGINEERING COLLEGE

ROC Space & some special points in the

space
 MATRUSRI
ENGINEERING COLLEGE

ROC curve
•In the case of certain classification algorithms, the
classifier may depend on a parameter.

•Different values of the parameter will give different

classifiers and these in turn give different values to
TPR and FPR.

•The ROC curve is the curve obtained by plotting in

the ROC space the points (TPR , FPR) obtained by
assigning all possible values to the parameter in the
classifier
 MATRUSRI
ENGINEERING COLLEGE

ROC curve
•The closer the ROC curve is to the top left corner (0,
1) of the ROC space, the better the accuracy of the
classifier.

•Among the three classifiers A, B, C with ROC curves ,

the classifier C is closest to the top left corner of the
ROC space.

•Hence, among the three, it gives the best accuracy

in predictions.
 MATRUSRI
ENGINEERING COLLEGE

Area under the ROC curve (AUC)

•The measure of the area under the ROC curve is
denoted by the acronym AUC .

• The value of AUC is a measure of the performance

of a classifier.

•For the perfect classifier, AUC = 1.0.

MATRUSRI
ENGINEERING COLLEGE
 MATRUSRI
ENGINEERING COLLEGE

Sample problem on ROC & AUC

•The body mass index (BMI) of a person is defined as
(weight(kg)/height(m)2 ).

•Researchers have established a link between BMI and the risk of

breast cancer among women.

•The higher the BMI the higher the risk of developing breast cancer.

•The critical threshold value of BMI may depend on several

parameters like food habits, socio-cultural-economic background,
life-style, etc

•Gives real data of a breast cancer study with a sample having 100
patients and 200 normal persons.

•The table also shows the values of TPR and FPR for various cut-off
values of BMI.
 MATRUSRI
ENGINEERING COLLEGE

Data for various values of BMI

 MATRUSRI
ENGINEERING COLLEGE

ROC curve of the data

 MATRUSRI
ENGINEERING COLLEGE

Given the following data, construct the ROC

curve of the data. Compute the AUC.
 MATRUSRI
ENGINEERING COLLEGE

Statistical Learning Theory

•Statistical learning theory is a framework for machine
learning, drawing from the fields of statistics and
functional analysis.
•Statistical learning theory deals with the problem of
finding a predictive function based on data.
•The goal of learning is prediction. Learning falls into
many categories:
-Supervised learning
-Unsupervised learning
-Semi-supervised learning
-Transfer Learning
-Online learning
-Reinforcement learning
 MATRUSRI
ENGINEERING COLLEGE

Statistical Learning Theory

•Statistical learning theory was introduced in the late 1960s but until
1990s it was simply a problem of function estimation from a given
collection of data.
•In the middle of the 1990s new types of learning algorithms (e.g.,
support vector machines) based on the developed theory were proposed.
This made statistical learning theory not only a tool for the theoretical
analysis but also a tool for creating practical algorithms for estimating
multidimensional functions.
•Statistical learning plays a key role in many areas of science, finance
and industry.
 MATRUSRI
ENGINEERING COLLEGE

Statistical modeling from the perspective of

supervised learning
•In supervised learning, an algorithm is given samples
that are labeled in some useful way. For example, the
samples might be descriptions of apples, and the labels
could be whether or not the apples are edible.
•Supervised learning involves learning from a training
set of data. Every point in the training is an input-
output pair, where the input maps to an output. The
learning problem consists of inferring the function that
maps between the input and the output in a predictive
fashion, such that the learned function can be used to
predict output from future input.
 MATRUSRI
ENGINEERING COLLEGE

Machine learning Vs Statistics

 MATRUSRI
ENGINEERING COLLEGE

Machine Learning Vs Statistical Modelling

•Machine Learning is … an algorithm that can
learn from data without relying on rules-
based programming.
•Statistical Modelling is … formalization of
relationships between variables in the form of
mathematical equations.
 MATRUSRI
ENGINEERING COLLEGE

Machine Learning Vs Statistical Modelling

•Machine Learning is … a subfield of computer

science and artificial intelligence which deals
with building systems that can learn from
data, instead of explicitly programmed
instructions.
•Statistical Modelling is … a subfield of
mathematics which deals with finding
relationship between variables to predict an
outcome
 MATRUSRI
ENGINEERING COLLEGE

Examples of the learning problems

•Predict whether a patient, hospitalized due to a heart attack, will have a
second heart attack. The prediction is to be based on demographic, diet
and clinical measurements for that patient.
•Predict the price of a stock in 6 months from now, on the basis of
company performance measures and economic data.
•Estimate the amount of glucose in the blood of a diabetic person, from
the infrared absorption spectrum of that person’s blood.
•Identify the risk factors for prostate cancer, based on clinical and
demographic variables.
 MATRUSRI
ENGINEERING COLLEGE

Sample problem
•Consider a set of patients coming for treatment in a certain clinic.
•Let A denote the event that a “Patient has liver disease” and B the
event that a “Patient is an alcoholic.”
•It is known from experience that 10% of the patients entering the
clinic have liver disease and 5% of the patients are alcoholics.
•Also, among those patients diagnosed with liver disease, 7% are
alcoholics.
•Given that a patient is alcoholic, what is the probability that he will
have liver disease?
 MATRUSRI
ENGINEERING COLLEGE

Using the notations of probability,

P(A)= 10% = 0.10
P(B)= 5% = 0.05
P(B∣A)= 7% = 0.07

P(A∣B)= P(B∣A)P(A) / P(B)

= 0.07×0.10/ 0.05
= 0.14
 MATRUSRI
ENGINEERING COLLEGE

A good learner is one that accurately predicts such an outcome.

•In essence, a statistical learning problem is learning from

the data.
• In a typical scenario, we have an outcome measurement,
usually quantitative (such as a stock price) or categorical
(such as heart attack/no heart attack), that we wish to predict
based on a set of features (such as diet and clinical
measurements).
•We have a Training Set which is used to observe the
outcome and feature measurements for a set of objects.
•Using this data we build a Prediction Model, or
a Statistical Learner , which enables us to predict the
outcome for a set of new unseen objects.
 MATRUSRI
ENGINEERING COLLEGE

Statistics + Machine Learning=Statistical Learning

 MATRUSRI
ENGINEERING COLLEGE

Questions & Answers

1) What are the different choices for performance evaluation.
2) What k-fold cross validation.
3) Differentiate bias & variance
4) Explain the terms underfitting & overfitting
5) What are the different evaluation metrics for classification &
regression
6) Explain the terminology in confusion matrix.
7) Explain about ROC & AUC