Confusion Matrix &

Cross Validation
 Confusion Matrix
• Machine learning models are increasingly used in
various applications to classify data into different
categories.
• However, evaluating the performance of these models
is crucial to ensure their accuracy and reliability. One
essential tool in this evaluation process is the confusion
matrix.
• The confusion matrix is a matrix used to determine the
performance of the classification models for a given set of
test data.
• It can only be determined if the true values for test data are
known. The matrix itself can be easily understood, but the
related terminologies may be confusing.
• Since it shows the errors in the model performance in the
form of a matrix, hence also known as an error matrix.
Some features of Confusion matrix are given below:
• Classification Models have multiple categorical outputs. Most
error measures will calculate the total error in our model, but
we cannot find individual instances of errors in our model.
• The model might misclassify some categories more than
others, but we cannot see this using a standard accuracy
measure.
• Furthermore, suppose there is a significant class imbalance in
the given data. In that case, i.e., a class has more instances of
data than the other classes, a model might predict the
majority class for all cases and have a high accuracy score;
when it is not predicting the minority classes. This is where
confusion matrices are useful.
A confusion matrix presents a table layout of the different
outcomes of the prediction and results of a classification
problem and helps visualize its outcomes.

It plots a table of all the predicted and actual values of a

classifier.
2x2 Confusion Matrix
• True Positive: The number of times our actual positive values
are equal to the predicted positive. You predicted a positive
value, and it is correct.
• False Positive: The number of times our model wrongly
predicts negative values as positives. You predicted a negative
value, and it is actually positive.
• True Negative: The number of times our actual negative values
are equal to predicted negative values. You predicted a negative
value, and it is actually negative.
• False Negative: The number of times our model wrongly
predicts negative values as positives. You predicted a negative
value, and it is actually positive.
Confusion
Matrix Metrics
• Consider a confusion matrix made
for a classifier that classifies
people based on whether they
speak English or Spanish.
• From the above diagram, we can
see that:
• True Positives (TP) = 86
• True Negatives (TN) = 79
• False Positives (FP) = 12
• False Negatives (FN) = 10
•Just from looking at the matrix, the
performance of our model is not very clear.
To find how accurate our model is, we use
the following metrics:

•Accuracy: The accuracy is used to find the

portion of correctly classified values. It tells
us how often our classifier is right. It is the
sum of all true values divided by total values.

In this case:
Accuracy = (86 +79) / (86 + 79 + 12 + 10)
= 0.8823 = 88.23%
•Precision: Precision is used to
calculate the model's ability to
classify positive values
correctly. It is the true positives
divided by the total number of
predicted positive values.

In this case,
Precision = 86 / (86 + 12) =
0.8775 = 87.75%
•Recall: It is used to calculate the
model's ability to predict positive
values. "How often does the
model predict the correct positive
values?". It is the true positives
divided by the total number of
actual positive values.

In this case,
Recall = 86 / (86 + 10) = 0.8983
= 89.83%
•F1-Score: It is the harmonic
mean of Recall and Precision.
It is useful when you need to
take both Precision and Recall
into account.

In this case,
F1-Score = (2* 0.8775 *
0.8983) / (0.8775 + 0.8983) =
0.8877 = 88.77%
 • To scale a confusion matrix, increase the
Scaling a number of rows and columns. All the True
Confusion Matrix Positives will be along the diagonal. The other
values will be False Positives or False Negatives.
• https://www.simplilearn.com/tutorials/machine-learning-
tutorial/confusion-matrix-machine-learning
 Cross Validation
• Cross-validation is a technique for validating the model
efficiency by training it on the subset of input data and
testing on previously unseen subset of the input data.
• We can also say that it is a technique to check
how a statistical model generalizes to an
independent dataset.
• In machine learning, there is always the need to test the
stability of the model. It means based only on the
training dataset; we can't fit our model on the training
dataset.
• For this purpose, we reserve a particular sample of the
dataset, which was not part of the training dataset.
• After that, we test our model on that sample before
deployment, and this complete process comes under
cross-validation.
• This is something different from the general train-test
split.
steps of cross-validations
• Reserve a subset of the dataset as a validation set.
• Provide the training to the model using the training
dataset.
• Now, evaluate model performance using the validation
set. If the model performs well with the validation set,
perform the further step, else check for the issues.
Methods used for Cross-Validation
1.Validation Set Approach
2.Leave-P-out cross-validation
3.Leave one out cross-validation
4.K-fold cross-validation
5.Stratified k-fold cross-validation
Validation Set Approach
• We divide our input dataset into a training set and test
or validation set in the validation set approach. Both the
subsets are given 50% of the dataset.
• But it has one of the big disadvantages that we are just
using a 50% dataset to train our model, so the model
may miss out to capture important information of the
dataset. It also tends to give the underfitted model.
Leave-P-out cross-validation
• In this approach, the p datasets are left out of the
training data. It means, if there are total n datapoints in
the original input dataset, then n-p data points will be
used as the training dataset and the p data points as
the validation set. This complete process is repeated for
all the samples, and the average error is calculated to
know the effectiveness of the model.
• There is a disadvantage of this technique; that is, it can
be computationally difficult for the large p.
Leave one out cross-validation
• This method is similar to the leave-p-out cross-validation, but
instead of p, we need to take 1 dataset out of training. It
means, in this approach, for each learning set, only one
datapoint is reserved, and the remaining dataset is used to
train the model. This process repeats for each datapoint.
Hence for n samples, we get n different training set and n test
set.
• It has the following features:
• In this approach, the bias is minimum as all the data points are used.
• The process is executed for n times; hence execution time is high.
• This approach leads to high variation in testing the effectiveness of
the model as we iteratively check against one data point.
K-Fold Cross-Validation
• K-fold cross-validation approach divides the input
dataset into K groups of samples of equal sizes. These
samples are called folds. For each learning set, the
prediction function uses k-1 folds, and the rest of the
folds are used for the test set. This approach is a very
popular Computer Vision approach because it is easy to
understand, and the output is less biased than other
methods.
steps for k-fold cross-validation
• Split the input dataset into K groups
• For each group:
• Take one group as the reserve or test data set.
• Use remaining groups as the training dataset
• Fit the model on the training set and evaluate the performance of
the model using the test set.
• Let's take an example of 5-folds cross-validation. So, the
dataset is grouped into 5 folds. On 1st iteration, the first fold is
reserved for test the model, and rest are used to train the
model. On 2nd iteration, the second fold is used to test the
model, and rest are used to train the model. This process will
continue until each fold is not used for the test fold.
Stratified k-fold cross-validation
• This technique is similar to k-fold cross-validation with
some little changes. This approach works on
stratification concept, it is a process of rearranging the
data to ensure that each fold or group is a good
representative of the complete dataset. To deal with the
bias and variance, it is one of the best approaches.
• It can be understood with an example of housing prices,
such that the price of some houses can be much high
than other houses. To tackle such situations, a stratified
k-fold cross-validation technique is useful.
Holdout Method
• This method is the simplest cross-validation technique
among all. In this method, we need to remove a subset
of the training data and use it to get prediction results
by training it on the rest part of the dataset.
• The error that occurs in this process tells how well our
model will perform with the unknown dataset. Although
this approach is simple to perform, it still faces the issue
of high variance, and it also produces misleading results
sometimes.
Comparison of Cross-validation to
train/test split
• Train/test split: The input data • Cross-Validation dataset: It is
is divided into two parts, that are
used to overcome the disadvantage
training set and test set on a
of train/test split by splitting the
ratio of 70:30, 80:20, etc. It
provides a high variance, which is dataset into groups of train/test splits,
one of the biggest disadvantages. and averaging the result. It can be
• Training Data: The training data is used if we want to optimize our model
used to train the model, and the that has been trained on the training
dependent variable is known. dataset for the best performance. It is
• Test Data: The test data is used to more efficient as compared to
make the predictions from the train/test split as every observation is
model that is already trained on the
training data. This has the same used for the training and testing both.
features as training data but not the
part of that.
Limitations of Cross-Validation
• For the ideal conditions, it provides the optimum output. But
for the inconsistent data, it may produce a drastic result. So,
it is one of the big disadvantages of cross-validation, as
there is no certainty of the type of data in machine learning.
• In predictive modeling, the data evolves over a period, due
to which, it may face the differences between the training
set and validation sets. Such as if we create a model for the
prediction of stock market values, and the data is trained on
the previous 5 years stock values, but the realistic future
values for the next 5 years may drastically different, so it is
difficult to expect the correct output for such situations.
Applications of Cross-Validation
• This technique can be used to compare the
performance of different predictive modeling methods.
• It has great scope in the medical research field.
• It can also be used for the meta-analysis, as it is already
being used by the data scientists in the field of medical
statistics.
• https://www.youtube.com/watch?v=PF2wLKv2lsI&t=151
s
x y
1 1.2
2 2.3
•Given the dataset:
3 2.9 •We will perform 5-fold cross-validation to
evaluate the performance of a linear
4 4.1 regression model on this dataset.
5 5.5 •Cross-validation provides a more reliable
estimate of model performance by
6 6.7 partitioning the data into different subsets
(folds) and evaluating the model's
performance on unseen data from each fold.
7 7.8
8 8.9
9 9.6
10 10.8
Steps:
1.Split the data into 5 folds.
2.For each fold:
1.Use 4 folds for training and 1 fold for testing.
2.Fit a linear regression model to the training data.
3.Calculate the mean squared error (MSE) on the test fold.
3.Compute the average MSE across all folds as the
overall performance metric.
• Let's perform this cross-validation.
Results of 5-Fold Cross-Validation
• Mean Squared Errors (MSE) for each fold:
• Fold 1: 0.0558
• Fold 2: 0.0452
• Fold 3: 0.1156
• Fold 4: 0.0093
• Fold 5: 0.0533
• Average MSE across all folds:
• Average MSE=0.0558