Model Evaluation

Introduction to Machine Learning

Yidnekachew Kibru, Department of Information systems

Lecture 5
2
Agenda
▪Data processing
▪ Data cleaning and transforming
▪ Feature selection and visualization
▪Model selection and tuning
▪Over-fitting and Under-fitting
▪ Bias, variance
▪Cross-Validation and Re-sampling methods
▪ K-Fold Cross-Validation
▪Gradient descent (batch, stochastic)
▪Performance evaluation methods
3
Data processing
▪Data Processing is the task of converting data from a
given form to a much more usable and desired form
▪ i.e., making it more meaningful and informative.
▪Using Machine Learning algorithms, mathematical
modeling, and statistical knowledge, this entire
process can be automated.
▪The output of this complete process can be in any
desired form like graphs, videos, charts, tables,
images, and many more,
4
Data - Preprocessing
▪Is technique of preparing the raw data to make it suitable
for a building and training ML models.
▪Why we need it
▪ Real-world data is incomplete, inconsistent, inaccurate and
often lacks specific attribute values/trends.
▪ Duplicate or missing values may give an incorrect view of the
overall statistics of data.
▪ Outliers and inconsistent data points often tend to disturb the
model’s overall learning, leading to false predictions.
▪It is a common thumb rule in ML that the greater the
amount of data we have, the better models we can train.
5
Data - Preprocessing
▪Features in machine learning
▪ Individual independent variables that operate as an input
in our machine learning model.
▪ They can be thought of as representations or attributes
that describe the data and help the models to predict the
classes/labels.
▪ Features in a structured dataset like in a CSV format refer
to each column representing a measurable piece of data
that can be used for analysis:
▪ E.g., Name, Age, Sex, Fare, and so on.
6
4 Steps in Data Preprocessing
7
Data Preprocessing: Cleaning
▪Missing values:- solve this issue by:
1. Ignore those tuples:
▪ when dataset is huge and numerous missing values are
present within a tuple.
2. Fill in the missing values
▪ There are many methods to achieve this, such as
▪ filling in the values manually,
▪ predicting the missing values using regression method, or numerical
methods like attribute mean.
8
Data Preprocessing: Cleaning…
▪Noisy Data: It involves removing a random error or
variance in a measured variable.
1. Binning: Works on sorted data values to smoothen any
noise present in it.
▪ The data is divided into equal-sized bins, and each bin/bucket is
dealt with independently.
▪ All data in a segment can be replaced by its mean, median or
boundary values.
2. Regression: is used for prediction.
▪ smoothen noise by fitting all the data points in a regression function.
3. Clustering: Creation of groups/clusters from data having
similar values.
9
Data Preprocessing: Cleaning…
▪Removing outliers:
▪ Clustering techniques group together similar data points.
▪ The tuples that lie outside the cluster are
outliers/inconsistent data.
10
Data Preprocessing: Cleaning…
11
Data Preprocessing…
▪Data Integration: merge the data present in multiple
sources into a single larger data store like a data
warehouse.
▪Data Transformation: Consolidate data into alternate
forms by changing the value, structure, or format of
dat.
▪Data Reduction: The size of the dataset in a data
warehouse can be too large to be handled by data
analysis and data mining algorithms.
12
Data Preprocessing: Best practices
▪The first step in Data Preprocessing is to understand your
data.
▪Use statistical methods or pre-built libraries that help you
visualize the dataset and give a clear image of how your
data looks in terms of class distribution.
▪Summarize your data in terms of the number of duplicates,
missing values, and outliers present in the data.
▪Drop the fields you think have no use for the modeling or
are closely related to other attributes.
▪Do some feature engineering and figure out which
attributes contribute most towards model training.
13
Feature selection and visualization
▪Garbage in Garbage out (GIGO)
▪ Whatever goes in, comes out.
▪ If we put garbage into our model, we can expect the
output to be garbage too.
▪ In this case, garbage refers to noise in our data.
▪Feature Selection is the method of reducing the input
variable to your model by using only relevant data
and getting rid of noise in data.
14
Feature selection and visualization…
15
Model selection and tuning

• Model selection is the process of

selecting one final ML model
from among a collection of
candidate ML models for a
training dataset.

• Selecting a model is not enough

unless you know the most
suitable hyperparameters to
progress.
16
Model selection and tuning…
▪Considerations in Model Selection
▪ In a business project, model selection is not only a technical
problem
▪ There are many considerations such as
▪ business requirements
▪ cost
▪ performance
▪ state-of-the-art
▪ maintenance
▪ It is difficult or impossible to find the best model meeting all
criteria.
▪ The best model in terms of technical performance may not be
the best in business.
17
Model selection and tuning…
▪Model tuning:
▪ is the experimental process of finding the optimal values
of hyperparameters to maximize model performance.
▪Hyperparameters are the set of variables whose
values cannot be estimated by the model from the
training data.
▪These values control the training process.
18
Model selection and tuning…
▪Hyperparameter: is a configuration that is external to
the model and whose value cannot be estimated from
data.
▪ They are often used in processes to help estimate model
parameters.
▪ They are often specified by the practitioner.
▪ They can often be set using heuristics.
▪ They are often tuned for a given predictive modeling
problem.
19
Model selection and tuning…
▪ Hyperparameters
▪ Train-test split ratio
▪ Learning rate in optimization algorithms (e.g. gradient descent)
▪ Choice of optimization algorithm (e.g., gradient descent, Adam optimizer)
▪ Choice of activation function in a NN layer (e.g. Sigmoid, ReLU, Tanh)
▪ The choice of cost or loss function the model will use
▪ Number of hidden layers in a NN
▪ The drop-out rate in NN (dropout probability)
▪ Number of iterations (epochs) in training a NN
▪ Number of clusters in a clustering task
▪ Kernel or filter size in convolutional layers
▪ Pooling size
▪ Batch size
20
Over-fitting and Under-fitting
▪Variance: is the difference between the error rate of
training data and testing data.
▪ If the difference is high, then it’s called high variance and
when the difference of errors is low then it’s called low
variance.
▪ Usually, we want to make a low variance for generalized our
model
▪Bias: Assumptions made by a model to make a function
easier to learn.
▪ It is the error rate of the training data.
▪ When the error rate has a high value, we call it High Bias and
when the error rate has a low value, we call it low Bias.
21
Over-fitting and Under-fitting…
▪The main goal of each machine learning model is to
generalize well.
▪Generalization defines the ability of an ML model to
provide a suitable output by adapting the given set of
unknown input.
▪Overfitting: when the model does not make accurate
predictions on testing data.
▪Underfitting: when it cannot capture the underlying
trend of the data,
22
Over-fitting and Under-fitting…
▪Overfitting: When a model gets trained with so much
data, it starts learning from the noise and inaccurate data
entries in our data set.
▪ Testing with test data results in High variance.
▪ Then the model does not categorize the data correctly,
because of too many details and noise.
▪Reasons for Overfitting :
▪ High variance and low bias
▪ The model is too complex
▪ The size of the training data
23
Over-fitting and Under-fitting…
▪Techniques to reduce overfitting:
▪ Increase training data.
▪ Reduce model complexity.
▪ Early stopping during the training phase (have an eye
over the loss over the training period as soon as loss
begins to increase stop training).
▪ Ridge Regularization and Lasso Regularization
▪ Use dropout for neural networks to tackle overfitting.
24
Over-fitting and Under-fitting…
▪Underfitting: it only performs well on training data
but performs poorly on testing data.
▪ (It’s just like trying to fit undersized pants!)
▪ It destroys the accuracy of our machine learning model.
▪ Its occurrence simply means that our model or the
algorithm does not fit the data well enough.
▪ It usually happens when we have fewer data to build an
accurate model and also when we try to build a linear
model with fewer non-linear data.
25
Over-fitting and Under-fitting…
▪Reasons for Underfitting:
▪ High bias and low variance
▪ The size of the training dataset used is not enough.
▪ The model is too simple.
▪ Training data is not cleaned and also contains noise in it.
26
Over-fitting and Under-fitting…
▪Techniques to reduce underfitting:
▪ Increase model complexity
▪ Increase the number of features, performing feature
engineering
▪ Remove noise from the data.
▪ Increase the number of epochs or increase the duration of
training to get better results.
27
Over-fitting and Under-fitting…
28
Cross-Validation
▪Cross-validation is a technique for validating the
model efficiency by training it on the subset of input
data and testing on previously unseen subset of the
input data.
▪We can also say that it is a technique to check how a
statistical model generalizes to an independent
dataset.
▪This is something different from the general train-test
split.
29
Cross-Validation…
▪Basic steps of cross-validations are:
▪ Reserve a subset of the dataset as a validation set.
▪ Provide the training to the model using the training
dataset.
▪ Now, evaluate model performance using the validation
set.
▪ If the model performs well with the validation set, perform the
further step, else check for the issues.
30
Cross-Validation: K-fold cross-validation
▪K-fold cross-validation approach divides the input
dataset into K groups of samples of equal sizes.
▪ These samples are called folds.
▪ For each learning set, the prediction function uses k-1
folds, and the rest of the folds are used for the test set.
▪ This approach is a very popular CV approach because it
is easy to understand, and the output is less biased than
other methods.
31
Gradient descent
▪Gradient Descent Algorithm is used during the
backward propagation to update the parameters of
the model.
32
Performance evaluation methods
▪The most important thing you can do to properly
evaluate your model is to not train the model on the
entire dataset.
▪Some methods
▪ Confusion matrix
▪ Accuracy
▪ Precision
▪ Recall
▪ Specificity
33
Performance evaluation methods…
▪ Assumptions
▪ discuss things in terms of a binary classification problem where:
▪ we’ll have to find if an image is of a cat or a dog.
▪ a patient is having cancer (positive) or is found healthy
(negative).
▪ Some common terms :
▪ True positives (TP): Predicted positive and are actually positive
▪ False positives (FP): Predicted positive and are actually negative
▪ True negatives (TN): Predicted negative and are actually negative
▪ False negatives (FN): Predicted negative and are actually positive
34
Confusion Matrix
35
Accuracy
▪Accuracy is defined as the percentage of correct
predictions for the test data.
▪It can be calculated easily by dividing the number of
correct predictions by the number of total predictions.

▪The most used metric to judge a model and is not a clear

indicator of the performance.
▪The worse happens when classes are imbalanced.
36
Precision
▪Precision is defined as the fraction of relevant
examples (true positives) among all of the examples
which were predicted to belong in a certain class.
37
Recall/Sensitivity/True Positive Rate
▪Recall is defined as the fraction of examples which
were predicted to belong to a class with respect to all
of the examples that truly belong in the class.
38
Specificity
▪Percentage of negative instances out of the total
actual negative instances.
Thank You!

C0 - Personal Information
39