Model Evaluation

Lecture 04
Discussion
2

Beware of your
deeds in this
life, for they
are adding up
in the next
Agenda:
•A Quick Review: Machine Learning Models
•Model Validation
•Objective Function and Cost Functions
•Predictive Vs Descriptive Models
•Model Evaluation Methods
•Hold-Out Method
•Cross Validation
•Types of Cross Validation
•Bootstrap Sampling
•Lazy Vs Eager Learners
•Activity

3
Machine Learning Models
• Some concepts are going to be discussed in detail
later: Classification Tasks
• Fitting a model:
• Learning is done to fit a model
• Best representation out of the data to answer unknown
facts
• Generic Model -> Trained Model (Model Parameters)
• Trained model is prepared to answer
• Facts that are already hidden inside the recorded
observations

4
Model Validation:
• process by which we ensure that our models can
perform acceptable in “the real world.”
• In more technical terms:
model validation allows you to predict how your
model will perform on datasets not used in the
training

5
Training Vs Testing Error:
• Model learns from the mistakes done during the training
process
• During training questions and their answers are shown to
the model and on the basis of errors it improved its
learning
• Just like student’s feedback in the class
• Training errors occurs during the learning process of the
model (when it is under learning or training)
• We want to reduce these errors
• Test data is provided to check the model performance
before deployment process
• E.g. R Squared that we used in Regression

6
Model Validation (Recap):

Training Error: We get by calculating the classification error of a model on the same
data the model was trained on

Test Error: We get this by using two completely disjoint datasets: one to train the
model and the other to calculate the classification error. Both datasets need to
have values for y. The first dataset is called training data and the second, test data.

7
Objective Function and Cost Function:
• Objective Function
• Objective: What is prime to achieve?
• Our objective is to minimize the prediction errors
• Takes data and model and attempts to find best values for parameters that
maximize the reward.
• Linear Regression: We tried to find the optimized o/p (Slope and Intercept)
• Finds out the values of model parameters based on any optimization
technique (OLS/LSM, Gradient Decent etc.)
• Cost Function/Error Function
• Error Function: During training a single error on one example
• Cost Function: On all examples accumulated calculated error
*We want to minimize the cost function and it is done by considering objective function
by using some optimization strategy

8
Machine Learning Models
•Needs training data to learn
•Needs test data to evaluate performance
•Predictive Vs. Descriptive Models
• Predictive Models (supervised learning – maps input
features to target)
• Descriptive Models (Unsupervised Learning – learns
hidden facts from data)

9
Predictive Vs Descriptive Models:
• Predictive Models
• Holds the predictive strength
• Builds a relationship from observations in the form of a function
that is used to give output of unknown instances
• Supervised Learning
• Descriptive Models
• Don’t give any o/p or answer but describe the provided instances
• They show the learned relationship and hidden patterns inside
the provided data
• Retrieve hidden facts out of the data which were unknown for us
• No fixed answer
• Unsupervised Learning

10
Training a Model:

• Main three methods of model

evaluation
• Hold-Out Method
• Cross Validation
• LOOCV
• Bootstrap Sampling
• Lazy Vs. eager learners

11
Hold Out Method

12
 Hold Out Method

• Data is split into training and test partition.

(usually, 80% and 20% respectively
• A small portion is held- out for testing
• Model learns on the training set with features and
associated target value
• After training, the model is evaluated on the test
set using an evaluation metric such R-Squared

13
 Hold Out Method
• Some times a third Partition i.e. validation data is
also factored out
• Training-validation-test data
• Validation data is used in place of test data after
each iteration of training to tune the model
parameters
• Model is tested on validation data and then training
continues to tune the model parameters for better
performance
• Test data is used only one at the end to evaluate the
performance of the final model

14
Hold on! point of ponder
• We validated our model once
• What if the split we made just happened to be very
conducive to this model?
• Didn’t we significantly reduce the size of our training
dataset by splitting it like that?

15
Hold Out Method-Limitation:
•Very restrictive in case of small volume of
dataset
•Partitioning can result in unbalanced class
representation in the training data
• Dividing the data into 3 partitions can result in
uneven representation of certain types of
instances (classificiation – unblanaced classes
problem)
• The test data may also be unbalanced
•Stratified Random Sampling may be used to
avoid unbalanced class problem to some extent
16
Cross Validation:
It is the process by which the machine learning
models are evaluated on a separate set known as
validation set or hold-out set with which the best
hyper-parameters are found, so that we get the
optimal model, that can be used on future data and
which is capable of yielding the best possible
predictions

17
Cross Validation:

Cross Validation Cycle

18
Types of Cross Validation in Data Science:

19
K-Fold Cross Validation Method
• Repeated hold-
out application
• K iteration of hold-out
validation are
undertaken
• The dataset is
partitioned into k
disjoint random
partitions called folds
• One fold is used as test
while the rest k-1 folds
are used for training
• The process is repeated k
times with next fold as
test fold

20
K-Fold Cross Validation Method

K = 10 21
 K-Fold Cross Validation Method
• 5 Fold and 10 fold cross validation methods are very popular
• In 10 fold CV, 10 folds of data are created by random sampling
• Each fold is disjoint with each other fold
• One after the other 1 fold is taken as test while the rest 9 folds
are used for training the model
• After k iterations the average model performance across all folds is
reported
• A special variant of k-fold validation is LOOCV
• Leave One out Cross Validation
• In LOOCV only one record is left out for validation and the rest of n-1
records are used for training in each iteration (n is the total number of
records)

22
LOOCV

Illustration of leave one out cross-validation (LOOCV) when n = 8 observations.

A total of 8 models will be trained and tested

23
Bootstrap Sampling

• Used to pick out training and test data from a data set.
• Usually employed when the size of data set is very small
• It randomly picks n instances from data set with replacement
• An instance may be duplicated in the training set
• It can create infinitely many training data sets of n instances from
a dataset of size n
• Records may be repeated multiple times.

24
 Lazy Vs. Eager Learners
• Eager Learners
• Build a generalized representation of learning into a
model
• Take more time in training
• Follows typical ML STEP
• Abstraction -> generalization->model
• Fast in predictions
• Lazy Learners
• Skip the learning phase
• No abstraction or generalization
• Depend on the training data for predictions
• Also called instance based learning or non-parameteric learning
• Take more time in prediction
25
Supervised Learning Models –
Capabilities Over Vs. Underfitting of Models
• Underfitting – Model too simple to capture essential details
• Overfitting - Model is overly complex and lacks generalization

26
Supervised Learning Models - Errors

• Model Error
• Incorrect prediction
• Errors due to Bias
• Generally resulted due to under
fitting
• Errors due to Variance
• Results due to overfitting
• A low bias and low variance
model is desirable

27
Thank you
Any Question?

28