Scribd
Upload
14 views7 pages

Evaluation 1 7

Uploaded by

Manish Mavi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
14 views7 pages

Evaluation 1 7

Uploaded by

Manish Mavi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

CHAPTER-11 EVALUATION

AI PROJECT CYCLE
Problem Scoping ----- > Data Acquisition ---- > Data
Exploring

------ > Modelling ---- > Evaluation

Evaluation
 Evaluation is the final stage in AI Project Cycle. Once a model
has been made and trained, it needs to go through proper
testing so that one can calculate the efficiency and performance
of the model. Hence, the model is tested with the help of
Testing Data.
 Evaluation is the process of understanding the reliability and
final performance of any AI model by giving the test data set into
the model and comparing it`s output with actual answers.
Q. We must keep in mind that it is not advisable to use the data
that we used to create the model to evaluate it. Why?
Or
What do you think by the term ‘Overfitting’? Explain.
Ans-Training data must not be used for evaluation purposes
because a model simply remembers the whole of training data,
therefore always predicts the correct output for any point in the
training set whenever training data is fed again. But it gives very
wrong answers if a new dataset is introduced to the model. This
situation is known as overfitting.
-----------------
Evaluation is basically done by two things:
1. Prediction: The output given by the machine after training
and testing the data is known as Prediction. (Output of the
machine)
2. Reality: Reality is the real situation and real scenario where
prediction has been made by the machine. (Reality or truth)

EXAMPLES OR SCENARIOS
1. Case 1
Is this a Football?

1. Prediction = YES
2. Reality = YES
3. True Positive
Here, we can see in the picture that it’s a football. The model’s
prediction is Yes which means it's football. The Prediction
matches Reality. Hence, this condition is
termed as True Positive.

2. Case 2
Is this a Football?
1. Prediction = NO
2. Reality = NO
3. True Negative
Here this is Not an image of Football hence the reality is
No. In this case, the machine has predicted it correctly as a
No. Therefore, this condition is termed as True Negative.

3. Case 3
Is this a Football?

1. Prediction = YES
2. Reality = NO
3. False Positive (Type 1 Error)
Here the reality is that it is not Football. But the machine
has incorrectly predicted that this is Football. This case is
termed False Positive.

4. Case 4
Is this a Football?

1. Prediction = NO
2. Reality = YES
3. False Negative (Type 2 Error)

Here, a Football has been in a different look because of


which the Reality is Yes but the machine has incorrectly
predicted it as a No which means the machine predicts
that it is not Football.
Therefore, this case becomes False Negative.

Confusion Matrix-
 The comparison between the results of Prediction and reality is
called the Confusion Matrix.
 It is a record that helps in evaluation.
 It is not a calculation; it is a performance measurement for
machine learning classification problems where output can be
two or more classes.

Now again consider the example of football:


Result of comparison between prediction and reality can be
recorded in a confusion matrix.

Parameters to evaluate the Model-

There are four methods to evaluate the model.


Accuracy-
 It is the percentage of correct predictions out of all the
observations.
 A prediction is correct if it matches the reality.
 All True positive and True Negative are the cases in which
the Prediction matches with reality.

Accuracy formula:
( )

OR
( )
( )

Note: Here Total cases/observations= TP+TN+FP+FN

EXAMPLE
Let us again take the football example.
Assume that the model always predicts that object is not football.
But in reality, there is a 5% chances of object being a football. In
this case, for 95 cases, the model will be right but for 5 cases in
which the object was a football, the model predicted it to be not a
football.
Here,
1. True Positives = 0
2. True Negatives = 95
3. Total cases = 100
4. Therefore, accuracy becomes:
95+0/100 = 95%

Precision Parameter-
It is defined as the percentage of true positive cases versus
all the cases where the prediction is true. It takes True
Positives and False Positives.

Note: If Precision is high, this means the True Positive cases


are more, giving lesser False predictions.

In the above example, all the Positive conditions would be taken


into account that is,
 True Positive (Prediction = Yes and Reality = Yes)
 False Positive (Prediction = Yes and Reality = No)

Recall Parameter
 Recall is an important metric because it measures the ability
of a model to identify all of the positive instances in a dataset.
 It is calculated as the number of true positives divided by the
total number of positives, including both true positives and
false negatives.

Note:
 Precision Focuses on the correctness of positive predictions while
Recall emphasizes capturing all relevant instances.
 Precision evaluates the correctness of positive predictions,
while recall determines how well the model recognizes all
pertinent instances.

F1 Score
 It can be defined as the measure of balance between precision
and recall.

 An ideal situation is there when we have a value of 1 for both


Precsion and Recall. Then F1 score would also be 1(100%). It
is known as the perfect value for F1 Score. A model is having
a good performance if F1 Score is high.

You might also like