Scribd
Upload
47 views

07 ML Classificaion Advanced Kappa

This document discusses various machine learning classification topics taught by Dr. Hikmat Ullah Khan. It includes examples of calculating classification metrics like accuracy, sensitivity and specificity using a confusion matrix. It also discusses evaluating classifier accuracy using holdout and cross-validation methods, issues that can affect model selection, and techniques for increasing accuracy like ensemble methods and addressing the no free lunch theorem.

Uploaded by

In Tech
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
47 views

07 ML Classificaion Advanced Kappa

This document discusses various machine learning classification topics taught by Dr. Hikmat Ullah Khan. It includes examples of calculating classification metrics like accuracy, sensitivity and specificity using a confusion matrix. It also discusses evaluating classifier accuracy using holdout and cross-validation methods, issues that can affect model selection, and techniques for increasing accuracy like ensemble methods and addressing the no free lunch theorem.

Uploaded by

In Tech
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 18

Classification

Advanced Topics
Dr. Hikmat Ullah Khan

1 (Dr Hikmat Ullah Khan)


Actual Class\Predicted cancer = yes cancer = no Total
class
cancer = yes 90 210 300

cancer = no 140 9560 9700

Total 230 9770 10000

Calculate
1. Proper PEMs

2 (Dr Hikmat Ullah Khan)


Classifier Evaluation Metrics: Example

Actual Class\Predicted cancer = cancer = Total Recognition(


class yes no %)
cancer = yes 90 210 300 30.00
(sensitivity
cancer = no 140 9560 9700 98.56
(specificity)
Total 230 9770 10000 96.40
(accuracy)

 Precision = 90/230 = 39.13%


 Recall = 90/300 = 30.00%

3
3
Class Labelling

• Amazon Mechanical Turk


– https://www.mturk.com/mturk/welcome
– https://en.wikipedia.org/wiki/Amazon_Mechanical_Turk
• Doing what Computers can not do (so far)…
– Opportunities
• Is Class Labelling a Sensitive task?
– Health
– Social Issues?
• R u interested in labelling?
• R u doing it already?

4 (Dr Hikmat Ullah Khan)


5 (Dr Hikmat Ullah Khan)
Problems in Labelling
 If we take labelling from multiple judges
 What happens if the agreement does not occur?
 How to solve it?
 How to measure it Statistically?

6 (Dr Hikmat Ullah Khan)


Kappa Statistics

7 (Dr Hikmat Ullah Khan)


Interpretation of Cohen’s kappa.

Value of Kappa Level of Agreement % of Data that are Reliable

0–.20 None 0–4%

.21–.39 Minimal 4–15%

.40–.59 Weak 15–35%

.60–.79 Moderate 35–63%

.80–.90 Strong 64–81%

Above.90 Almost Perfect 82–100%

8 (Dr Hikmat Ullah Khan)


Use of Cohen Kappa Statistics
 Agreement between Raters or Annotators
 May also be applied for Agreed results of Human vs
Classifiers

 Exercise

9 (Dr Hikmat Ullah Khan)


Exercise

 The following hypothetical data come from a medical test where


two radiographers rated 50 images for needing further study.
The researchers (A and B) either said Yes (for further study) or
No (No further study needed).
 20 images were rated Yes by both and 15 images were rated No
by both.
 Overall, rater A said Yes to 25 images and No to 25. Overall,
Rater B said Yes to 30 images and No to 20.
 Computer Kappa Statistics using K=OA-AC/(1-AC)
equation.

10 (Dr Hikmat Ullah Khan)


Evaluating Classifier Accuracy:
Holdout Methods
 Holdout method
 Given data is randomly partitioned into two independent sets
 Training set (e.g., 2/3) for model construction
 Test set (e.g., 1/3) for accuracy estimation
 Overfitting and under-fitting can be based on wrong split
 90-10
 50-50
 Standard usually 70-30
 Random sampling:
 a variation of holdout
 Repeat holdout k times,
 accuracy = avg. of the accuracies obtained

11 (Dr Hikmat Ullah Khan)


11
Evaluating Classifier Accuracy:
Cross-Validation Methods
 Cross-validation
 (k-fold, where k = 10 is most popular)
 Randomly partition the data into k mutually exclusive subsets,
 Each approximately equal size
 At i-th iteration, use Di as test set and others as training set

 Which is better, Hold out or Cross Validation?


 Why?

12 (Dr Hikmat Ullah Khan)


12
Issues and Improving Accuracy

Insight into Improvement

13 (Dr Hikmat Ullah Khan)


Issues Affecting Model Selection
 Accuracy
 classifier accuracy: predicting class label
 Speed
 time to construct the model (training time)
 time to use the model (classification/prediction time)
 Robustness:
 Handling noise and missing values
 Scalability:
 Efficiency in disk-resident databases

14 (Dr Hikmat Ullah Khan)


14
Ensemble Methods:
Increasing the Accuracy

 Ensemble methods
 Use a combination of models to increase accuracy
 Combine a series of k learned models, M1, M2, …, Mk, with the
aim of creating an improved model M*

15 (Dr Hikmat Ullah Khan)


15
Ensemble Methods: Increasing the Accuracy

 Ensemble:
 combining a set of heterogeneous classifiers

 Popular ensemble methods


 Bagging: (Bootstrap Aggregation)
 Averaging the prediction over a collection of
classifiers
 Boosting:
 weighted vote with a collection of classifiers
16 (Dr Hikmat Ullah Khan)
16
No Free Lunch Theorem

• RELATIVITY
• Learning algorithm 1 is better than learning
algorithm 2 ?
• Such statements are true about the relevant target
functions
• THUMB S RULE
• Experience with a broad range of techniques is the
best insurance for solving arbitrary new
classification problems

17 (Dr Hikmat Ullah Khan)


18 (Dr Hikmat Ullah Khan)

You might also like