WEKA: Credibility Evaluating Whats Been Learned
6 likes2,786 views
This document discusses various techniques for evaluating machine learning models and comparing their performance, including: - Measuring error rates on separate test and training sets to avoid overfitting - Using techniques like cross-validation, bootstrapping, and holdout validation when data is limited - Comparing algorithms using statistical tests like paired t-tests - Accounting for costs of different prediction outcomes in evaluation and model training - Visualizing performance using lift charts and ROC curves to compare models - The Minimum Description Length principle for selecting the model that best compresses the data
![Predicting performanceExpected success rate = 100 – error rate (If error rate is also in percentage)We want the true success rateCalculation of true success rateSuppose we have expected success rate(f) = s/n, where s is the number of success out of a total n instancesFor large value of n, f follows a normal distributionNow we will predict the true success rate (p) based on the confidence percentage we want For example say our f = 75%, then p will lie in [73.2%,76.7%] with 80% confidence](https://image.slidesharecdn.com/credibilityevaluatingwhatsbeenlearned-100203234919-phpapp02/85/WEKA-Credibility-Evaluating-Whats-Been-Learned-3-320.jpg)
![Cost curvesC[+|-] is the cost of predicting + when the instance is –C[-|+] is the cost of predicting - when the instance is +](https://image.slidesharecdn.com/credibilityevaluatingwhatsbeenlearned-100203234919-phpapp02/85/WEKA-Credibility-Evaluating-Whats-Been-Learned-35-320.jpg)
![Minimum Description Length PrincipleThe 2nd one is easy. Just use informational loss functionFor 1st we need a method to encode the modelL[T] = “length” of the theoryL[E|T] = training set encoded wrt the theory](https://image.slidesharecdn.com/credibilityevaluatingwhatsbeenlearned-100203234919-phpapp02/85/WEKA-Credibility-Evaluating-Whats-Been-Learned-37-320.jpg)
WEKA: Credibility Evaluating Whats Been Learned
- 1. Credibility: Evaluating what’s Been Learned
- 2. Training and TestingWe measure the success of a classification procedure by using error rates (or equivalent success rates)Measuring success rate using training set is highly optimisticThe error rate on training set is called resubstitution errorWe have a separate test set for calculating success errorTest set should be independent of the training setAlso some time to improve our classification technique we use a validation setWhen we hold out some part of training set for testing (which is now not used for training), this process is called holdout procedure
- 3. Predicting performanceExpected success rate = 100 – error rate (If error rate is also in percentage)We want the true success rateCalculation of true success rateSuppose we have expected success rate(f) = s/n, where s is the number of success out of a total n instancesFor large value of n, f follows a normal distributionNow we will predict the true success rate (p) based on the confidence percentage we want For example say our f = 75%, then p will lie in [73.2%,76.7%] with 80% confidence
- 4. Predicting performanceNow using properties of statistics we know that the mean of f is p and the variance is p(1-p)/nTo use normal distribution we will have to make the mean of f = 0 and standard deviation = 1 So suppose our confidence = c% and we want to calculate pWe will use the two tailed property of normal distributionAnd also that the are covered by normal distribution is taken as 100% so the are we will leave is 100 - c
- 5. Predicting performanceFinally after all the manipulations we have ,true success rate as:Here, p -> true success rate f - > expected success rate N -> Number of instances Z -> Factor derived from a normal distribution table using the 100-c measure
- 6. Cross validationWe use cross validation when amount of data is small and we need to have independent training and test set from itIt is important that each class is represented in its actual proportions in the training and test set: Stratification An important cross validation technique is stratified 10 fold cross validation, where the instance set is divided into 10 foldsWe have 10 iterations with taking a different single fold for testing and the rest 9 folds for training, averaging the error of the 10 iterationsProblem: Computationally intensive
- 7. Other estimatesLeave-one-out:StepsOne instance is left for testing and the rest are used for trainingThis is iterated for all the instances and the errors are averagedLeave-one-out:AdvantageWe use larger training setsLeave-one-out:DisadvantageComputationally intensiveCannot be stratified
- 8. Other estimates0.632 BootstrapDataset of n samples is sampled n times, with replacements, to give another dataset with n instancesThere will be some repeated instances in the second setHere error is defined as:e = 0.632x(error in test instances) + 0.368x(error in training instances)
- 9. Comparing data mining methodsTill now we were dealing with performance predictionNow we will look at methods to compare algorithms, to see which one did betterWe cant directly use Error rate to predict which algorithm is better as the error rate might have been calculated on different data setsSo to compare algorithms we need some statistical testsWe use Student’s t- test to do this. This test help us to figure out if the mean error of two algorithm are different or not for a given confidence level
- 10. Comparing data mining methodsWe will use paired t-test which is a slight modification of student’s t-testPaired t-testSuppose we have unlimited data, do the following:Find k data sets from the unlimited data we haveUse cross validation with each technique to get the respective outcomes: x1, x2, x3,….,xk and y1,y2,y3,……,ykmx = mean of x values and similarly mydi = xi – yiUsing t-statistic:
- 11. Comparing data mining methodsBased on the value of k we get a degree of freedom, which enables us to figure out a z for a particular confidence valueIf t <= (-z) or t >= (z) then, the two means differ significantly In case t = 0 then they don’t differ, we call this null hypothesis
- 12. Predicting ProbabilitiesTill now we were considering a scheme which when applied, results in either a correct or an incorrect prediction. This is called 0 – loss functionNow we will deal with the success incase of algorithms that outputs probability distribution for e.g. Naïve Bayes
- 13. Predicting ProbabilitiesQuadratic loss function:For a single instance there are k out comes or classesProbability vector: p1,p2,….,pkThe actual out come vector is: a1,a2,a3,…..ak (where the actual outcome will be 1, rest all 0)We have to minimize the quadratic loss function given by:The minimum will be achieved when the probability vector is the true probability vector
- 14. Predicting ProbabilitiesInformational loss function:Given by:–log(pi)Minimum is again reached at true probabilitiesDifferences between Quadratic loss and Informational lossWhile quadratic loss takes all probabilities under consideration, Informational loss is based only on the class probability While quadratic loss is bounded as its maximum output is 2, Informational loss is unbounded as it can output values up to infinity
- 15. Counting the costDifferent outcomes might have different costFor example in loan decision, the cost of lending to a defaulter is far greater that the lost-business cost of refusing a loan to a non defaulterSuppose we have two class prediction. Outcomes can be:
- 16. Counting the costTrue positive rate: TP/(TP+FN)False positive rate: FP/(FP+TN)Overall success rate: Number of correct classification / Total Number of classificationError rate = 1 – success rateIn multiclass case we have a confusion matrix like (actual and a random one):
- 17. Counting the costThese are the actual and the random outcome of a three class problemThe diagonal represents the successful casesKappa statistic = (D-observed - D-actual) / (D-perfect - D-actual)Here kappa statistic = (140 – 82)/(200-82) = 49.2%Kappa is used to measure the agreement between predicted and observed categorizations of a dataset, while correcting for agreements that occurs by chanceDoes not take cost into account
- 18. Classification with costsExample Cost matrices (just gives us the number of errors):Success rate is measured by average cost per predictionWe try to minimize the costsExpected costs: dot products of vectors of class probabilities and appropriate column in cost matrix
- 19. Classification with costsSteps to take cost into consideration while testing:First use a learning method to get the probability vector (like Naïve Bayes) Now multiple the probability vector to each column of a cost matrix one by one so as to get the cost for each class/columnSelect the class with the minimum(or maximum!!) cost
- 20. Cost sensitive learningTill now we included the cost factor during evaluationWe will incorporate costs into the learning phase of a methodWe can change the ratio of instances in the training set so as to take care of costsFor example we can do replication of a instances of particular class so that our learning method will give us a model with less errors of that class
- 21. Lift ChartsIn practice, costs are rarely knownIn marketing terminology the response rate is referred to as the lift factorWe compare probable scenarios to make decisionsA lift chart allows visual comparisonExample: promotional mail out to 1,000,000 householdsMail to all: 0.1%response (1000)Some data mining tool identifies subset of 100, 000 of which 0.4% respond (400)A lift of 4
- 22. Lift ChartsSteps to calculate lift factor:We decide a sample sizeNow we arrange our data in decreasing order of the predicted probability of a class (the one which we will base our lift factor on: positive class)We calculate:Sample success proportion = Number of positive instances / Sample size Lift factor = Sample success proportion / Data success proportionWe calculate lift factor for different sample size to get Lift Charts
- 23. Lift ChartsA hypothetical lift chart
- 24. Lift ChartsIn the lift chart we will like to stay towards the upper left cornerThe diagonal line is the curve for random samples without using sorted dataAny good selection will keep the lift curve above the diagonal
- 25. ROC CurvesStands for receiver operating characteristicDifference to lift charts:Y axis showspercentage of true positive X axis shows percentage of false positives in samplesROC is a jagged curveIt can be smoothened out by cross validation
- 26. ROC CurvesA ROC curve
- 27. ROC CurvesWays to generate cost curves(Consider the previous diagram for reference)First way:Get the probability distribution over different folds of dataSort the data in decreasing order of the probability of yes classSelect a point on X-axis and for that number of no, get the number of yes for each probability distributionAverage the number of yes from all the folds and plot it
- 28. ROC CurvesSecond way:Get the probability distribution over different folds of dataSort the data in decreasing order of the probability of yes classSelect a point on X-axis and for that number of no, get the number of yes for each probability distributionPlot a ROC for each fold individually Average all the ROCs
- 29. ROC CurvesROC curves for two schemes
- 30. ROC CurvesIn the previous ROC curves:For a small, focused sample, use method AFor a large one, use method BIn between, choose between A and B with appropriate probabilities
- 31. Recall – precision curvesIn case of a search query:Recall = number of documents retrieved that are relevant / total number of documents that are relevantPrecision = number of documents retrieved that are relevant / total number of documents that are retrieved
- 32. A summary Different measures used to evaluate the false positive versus the false negative tradeoff
- 33. Cost curvesCost curves plot expected costs directlyExample for case with uniform costs (i.e. error):
- 34. Cost curvesExample with costs:
- 35. Cost curvesC[+|-] is the cost of predicting + when the instance is –C[-|+] is the cost of predicting - when the instance is +
- 36. Minimum Description Length PrincipleThe description length is defined as:Space required to describe a theory + space required to describe the theory’s mistakesTheory = Classifier and mistakes = errors on the training dataWe try to minimize the description lengthMDL theory is the one that compresses the data the most. I.e to compress a data set we generate a model and then store the model and its mistakesWe need to compute:Size of the modelSpace needed to encode the error
- 37. Minimum Description Length PrincipleThe 2nd one is easy. Just use informational loss functionFor 1st we need a method to encode the modelL[T] = “length” of the theoryL[E|T] = training set encoded wrt the theory
- 38. Minimum Description Length PrincipleMDL and clusteringDescription length of theory: bits needed to encode the clusters. E.g. cluster centersDescription length of data given theory: encode cluster membership and position relative to cluster. E.g. distance to cluster centersWorks if coding scheme uses less code space for small numbers than for large ones
- 39. Visit more self help tutorialsPick a tutorial of your choice and browse through it at your own pace.The tutorials section is free, self-guiding and will not involve any additional support.Visit us at www.dataminingtools.net