Scribd
Upload
18 views

Evaluation Machine Learning

This document evaluates the performance of Support Vector Machines (SVM) for regression and classification using a social network ads dataset, focusing on metrics like accuracy, precision, recall, F1-score, and AUC. It discusses the model's strengths and weaknesses, suggesting improvements through feature engineering, kernel optimization, and hyperparameter tuning. The conclusion emphasizes that while SVM can be effective, further adjustments may be necessary to address any imbalances in performance metrics.

Uploaded by

tsaiprakash16
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
18 views

Evaluation Machine Learning

This document evaluates the performance of Support Vector Machines (SVM) for regression and classification using a social network ads dataset, focusing on metrics like accuracy, precision, recall, F1-score, and AUC. It discusses the model's strengths and weaknesses, suggesting improvements through feature engineering, kernel optimization, and hyperparameter tuning. The conclusion emphasizes that while SVM can be effective, further adjustments may be necessary to address any imbalances in performance metrics.

Uploaded by

tsaiprakash16
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 5

Data Mining

Kamuluru Safiya
Student id : S02070828
Evaluation Machine Learning
Introduction
SVMs are strong supervised learning models that are utilised for regression and
classification applications. We assess an SVM model's performance in this
research using a dataset pertaining to social network ads. Assessing the model's
capacity to categorise users according to particular attributes and figuring out
how well it predicts outcomes are the main goals. Accuracy, precision, recall,
F1-score, and other conventional performance indicators are used in the
evaluation process, along with visualisation tools like the confusion matrix and
Receiver Operating Characteristic (ROC) curve. Furthermore, GridSearchCV is
used for hyperparameter tuning, which chooses the optimal set of parameters,
including the regularisation factor (C) and kernel type (linear, polynomial, RBF,
or sigmoid), to maximise the model's performance. We can ascertain the SVM
model's practicality and room for improvement by examining these metrics,
which provide us with information about the model's advantages and
disadvantages.

Analysis of Performance Metrics The following metrics are anticipated to form


part of the analysis, based on common classification evaluation techniques:
Accuracy: Indicates how accurate a prediction is overall. Calculates the
percentage of accurate predictions overall: (TP+TN)/(TP+TN+FP+FN) is the
accuracy. Evaluates the model's overall accuracy, good predictive performance
is shown by high accuracy, but if the dataset is unbalanced, this might be
deceptive.
Precision: Evaluates the proportion of accurately detected positive cases.
determines the proportion of expected positives that are actually positive:
Accuracy = TP/(TP+FP). In situations like spam detection or medical diagnosis,
high precision results in fewer false positives.
Recall (Sensitivity): Assesses the percentage of true positives that were
accurately identified. calculates the percentage of true positives that were
accurately identified: Recall is equal to TP/(TP+FN) evaluates the model's
ability to capture real positives. When it's expensive to overlook positive cases,
like in fraud detection, high recall is preferred.
F1-score: Offers a harmony between recall and precision. F1
score=(2*Precision*Recall)/(Precision+Recall) is a useful formula for
calculating the harmonic mean of precision and recall, which balances the two
variables in imbalanced datasets. Strong model performance is indicated by a
high F1-score, particularly in datasets that are unbalanced.
ROC curve and AUC: The ROC curve and AUC analyze the balance between
true positive and false positive rates. The Area Under Curve (AUC)
encapsulates the efficacy of the Receiver Operating Characteristic (ROC) curve,
showcasing True Positive Rate (TPR) versus False Positive Rate (FPR). AUC
near 1 indicates an excellent classifier. Approximately 0.5 AUC -> Random
guessing. AUC < 0.5 -> The classifier is poor.
Confusion Matrix: Indicates how many predictions were right and wrong for
each class and displays the prediction distribution, which includes False
Negatives (FN), True Negatives (TN), True Positives (TP), and False Positives
(FP).
Model Behaviour
High Precision, Low Recall: The model predicts positives conservatively,
missing actual positives (more false negatives) while producing fewer false
positives.
One example might be a fraud detection system that only identifies cases of
obvious fraud, leaving borderline cases unnoticed.
High Recall, Low Precision: The model makes a lot of optimistic predictions.
The majority of true positives are captured, although false positives are also
included.
An example would be a spam filter that flags a large number of emails as spam,
even though some of them are not.
Balanced Precision and memory (High F1-Score): This shows that the model
is effectively balancing precision and memory, making it appropriate for
situations like medical diagnosis where both false positives and false negatives
are significant.
Model Selection and Complexity Metrics: By penalising complexity,
Millard's Cp, AIC, and BIC aid in model selection.
Model generalisation is ensured by adjusted R^2, which takes into account the
number of predictors.
Views on the Performance of the Model:
Strengths: The model is functioning well if the accuracy, F1-score, and AUC-
ROC values are high.
Good class separation is indicated by a high AUC, which indicates that the
SVM is producing accurate predictions.
It is beneficial for real-world applications due to its balanced precision and
recall.
Weaknesses: It may be too cautious in positive forecasts if precision is
significantly higher than recall.
Recall may be overpredicting positives, resulting in false alarms, if it is
significantly greater than precision.
Overfitting risks: The model may be too complicated for the dataset if the
AIC/BIC values are high.
Views on Model Performance
SVM is a strong option for the dataset if it shows good performance (high
accuracy, balanced F1-score, AUC > 0.85, etc.). Outcomes can be enhanced
further with minor tweaks to feature engineering and hyperparameter tuning.
SVM may not be the best choice if the model displays negative traits, including
low recall, inadequate AUC, or elevated misclassification rates. Exploring
different models such as Random Forest, Neural Networks, or Gradient
Boosting might be beneficial.
Recommendations for Improvement
Feature Engineering: To increase class separability, improve feature selection.
Kernel optimisation: Try out various SVM kernels (polynomial, RBF, and
linear) to determine which one fits the data the best.
Hyperparameter tuning: To enhance model generalisation, modify gamma
(for the RBF kernel) and C (the regularisation parameter).
Managing Unbalanced Data: To increase recall and F1-score, use strategies
such class weighting, undersampling or oversampling.
Conclusion: The SVM model works well if metrics show good balance.
Hyperparameter tweaking (e.g., kernel selection, regularisation) or feature
engineering may be required if imbalances are present.

You might also like