Assignment 0.

2: Comparing the
Functioning of Machine Learning
Algorithms
Objective:
The goal of this assignment is to understand and compare how different machine learning
algorithms function, their strengths, weaknesses, and best use cases.

Algorithms for Comparison:

1. Support Vector Machine (SVM)
2. Naïve Bayes
3. Random Forest
4. K-Nearest Neighbors (KNN)

Comparison Table:
K-Nearest
Support Vector
Criteria Naïve Bayes Random Forest Neighbors
Machine (SVM)
(KNN)
Uses Bayes’
Finds an optimal Classifies based
theorem to Uses multiple
hyperplane that on the majority
How does it calculate the decision trees and
best separates the class of the k-
work? probability of a aggregates their
data points of nearest
class given the output
different classes neighbors
data
Decision Probability,
Entropy, Gini Distance metrics
Core Math boundaries, Bayes’ theorem,
impurity, (Euclidean,
Concepts Support vectors, Conditional
Information gain Manhattan, etc.)
Kernel functions independence
Training: O(n^2) Training: O(n log
Speed (Big-O Training: O(n), Training: O(1),
to O(n^3), n), Prediction:
Notation) Prediction: O(1) Prediction: O(n)
Prediction: O(n) O(log n)
High (especially
Low (if Medium to High (if High (if k is too
Overfitting Risk with a non-linear
assumptions hold) too many trees) small)
kernel)
Hard to interpret, Easy to interpret
Hard to interpret
Interpretability especially with Easy to explain for small
(black-box)
non-linear kernels datasets
Best Use Cases High-dimensional Spam detection, Complex decision Small datasets,
 Pattern
Sentiment tasks, Financial
data, Text recognition,
analysis, Medical modeling, Fraud
classification Image
diagnosis detection
classification
Assumes feature Slow for large
Slow for large Prone to overfitting
independence, datasets,
datasets, Kernel if too many trees,
Limitations Can fail if features Sensitive to
selection is Computationally
are highly irrelevant
critical expensive
correlated features

Detailed Explanation of Each Algorithm

1. Support Vector Machine (SVM)

Working Principle:

 SVM aims to find the optimal decision boundary (hyperplane) that maximizes the
margin between different classes.
 It uses support vectors (critical data points) to define the boundary.
 Non-linearly separable data can be handled using kernel functions like RBF,
polynomial, or sigmoid.

Mathematical Concepts:

 Maximizing the margin between different classes.

 Kernel trick for non-linearly separable data.
 Lagrange multipliers for optimization.

Use Cases:

 Text classification (e.g., spam filtering).

 Image recognition.
 Bioinformatics (e.g., cancer classification).

2. Naïve Bayes

Working Principle:

 Based on Bayes’ theorem, which calculates the probability of a class given input
features.
 Assumes that all features are independent (hence "naïve").
 Fast and efficient for large datasets with categorical data.

Mathematical Concepts:

 Conditional probability: P(A|B) = P(B|A) * P(A) / P(B)

 Feature independence assumption simplifies computation.
Use Cases:

 Sentiment analysis.
 Spam detection.
 Medical diagnosis.

3. Random Forest

Working Principle:

 An ensemble learning method that constructs multiple decision trees and aggregates
their predictions.
 Uses bagging (bootstrap aggregation) to improve accuracy.
 Reduces variance and overfitting compared to a single decision tree.

Mathematical Concepts:

 Entropy & Gini impurity to measure node purity.

 Information gain to split nodes.
 Majority voting for classification.

Use Cases:

 Fraud detection.
 Credit scoring.
 Disease prediction.

4. K-Nearest Neighbors (KNN)

Working Principle:

 A lazy learning algorithm that memorizes training data.

 Classifies new points based on the majority vote of their k nearest neighbors.
 Distance metrics such as Euclidean distance determine proximity.

Mathematical Concepts:

 Distance metrics: Euclidean, Manhattan, Minkowski.

 Curse of dimensionality affects performance.

Use Cases:

 Recommender systems.
 Image recognition.
 Anomaly detection.

Decision Boundary Plots

Below is Python code to visualize decision boundaries for these algorithms:

import numpy as np

import matplotlib.pyplot as plt

from sklearn import datasets

from sklearn.model_selection import train_test_split

from sklearn.preprocessing import StandardScaler

from sklearn.svm import SVC

from sklearn.naive_bayes import GaussianNB

from sklearn.ensemble import RandomForestClassifier

from sklearn.neighbors import KNeighborsClassifier

from mlxtend.plotting import plot_decision_regions

# Load dataset

X, y = datasets.make_classification(n_features=2, n_classes=2, n_clusters_per_class=1,

n_redundant=0, random_state=42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()

X_train = scaler.fit_transform(X_train)

X_test = scaler.transform(X_test)

# Define classifiers

classifiers = {

"SVM": SVC(kernel='linear'),

"Naïve Bayes": GaussianNB(),

"Random Forest": RandomForestClassifier(n_estimators=10),

"KNN": KNeighborsClassifier(n_neighbors=3)
}

# Plot decision boundaries

fig, axes = plt.subplots(2, 2, figsize=(12, 10))

axes = axes.flatten()

for i, (name, clf) in enumerate(classifiers.items()):

clf.fit(X_train, y_train)

plot_decision_regions(X_test, y_test, clf=clf, ax=axes[i])

axes[i].set_title(name)

plt.tight_layout()

plt.show()

OUTPUT:

Naïve Bayes:
Random Forest:
KNN: