21CS54: Artificial Intelligence and Machine Learning

Module 3
Topics: Basics of Learning theory, Similarity Based Learning, Regression Analysis
Textbook 2: Chapter 3 - 3.1 to 3.4, Chapter 4, chapter 5.1 to 5.4
Chapter 4
Similarity Based Learning
4.1 Similarity or Instance-based Learning

4.1.1 Difference between Instance-and Model-based Learning

Some examples of Instance-based Learning algorithms are:

a) KNN
b) Variants of KNN
c) Locally weighted regression
d) Learning vector quantization
e) Self-organizing maps
f) RBF networks

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

4.2 Nearest-Neighbor Learning

• A powerful classification algorithm used in pattern recognition.
• K nearest neighbors stores all available cases and classifies new cases based on a
similarity measure (e.g distance function).
• One of the top data mining algorithms used today.
• A non-parametric lazy learning algorithm (An Instance based Learning method).
• Used for both classification and regression problems.

In the above diagram 4.1, 2 classes of objects called C1 and C2. When given a test
instance T,the category of this test instance is determined by looking at the class ofk=3
nearest neighbors. Thus, the class of this test instance T ispredicted as C2.

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

Consider the student performance training, dataset of 8 data instances shown in Table 4.2
which describes the performance of individual students in a course and their CGPA
obtained in the previous semesters. The independent attributes are CGPA, Assessment and
Project. The target variable is ‘Result’ which is a discrete valued variable that takes two
values ‘Pass’ or ‘Fail. Based on the performance of a student, classify whether a student
will pass or fail in that course.

Sl No. CGPA Assessment Project Submitted Result

1 9.2 85 8 Pass
2 8 80 7 Pass
3 8.5 81 8 Pass
4 6 45 5 Fail
5 6.5 50 4 Fail
6 8.2 72 7 Pass
7 5.8 38 5 Fail
8 8.9 91 9 Pass

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

4.3 Weighted K-Nearest Neighbor Algorithm

The weighted KNN is an extension of k-NN.It chooses the neighbors by using the
weighted distance. In weighted kNN, the nearest k points are given a weight using a
function called as the kernel function. The intuition behind weighted kNN, is to give
more weight to the points which are nearby and less weight to the points which are
farther away.

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

4.4 Nearest Centroid Neighbor

The Nearest Centroids algorithm assumes that the centroids in the input feature space
are different for each target label. The training data is split into groups by class label,
then the centroid for each group of data is calculated. Each centroid is simply the mean
value of each of the input variables, so it is also called as Mean Difference classifier.
If there are two classes,then two centroids or points are calculated; three classes give
three centroids, and so on.

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

4.5 Locally Weighted Regression (LWR)

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

Where, г is called the bandwidth parameter and controls the rate at which wi reduces to
zero with distance from xi.

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

CHAPTER 5
REGRESSION ANALYSIS
5.1 Introduction to Regression
Regression analysis is a fundamental concept that consists of a set of machine learning
methods that predict a continuous outcome variable (y) based on the value of one or
multiple predictor variables (x). OR
Regression analysis is a statistical method to model the relationship between a
dependent (target) and independent (predictor) variables with one or more independent
variables.
Regression is a supervised learning technique which helps in finding the correlation
between variables.
It is mainly used for prediction, forecasting, time series modelling, and determining the
causal- effect relationship between variables.
Regression shows a line or curve that passes through all the datapoints on target-
predictor graph in such a way that the vertical distance between the datapoints and the
regression line is minimum." The distance between datapoints and line tells whether a
model has captured a strong relationship or not.
• Function of regression analysis is given by: Y=f(x)
Here, y is called dependent variable and x is called independent variable.
Applications of Regression Analysis
1) Sales of a goods or services
2) Value of bonds in portfolio management
3) Premium on insurance companies
4) Yield of crop in agriculture
5) Prices of real estate
5.2 INTRODUCTION TO LINEARITY, CORRELATION AND CAUSATION
A correlation is the statistical summary of the relationship between two sets of
variables. It is a core part of data exploratory analysis, and is a critical aspect of
numerous advanced machine learning techniques. Correlation between two variables
can be found using a scatter plot.
There are different types of correlation:

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

• Positive Correlation: Two variables are said to be positively correlated when their
values move in the same direction. For example, in the image below, as the value
for X increases, sodoes the value for Y at a constant rate.
• Negative Correlation: Finally, variables X and Y will be negatively correlated
when their values change in opposite directions, so here as the value for X increases,
the value for Y decreases at a constant rate.
• Neutral Correlation: No relationship in the change of variables X and Y. In this
case, the values are completely random and do not show any sign of correlation,
as shown in the following image:

Causation
Causation is about relationship between two variables as x causes y. This is called x
implies b.Regression is different from causation. Causation indicates that one event is
the result of the occurrence of the other event; i.e. there is a causal relationship between
the two events.
Linear and Non-Linear Relationships
• The relationship between input features (variables) and the output (target) variable is
fundamental. These concepts have significant implications for the choice of algorithms,
model complexity, and predictive performance.
• Linear relationship creates a straight line when plotted on a graph, a Non-Linear
relationship does not create a straight line but instead creates a curve.
• Example:
Linear-the relationship between the hours spent studying and the grades obtained in a
class.
Non-Linear- GPS Signal

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

• Linearity:
Linear Relationship: A linear relationship between variables means that a change in
one variable is associated with a proportional change in another variable.
Mathematically, it can be represented as y = a * x + b, where y is the output, x is the
input, and a and b are constants.
Linear Models: Goal is to find the best-fitting line (plane in higher dimensions) to the
data points. Linear models are interpretable and work well when the relationship
between variablesis close to being linear.
Limitations: Linear models may perform poorly when the relationship between
variables is non-linear. In such cases, they may underfit the data, meaning they are too
simple to capture the underlying patterns.
• Non-Linearity:
Non-Linear Relationship: A non-linear relationship implies that the change in one
variable isnot proportional to the change in another variable. Non-linear relationships
can take various forms, such as quadratic, exponential, logarithmic, or arbitrary shapes.
Non-Linear Models: Machine learning models like decision trees, random forests,
support vector machines with non-linear kernels, and neural networks can capture non-
linear relationships. These models are more flexible and can fit complex data patterns.
Benefits: Non-linear models can perform well when the underlying relationships in the
data are complex or when interactions between variables are non-linear. They have the
capacity to capture intricate patterns.

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

Types of Regression

Linear Regression:
Single Independent Variable: Linear regression, also known as simple linear
regression, is used when there is a single independent variable (predictor) and one
dependent variable (target).
Purpose: Linear regression is used to establish a linear relationship between two
variables and make predictions based on this relationship. It's suitable for simple
scenarios where there's onlyone predictor.
Multiple Regression:
Multiple Independent Variables: Multiple regression, as the name suggests, is used
when there are two or more independent variables (predictors) and one dependent
variable (target).
Purpose: Multiple regression allows you to model the relationship between the
dependent variable and multiple predictors simultaneously. It is used when there are
multiple factors that may influence the target variable, and you want to understand their
combined effect and makepredictions based on all these factors.
Polynomial Regression:
Polynomial regression is an extension of multiple regression used when the relationship
between the independent and dependent variables is non-linear.
Logistic Regression:
Logistic regression is used when the dependent variable is binary (0 or 1). It models the
probability of the dependent variable belonging to a particular class.
Lasso Regression (L1 Regularization):
Lasso regression is used for feature selection and regularization. It penalizes the
absolutevalues of the coefficients, which encourages sparsity in the model.

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

Ridge Regression (L2 Regularization):

Ridge regression is used for regularization to prevent overfitting in multiple regression.
Itpenalizes the square of the coefficients.
Limitations of Regression

5.3 INTRODUCTION TO LINEAR REGRESSION

Linear regression model can be created by fitting a line among the scattered data points.
The line is of the form:
y= a0+a1*x+e

Ordinary Least Square Approach

The ordinary least squares (OLS) algorithm is a method for estimating the parameters
of a linear regression model. Aim: To find the values of the linear regression model's
parameters (i.e., the coefficients) that minimize the sum of the squared residuals.
In mathematical terms, this can be written as: Minimize ∑(yi – ŷi)^2
where yi is the actual value, ŷi is the predicted value.

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

A linear regression model used for determining the value of the response variable, ŷ,
can berepresented as the following equation.
y = b0 + b1x1 + b2x2 + … + bnxn + e
where: y - is the dependent variable, b0 is the intercept, e isthe error term
b1, b2, …, bn are the coefficients of the independentvariables x1, x2, …, xn
The coefficients b1, b2, …, bn can also be called the coefficientsof determination. The
goal of the OLS method can be used to estimate the unknown parameters (b1, b2, …,
bn) by minimizingthe sum of squared residuals (RSS). The sum of squared residualsis
also termed the sum of squared error (SSE).
This method is also known as the least-squares method for regression or linear
regression.Mathematically the line of equations for points are:
y1=(a0+a1x1)+e1
y2=(a0+a1x2)+e2 ……. yn=(a0+a1xn)+en.
In general ei=yi - (a0+a1x1)

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

Linear Regression Example

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

Linear Regression in Matrix Form

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

5.4 VALIDATION OF REGRESSION METHODS

The regression should be evaluated using some metrics for checking the correctness.
Thefollowing metrics are used to validate the results of regression.

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

Coefficient of Determination

The coefficient of determination (R² or r-squared) is a statistical measure in a regression

model that determines the proportion of variance in the dependent variable that can be
explained by the independent variable.
The sum of the squares of the differences between the y-value of the data pair and the
average of y is called total variation. Thus, the following variation can be defined as,
The explained variation is given by, =∑( Ŷi – mean(Yi))2 The unexplained variation is
given by, =∑( Yi - Ŷi )2
Thus, the total variation is equal to the explained variation and the unexplained
variation. The coefficient of determination r2 is the ratio of the explained and
unexplained variations.

Standard Error Estimate:

Standard Error Estimate is another useful measure of regression. It is standard
deviation of the observed values to the predicted values. This is given as:

Dept of ISE, RNSIT VS, Dr SNK, AG

 21CS54: Artificial Intelligence and Machine Learning

Dept of ISE, RNSIT VS, Dr SNK, AG