Linear Regression

Prerequisite:
Statistics- Mean, Median, Mode, Variance and Standard deviation, Correlation, and
Covariance.
Exploratory Data Analysis- Data distribution, Scatter plot, correlation matrix, Heat map.

Objectives:

 Understand what is Linear Regression and the motivation behind Linear Regression
 What is the best fit line and the concept of residual(s) of Linear Regression
 Least Square method to find Best Fit line of Regression.
 Gradient Descent method to find Best Fit line of Regression

Linear Regression
Linear regression
[email protected] is a way to identify a relationship between two or more variables. We
21YORICED7
use this relationship to predict the values for one variable for a given set of value(s) of the
other variable(s). The variable, which is used in prediction is termed as
independent/explanatory/regressor variable where the predicted variable is termed as
dependent/target/response/regressand variable. Linear regression assumes that the
dependent variable is linearly related to the estimated parameter(s).
𝑦 = 𝑐 + 𝑚𝑥
In machine learning and regression literature the above equation is used in the form:
𝑦 = 𝑤0 + 𝑤1 𝑥

Where w0 is intercept on y-axis, w1 is slope of line, x is an explanatory variable and y is

the response variable.
Motivational Examples

1. Let us see a use case of linear regression for the prediction of house prices. For each house
we have been given the complete information of the plot size area and the price at which
the house was sold. Can we use this information to predict the selling price of a house for

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 1
Sharing or publishing the contents in part or full is liable for legal action.
 a given plot size area? The problem can be modelled as a linear regression with plot_
size(x) as an explanatory variable and house price(y) as the response variable.
𝐻𝑜𝑢𝑠𝑒𝑃𝑟𝑖𝑐𝑒(𝑦) = 𝑤0 + 𝑤1 𝑃𝑙𝑜𝑡𝑆𝑖𝑧𝑒
2. Consider a scenario where we have been given medical data about some patients. The data
contains the information of the blood pressure for a patient along with his/her age. Can we
use this information to predict the blood pressure level of patient for a given age? This
problem is modelled as a linear regression problem with age as an explanatory variable and
blood pressure as the response variable.
𝐵𝑙𝑜𝑜𝑑𝑃𝑟𝑒𝑠𝑠𝑢𝑟𝑒(𝑦) = 𝑤0 + 𝑤1 𝐴𝑔𝑒

The above two examples are examples of a Simple Linear Regression. A regression which
has one independent variable. In such cases, we are using one response (x) variable to
predict the target variable (y).

3. Next, consider a problem where we need to predict the price of a used car. The selling price
of a used
[email protected] car depends on many attributes; some of them may be mileage (km/litre), model
21YORICED7
(Maruti, Hyundai, Honda, Toyota, Tata), segment (Small, Medium, Luxury). In this
scenario the selling price is the response or the target variable which depends on mileage,
model and segment (explanatory variables). This problem can be modelled as a Multiple
Linear Regression as there is more than one explanatory variable involved in the prediction
of the target variable.
𝑆𝑒𝑙𝑙𝑖𝑛𝑔𝑃𝑟𝑖𝑐𝑒(𝑦) = 𝑤0 + 𝑤1 𝑀𝑖𝑙𝑒𝑎𝑔𝑒 + 𝑤2 𝑀𝑜𝑑𝑒𝑙 + 𝑤3 𝑆𝑒𝑔𝑚𝑒𝑛𝑡

In real scenarios, we rarely have one explanatory variable, so we use multiple linear
regression rather than simple linear regression. However, here we take an example of
simple linear regression to understand the fundamentals of regression.

Example: Consider a toy example where we are interested to find the effect of studying
hours per day with grades in an examination and predict the grades of a student for a given
number of study hours. We have a sample data for about six students for their grades and
the total study hours per day.

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 2
Sharing or publishing the contents in part or full is liable for legal action.
 From the given data we get an idea that study hours per day and grades have a positive
relationship. So one can say that if a student spends more hours studying per day he is
likely to get good grades in his/her examination.
The scatter plot of given data is shown in Figure-1. Scatter plot is a useful tool to judge the
strength and nature of relationship between two variables. A valuable measure to quantify
the linear relationship between two variables is the correlation coefficient. The correlation
coefficient value ranges between -1 to 1 to indicate the strength of the relationship. -1
(minus one) indicates the strong negative relation where an increase in one variable results
in a decrease of other variable. +1 (plus one) indicates a strong positive relation which tells
us that an increase in a variable result in the increase of the other variable. 0 (zero) shows
no correlation between the two variables and therefore no linear relation is present between
the two variables.
From the scatter plot shown in Figure-1, we get some intuition that there is a positive
correlation between the studying hours per day and the grades in exam.

[email protected] Table-1
21YORICED7

Study Hours Grades

per day

6 7
5 5
4 2
7 9
8 6 Figure-1: Scatter Graph
2 3

To fit a linear regression model to the given data we can draw multiple lines which goes
through our data points and out of them one will be our best fit line. Let the equation of the
linear regression model be given by:
𝑦(𝐺𝑟𝑎𝑑𝑒𝑠) = 𝑤0 + 𝑤1 𝑋(𝑆𝑡𝑢𝑑𝑦 𝐻𝑜𝑢𝑟𝑠 𝑝𝑒𝑟 𝑑𝑎𝑦)

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 3
Sharing or publishing the contents in part or full is liable for legal action.
 Note: Here, we are trying to model the ‘Grade' based on 'Study Hours per day' and thus we have
appropriately chosen the ‘x' and 'y' variables
Now we need to define the criteria for our best fit line.
Any line that we might come up with has to have some fixed intercept w0 and a slop w1. This line
may include some data points on it but cannot cover all of them. In our example we have given
with six data points let us label these points by (x1, y1), (x2, y2), (x3, y3), (x4,y4), (x5, y5) and
(x6, y6), with values (6, 7), (5, 5), (4, 2), (7, 9), (8, 7) and (2, 3). For any given point 𝑋𝑖 the
prediction of 𝑦ℎ𝑎𝑡𝑖 is given by:
𝑦ℎ𝑎𝑡𝑖 = 𝑤0 + 𝑤1 𝑥𝑖
Unless the line passes through (𝑥𝑖 , 𝑦𝑖 ) the value of 𝑦ℎ𝑎𝑡𝑖 differs from the observed value
of 𝑦𝑖 . The difference between the two values is denotes as error or residual of regression.
𝑒𝑖 = 𝑦𝑖 − 𝑦ℎ𝑎𝑡𝑖
The best line is the line which minimizes the sum of the squared error:
𝑒𝑖2 = (𝑦1 − 𝑦ℎ𝑎𝑡1 ) 2 + (𝑦2 − 𝑦ℎ𝑎𝑡2 ) 2 + … … … + (𝑦𝑛 − 𝑦ℎ𝑎𝑡𝑛 ) 2

Following graphs
[email protected] illustrate the process to find the best line of regression.
21YORICED7

(a) (b)

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 4
Sharing or publishing the contents in part or full is liable for legal action.
 (c) (d)

Figure-2 (a) Fit a line of equation y =x. (b) Fit a line of equation y = 1.667 + 0.667x
(c) Fit a line of equation 0.457 + 0.914x (d) Combine all three lines and choose the Best Fit
Lines with minimum Residual.

[email protected]
21YORICED7

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 5
Sharing or publishing the contents in part or full is liable for legal action.
 Methods to Find Best Fit Line - We can use two different methods to find best fit line of linear
regression.
1. Principle of Least Squares.
2. Gradient Descent.

Least Square- Let the equation of regression line of y on x is:

𝑦 = 𝑤0 + 𝑤1 𝑥
According to the least square principle the equations to estimate the values of w0 and w1
are:
𝑛 𝑛
∑ 𝑦𝑖 = 𝑛𝑤0 + 𝑤1 ∑ 𝑥𝑖 … … … (1)
𝑖=1 𝑖=1
𝑛 𝑛 𝑛
∑ 𝑋𝑖 𝑦𝑖 = 𝑤0 ∑ 𝑥𝑖 + 𝑤1 ∑ 𝑥𝑖2 … … … . (2)
𝑖=1 𝑖=1 𝑖=1

These two equations are called the normal equations of a linear regression. These equations are
n
obtained by the partial differentiation of the function (f) =∑i=1[yi(actual) – yi(predicted)]2
[email protected]
21YORICED7with respect to w0 and w1 and equating it to 0 to minimise the function (f).

Dividing equation (1) by n we get,

1 𝑛 𝑤1 𝑛
∑ 𝑦𝑖 = 𝑤0 + ∑ 𝑥𝑖
𝑛 𝑖=1 𝑛 𝑖=1

𝑦̅𝑖 = 𝑤0 + 𝑤1 𝑥̅𝑖 … … … . (3)

Thus we can say the line of regression always passes through the points (𝑥̅ , 𝑦̅)
Now we need to estimate the values for w0 and w1,
We know,
1 1
𝑐𝑜𝑣(𝑥, 𝑦) = ∑ 𝑥𝑖 𝑦𝑖 − 𝑥̅ 𝑦̅ ⟹ ∑ 𝑥𝑖 𝑦𝑖 ⟹ 𝑐𝑜𝑣(𝑥, 𝑦) + 𝑥̅ 𝑦̅ … … … . . (4)
𝑛 𝑛
𝑖 𝑖

Also,
1 𝑛 1 𝑛
𝑣𝑎𝑟(𝑥) = ∑ 𝑥𝑖2 − 𝑥̅ 2 ⟹ ∑ 𝑥𝑖2 = 𝑣𝑎𝑟(𝑥) + 𝑥̅ 2 … … … . . (5)
𝑛 𝑖=1 𝑛 𝑖=1

Dividing equation (2) by n and using equation (4) and (5),

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 6
Sharing or publishing the contents in part or full is liable for legal action.
 𝑐𝑜𝑣(𝑥, 𝑦) + 𝑥̅ 𝑦̅ = 𝑤0 𝑥̅ + 𝑤1 (𝑣𝑎𝑟(𝑥) + 𝑥̅ 2 ) … … . . (6)
By solving equation (3) and (6) we get,
𝑐𝑜𝑣(𝑥, 𝑦)
𝑤1 = … … … (7)
𝑣𝑎𝑟(𝑥)

and

𝑤0 = 𝑦̅ − 𝑤1 𝑥̅
The straight line defined by 𝑦 = 𝑤0 + 𝑤1 𝑥 satisfies the residual (least squares)
condition error = E{( y -(w0 + w1x))2} is minimum for variations in a and b, is called the
line of regression of y on x. Let us try these equations to estimate best fit line on our data
given in Table-1.
To estimate w0 and w1, we need to find covariance between x and y [Cov(x, y)], variance
of x [var(x)] and mean of x and y variables (x, and y) .For given data we get,
6+5+4+7+8+2
𝑥̅ = = 5.333
6
7+5+2+9+6+3
[email protected]
𝑦̅ = = 5.333
6
21YORICED7
𝑐𝑜𝑣(𝑥, 𝑦) = 3.5555
𝑣𝑎𝑟(𝑥) = 3.8889
when we substitute these values in equation (7) and (8) we get,
𝑤0 = 0.4571 𝑎𝑛𝑑 𝑤1 = 0.9143
which are exactly the same as shown in Figure-2(c) for the line y = 0.457 + 0.914x, which
gives the minimum residual among all the lines.
Performance metric for least square regression- Performance metrics are the way to
quantify and compare the efficiency of any machine learning model. Least square
regression uses R2( R-squared) and Radj 2 (Adjusted R-Square) metrics to measure the
performance of regression model. Radj2 (Adjusted R-Square) is used with multiple linear
regression. Both of these metrics denotes the power of explain ability of selected
independent variable(s) to the variation of response variable. The equations of R2( R-
squared) and Radj2(Adjusted R-Square) are given by:

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 7
Sharing or publishing the contents in part or full is liable for legal action.
 1 𝑛
∑𝑖=1(𝑦𝑖 − 𝑦ℎ𝑎𝑡𝑖 )2
2
𝑅 =1− 𝑛
1 𝑛 2
∑
𝑛 𝑖=1(𝑦𝑖 − 𝑦̅)
The numerator term gives the average of squares of residuals and denominator shows the
variance in y(response) value. A small value for R2 or higher mean residual error denote
poor model.
(1 − 𝑅 2 )(𝑛 − 1)
𝑅𝑎𝑑𝑗 2 = 1 −
𝑛−𝑘−1

Where n is the total number of observations in data and k is the number of explanatory
variables. Radj 2 (Adjusted R-Square) is slight improvement over R2( R-squared) by adding
an additional term to it. The problem with R2 (R-squared) is that, R2( R-squared) increases
with increase in number of terms in the model irrespective of whether the added terms
significantly contribute in prediction/explaining the dependent variable or not. On the
contrary, the value of Radj 2 (Adjusted R-Square) is only affected by if significant terms are
added to the model. The relation between R2( R-squared) and Radj 2 (Adjusted R-Square)
[email protected]
21YORICED7 is:
2
𝑅𝑎𝑑𝑗 ≤ 𝑅2

Gradient Descent- Let the equation of regression line of y on x is:

𝑦 = 𝑤0 + 𝑤1 𝑥
This straight line tries to approximate the relationship between x and y for a given set of
data. By varying the values of w0 and w1, we can find the best fit line. With the above
discussion we know that the best fit line is one which minimizes the total error in
prediction. Gradient descent method defines a cost function of parameter w0 and w1 and
uses a systematic approach to optimize the values of parameters to get minimum cost
function. Let us dive a bit deep into the mathematics of algorithm.
Let the model be defined as:
𝑦 = 𝑤0 + 𝑤1 𝑥
Now defining the cost function of gradient descent as Mean Squared Error of prediction:
𝑛 𝑛
1 1
𝑐𝑜𝑠𝑡 (𝑤0 , 𝑤1 ) = ∑(𝑦𝑖 − 𝑦ℎ𝑎𝑡𝑖 )2 = ∑(𝑦𝑖 − 𝑤0 − 𝑤1 𝑥𝑖 )2
𝑛 𝑛
𝑖=1 𝑖=1

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 8
Sharing or publishing the contents in part or full is liable for legal action.
 Note: The cost function is sometimes also referred to as the Loss Function.

The cost function includes two parameters w0 and w1, which control the value of cost
function. As we know the derivatives give us the rate of change in one variable with respect
to others, so we can use partial derivatives to find the impact of individual parameter over
the cost function.
The principle of gradient descent is that we always make progress in the direction where
the partial derivatives of w0 and w1 are steepest. If the derivatives of parameters are
approaching zero or becoming very less, this points to the situation of either a maxima or
minima on the surface of the cost function. The process of gradient descent is started with
a random initialization of w0 and w1. Every iteration of gradient descent improves in the
direction of optimal values for w0 and w1 parameters for which the cost function will have
a minimum value. Following figure illustrate the process of optimization.

[email protected]
21YORICED7

Figure 3: Gradient Descent Iteration

Gradient descent works in following steps:
1. Random initialization of parameters.
2. Calculate the partial derivatives of the cost function with respect to each
parameter(gradients).
3. Update the parameters in the opposite direction of gradients.

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 9
Sharing or publishing the contents in part or full is liable for legal action.
 4. Repeat step 3 and 4 till maximum iteration reached or minimum cost function value
achieved.
Partial derivatives:
We have,
𝑛
1
𝑒𝑟𝑟𝑜𝑟 = 𝑐𝑜𝑠𝑡 (𝑤0 , 𝑤1 ) = ∑(𝑦𝑖 − 𝑤0 − 𝑤1 𝑥𝑖 )2
𝑛
𝑖=1

Partial derivative w. r. t. w0 and w1:

𝑛 𝑛
𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 ) 1 −2
= ∑(𝑦𝑖 − 𝑤0 − 𝑤1 𝑥𝑖 )(−2) = ∑ 𝑒𝑟𝑟𝑜𝑟𝑖
𝜕𝑤0 𝑛 𝑛
𝑖=1 𝑖=1
𝑛 𝑛
𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 ) 1 −2
= ∑(𝑦𝑖 − 𝑤0 − 𝑤1 𝑥𝑖 )(−2𝑥𝑖 ) = ∑ 𝑒𝑟𝑟𝑜𝑟𝑖 (𝑥𝑖 )
𝜕𝑤1 𝑛 𝑛
𝑖=1 𝑖=1

[email protected]
21YORICED7 Parameter updates:
𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 )
𝑤0 = 𝑤0 − 𝑙𝑟𝑎𝑡𝑒
𝜕𝑤0
𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 )
𝑤1 = 𝑤1 − 𝑙𝑟𝑎𝑡𝑒
𝜕𝑤1

lrate is the learning rate which controls the step size of parameter update.
Let’s run it on our example:
X: 6 5 4 7 8 2
y: 7 5 2 9 6 3

Let’s initialize both coefficient 𝑤0 𝑎𝑛𝑑 𝑤1 with 0.0,

𝑤0 = 0.0
𝑤1 = 0.0
Iteration #1:
𝑦ℎ𝑎𝑡𝑖 = 0.0 + 0.0 𝑥𝑖

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 10
Sharing or publishing the contents in part or full is liable for legal action.
 Calculate gradients:
𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 ) −2
= (7 + 5 + 2 + 9 + 6 + 3) = −10.6667
𝜕𝑤0 6

𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 ) −2
= (7 ∗ 6 + 5 ∗ 5 + 2 ∗ 4 + 9 ∗ 7 + 6 ∗ 8 + 3 ∗ 2) = −64
𝜕𝑤1 6

Update parameters: (lrate = 0.01)

𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 )
𝑤0 = 𝑤0 − 𝑙𝑟𝑎𝑡𝑒 = 0.0 − 0.01 (−10.6667) = 0.1066
𝜕𝑤0

𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 )
𝑤1 = 𝑤1 − 𝑙𝑟𝑎𝑡𝑒 = 0.0 − 0.01 (−64) = 0.64
𝜕𝑤1

Iteration #2:
𝑦ℎ𝑎𝑡𝑖 = 0.1067 + 0.64 𝑥𝑖

Calculate gradients:

[email protected]
21YORICED7 𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 ) −2
= (3.0533 + 1.6933 − 0.6667 + 4.4133 + 0.7733 + 1.6133) = −3.6266
𝜕𝑤0 6

𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 ) −2
= (3.0533 ∗ 6 + 1.6933 ∗ 5 − 0.6667 ∗ 4 + 4.4133 ∗ 7 + 0.7733 ∗ 8
𝜕𝑤1 6
+ 1.6133 ∗ 2)
= −21.475

Update parameters: ( lrate = 0.01)

𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 )
𝑤0 = 𝑤0 − 𝑙𝑟𝑎𝑡𝑒 = 0.1067 − 0.01 (−3.6266) = 0.14296
𝜕𝑤0

𝜕𝑐𝑜𝑠𝑡(𝑤0 , 𝑤1 )
𝑤1 = 𝑤1 − 𝑙𝑟𝑎𝑡𝑒 = 0.64 − 0.01 (−21.475) = 0.8547
𝜕𝑤1

Similarly, the number of iteration for gradient descent are performed till the minimum
value of cost function of error is achieved or some finite iterations are reached.

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 11
Sharing or publishing the contents in part or full is liable for legal action.
 Multiple Linear Regression: Till we have discussed the case of simple linear regression
with just one explanatory variable. But in real scenarios the target variable might depend
on multiple explanatory variables which need to be cater during the development of linear
regression model. The model is expressed as:
𝑦 = 𝑤0 + 𝑤1 𝑥1 + 𝑤2 𝑥2 + 𝑤3 𝑥3 … … … … . . +𝑤𝑛 𝑥𝑛
Where x1, x2, x3…….xn are the explanatory variables and y is target variable.
Evaluation of Linear regression model- Evaluation helps to judge the performance of
any machine learning model that would provide best results to our test data. Fundamentally
three types of evaluation metrics are used to evaluate linear regression model.
- R2 measure (discussed with least square method)
- Mean Absolute Error (MAE)
- Root Mean Square Error (RMSE)
Mean Absolute Error(MAE)- Mean Absolute Error is the average of the difference
between actual and predicted value of target variable.
𝑛
[email protected] 1
21YORICED7 𝑀𝐴𝐸 = ∑|𝑦𝑖 − 𝑦ℎ𝑎𝑡𝑖 |
𝑛
𝑖=1

Root Mean Square Error(RMSE)- defined as:

𝑛
1
𝑅𝑀𝑆𝐸 = √ ∑(𝑦𝑖 − 𝑦ℎ𝑎𝑡𝑖 )2
𝑛
𝑖=1

Pros and cons of Linear Regression:

Pros- Linear regression models are very simple and easy to implement. These models are
said to be most interpretable.
Cons- Linear regression models are largely affected by the presence of outlier in training
data.

********

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited. 12
Sharing or publishing the contents in part or full is liable for legal action.