Linear Regression

Student : Mohammed Abu Musameh

Supervisor : Eng. Akram Abu Garad
Abstract
 Father of Regression Analysis Carl F. Gauss (1777-1855).
 contributions to physics, Mathematics & astronomy.
 The term “Regression” was first used in 1877 by Francis Galton.
 Regression is a statistical method used in finance, investing, and other disciplines that
attempts to determine the strength and character of the relationship between one
dependent variable (usually denoted by Y) and a series of other variables (known as
independent variables).
 Regression helps investment and financial managers to value assets and understand the
relationships between variables, such as commodity prices and the stocks of businesses
dealing in those commodities.
Introduction
 Linear regression is used for predictive analysis. Linear regression is a linear
approach for modeling the relationship between the criterion or the scalar response
and the multiple predictors or explanatory variables. Linear regression focuses on the
conditional probability distribution of the response given the values of the predictors.
For linear regression, there is a danger of overfitting. The formula for linear
regression is: Y = bX + A .
Outline
 Regression Analysis
 Regression vs Classification
 When Regression is chosen?
 Types of Regression
 Linear Regression
 Gradient Descent in Linear Regression
 Example for Linear Regression working
 Application of Linear Regression
 Conclusion
Regression Analysis
 Is a statistical process for estimating the relationships between the dependent
variables or criterion variables and one or more independent variables or predictors.
Regression analysis explains the changes in criterions in relation to changes in select
predictors. The conditional expectation of the criterions based on predictors where
the average value of the dependent variables is given when the independent
variables are changed. Three major uses for regression analysis are determining the
strength of predictors, forecasting an effect, and trend forecasting.
Regression vs Classification
Classification and Regression are two major prediction problems which are usually dealt with Data
mining and machine learning.

◦ Regression is the process of finding a model or function for distinguishing the data into continuous real values
instead of using classes or discrete values. It can also identify the distribution movement depending on the
historical data. Because a regression predictive model predicts a quantity, therefore, the skill of the model
must be reported as an error in those predictions.

◦ Classification is the process of finding or discovering a model or function which helps in separating the data
into multiple categorical classes i.e. discrete values. In classification, data is categorized under different labels
according to some parameters given in input and then the labels are predicted for the data.
Regression vs Classification (con’t)
PARAMENTER REGRESSION CLASSIFICATION
Mapping Function is used for mapping Mapping Function is used for mapping
Basic
of values to continuous output. of values to predefined classes.
Involves prediction of Continuous values Discrete values
Nature of the predicted data Ordered Unordered
by measurement of root mean square
Method of calculation by measuring accuracy
error

Regression tree (Random forest),

Example Algorithms Linear regression, etc. Decision tree, logistic regression, etc.
When Regression is chosen?
 A regression problem is when the output variable is a real or continuous value, such
as “salary” or “weight”. Many different models can be used, the simplest is the linear
regression. It tries to fit data with the best hyperplane which goes through the points.
Types of Regression
 Linear regression
 Logistic regression
 Polynomial regression
 Stepwise regression
 Ridge regression
 Lasso regression
 ElasticNet regression
Linear Regression
 Linear Regression is a machine learning algorithm based on supervised learning.
It performs a regression task. Regression models a target prediction value based
on independent variables. It is mostly used for finding out the relationship between
variables and forecasting. Different regression models differ based on – the kind of
relationship between dependent and independent variables, they are considering
and the number of independent variables being used.
Linear Regression (con’t)
 Linear regression performs the task to predict a dependent variable value (y) based
on a given independent variable (x). So, this regression technique finds out a linear
relationship between x (input) and y(output).
 In the figure, X (input) is the work experience and Y (output) is the salary of a
person. The regression line is the best fit line for our model.
Linear Regression (con’t)
 Hypothesis function for Linear Regression :

While training the model we are given :

x: input training data (univariate – one input variable(parameter))
y: labels to data (supervised learning)
When training the model – it fits the best line to predict the value of y for a given value of x.
The model gets the best regression fit line by finding the best θ1 and θ2 values.
θ1: intercept
θ2: coefficient of x

 Once we find the best θ1 and θ2 values, we get the best fit line. So when we are finally using our
model for prediction, it will predict the value of y for the input value of x.
Linear Regression (con’t)
 How to update θ1 and θ2 values to get the best fit line ?
Cost Function (J):
By achieving the best-fit regression line, the model aims to predict y value such that the error
difference between predicted value and true value is minimum. So, it is very important to
update the θ1 and θ2 values, to reach the best value that minimize the error between predicted y
value (pred) and true y value (y).

 Cost function(J) of Linear Regression is the Root Mean Squared Error (RMSE) between predicted y value
(pred) and true y value (y).
Gradient Descent in Linear Regression

 To update θ1 and θ2 values in order to reduce Cost function (minimizing RMSE value) and
achieving the best fit line the model uses Gradient Descent. The idea is to start with random
θ1 and θ2 values and then iteratively updating the values, reaching minimum cost.

 In linear regression, the model targets to get the best-fit regression line to predict the
value of y based on the given input value (x). While training the model, the model
calculates the cost function which measures the Root Mean Squared error between the
predicted value (pred) and true value (y). The model targets to minimize the cost function.
Gradient Descent in Linear Regression
(con’t)

 To minimize the cost function, the model needs to have the best value of θ1 and θ2.
Initially model selects θ1 and θ2 values randomly and then iteratively update these value in
order to minimize the cost function until it reaches the minimum. By the time model
achieves the minimum cost function, it will have the best θ1 and θ2 values. Using these
finally updated values of θ1 and θ2 in the hypothesis equation of linear equation, model
predicts the value of x in the best manner it can.
Gradient Descent in Linear Regression
(con’t)
 How θ1 and θ2 values get updated ?
o Linear Regression Cost Function:
Gradient Descent in Linear Regression
(con’t)
 How θ1 and θ2 values get updated ?
o Gradient Descent Algorithm For Linear Regression
Gradient Descent in Linear Regression
(con’t)
 How θ1 and θ2 values get updated ?
o Gradient Descent Algorithm For Linear
Regression

θj : Weights of the hypothesis.

hθ(Xi) : predicted y value for i input. 
j : Feature index number (can be 0, 1, 2,
......, n).
α : Learning Rate of Gradient Descent.
Gradient Descent in Linear Regression
(con’t)
 We graph cost function as a function of parameter estimates i.e. parameter range of
our hypothesis function and the cost resulting from selecting a particular set of
parameters. We move downward towards pits in the graph in next slide, to find the
minimum value. Way to do this is taking derivative of cost function as explained in
the above figure. Gradient Descent step downs the cost function in the direction of
the steepest descent. Size of each step is determined by parameter α known as
Learning Rate.
Gradient Descent in Linear Regression
(con’t)
 In the Gradient Descent algorithm, one can understand two points :

1. If slope is +ve : θj = θj – (+ve value). Hence value of θj decreases.

Gradient Descent in Linear Regression
(con’t)
 In the Gradient Descent algorithm, one can understand two points :

2. If slope is -ve : θj = θj – (-ve value). Hence value of θj increases.

Gradient Descent in Linear Regression
(con’t)
 The choice of correct learning rate is very important as it ensures that Gradient
Descent converges in a reasonable time.

1. If we choose α to be very large, Gradient Descent can overshoot the minimum. It may
fail to converge or even diverge.
Gradient Descent in Linear Regression
(con’t)
 The choice of correct learning rate is very important as it ensures that Gradient
Descent converges in a reasonable time.

2. If we choose α to be very small, Gradient Descent will take small steps to reach local
minima and will take a longer time to reach minima.

Note : For linear regression Cost Function graph is always convex shaped.
Example for Linear Regression working
 Suppose we are given a dataset
o Given is a Work vs Experience dataset of
a company and the task is to predict the
salary of a employee based on his / her
work experience.
o Our aims to explain how in reality Linear
regression mathematically works when
we use a pre-defined function to perform
prediction task.
o Let us explore how the stuff works when
Linear Regression algorithm gets trained.
Example for Linear Regression working
(con’t)
 Iteration 1 : In the start , θ0 and θ1 values are randomly chosen.
Let us suppose, θ0 = 0 and θ1 = 0 .
o Predicted values after iteration 1 with Linear regression hypothesis.
Example for Linear Regression working
(con’t)
o Cost Function – Error
Example for Linear Regression working
(con’t)
o Gradient Descent – Updating θ0 value
• Here, j = 0
Example for Linear Regression working
(con’t)
o Gradient Descent – Updating θ1 value
• Here, j = 1
Example for Linear Regression working
(con’t)
 Iteration 2 : θ0 = 0.005 and θ1 = 0.02657
o Predicted values after iteration 1 with Linear regression hypothesis.

o Now, similar to iteration no. 1 performed above we will again calculate Cost function and update θj
values using Gradient Descent.
o We will keep on iterating until Cost function doesn’t reduce further. At that point, model achieves
best θ values. Using these θ values in the model hypothesis will give the best prediction results.
Application of Linear Regression
 Machine learning
o Linear regression plays an important role in the field of artificial intelligence such as machine
learning. The linear regression algorithm is one of the fundamental supervised machine-learning
algorithms due to its relative simplicity and well-known properties.

 Finance
o The capital asset pricing model uses linear regression as well as the concept of beta for analyzing and
quantifying the systematic risk of an investment. This comes directly from the beta coefficient of the
linear regression model that relates the return on the investment to the return on all risky assets.
Application of Linear Regression (con’t)
 Environmental science
o Linear regression finds application in a wide range of environmental science applications. In Canada,
the Environmental Effects Monitoring Program uses statistical analyses on fish and benthic surveys to
measure the effects of pulp mill or metal mine effluent on the aquatic ecosystem.

 Economics
o Linear regression is the predominant empirical tool in economics. For example, it is used to predict
consumption spending, fixed investment spending, inventory investment, purchases of a country's
exports, spending on imports, the demand to hold liquid assets, labor demand, and labor supply.
Application of Linear Regression (con’t)
 Trend line
o A trend line represents a trend, the long-term movement in time series data after other components
have been accounted for. It tells whether a particular data set (say GDP, oil prices or stock prices)
have increased or decreased over the period of time.

 Epidemiology
o Early evidence relating tobacco smoking to mortality and morbidity came from observational studies
employing regression analysis. In order to reduce spurious correlations when analyzing observational
data, researchers usually include several variables in their regression models in addition to the
variable of primary interest.
Conclusion
 Linear regression is one of the most common machine learning
processes in the world and it helps prepare businesses in a volatile and
dynamic environment.

 Linear Regression is a very powerful statistical technique and can be

used to generate insights on consumer behavior, understanding business
and factors influencing profitability.
References
https://www.investopedia.com/terms/r/regression.asp
https://www.slideshare.net/oxygen024/regression-analysis-8672920
https://www.geeksforgeeks.org/ml-linear-regression/
https://analyticstraining.com/popular-applications-of-linear-regression-for-businesses/
https://en.wikipedia.org/wiki/Linear_regression#Applications