LINEAR REGRESSION

Lecture # 02

Dr. Imran Khalil

[email protected]
 Contents
• What is Linear Regression?
• Linear Regression with one
Variable
• Cost Function
• Squared Error Cost Function
• Visualizing Cost Function

2
 Linear Regression
• Linear regression is a supervised learning algorithm used in
machine learning to model the relationship between a
dependent variable and one or more independent variables.
• It is a simple yet powerful technique used for both regression
(predicting continuous values) and understanding the
relationship between variables.
• The goal of linear regression is to find the best-fit line that
minimizes the distance between the predicted values and the
actual values in the training data.

3
Linear Regression with One Variable
❑ Example: Predicting House Price

4
Linear Regression with One Variable
❑ What is the price of house whose size is 750 sq. feet?

❑ Supervised Learning Model

5
Linear Regression with One Variable
❑ Training set of housing prices
Size in feet2 Price ($) in 1000’s
(𝑥) (𝑦)
2104 460
1416 232
1534 315
852 178
…. ….

❑ Training Set: The data set that is used to train the model.
6
Linear Regression with One Variable
❑ Notations

Notation Description
𝑥 Input variable or features
𝑦 Output variable or Target
𝑚 Number of training example
(𝑥, 𝑦) A single training example
(𝑥 𝑖 , 𝑦 𝑖 ) 𝑖𝑡ℎ row in the training set
(𝑥 2 , 𝑦 2 ) A training set of 2𝑛𝑑 row

7
 Linear Regression with One Variable
Training
Set

Learning
Algorithm

Size of the Function Estimated

House (𝒙) (𝒇) Price (ෝ
𝒚)

❑ 𝒇 maps 𝒙 (size of the house) to 𝒚 (price of the house)

8
Linear Regression with One Variable
❑ How to represent 𝒇?
𝒇𝒘,𝒃 𝒙 = 𝒘𝒙 + 𝒃
We can write as
𝒇 𝒙 = 𝒘𝒙 + 𝒃
❑ 𝒘 = slope of the function
❑ 𝒃 = 𝒚-intercept
❑ Univariate linear regression
9
Cost Function
❑ Training set of housing prices

Size in feet2 Price ($) in 1000’s

(x) (y)
2104 460
1416 232
1534 315
852 178
…. ….

❑ 𝒇 𝒙 = 𝒘𝒙 + 𝒃
❑ 𝒘, 𝒃 are parameters
❑ Question: How to choose 𝒘, 𝒃 10
Cost Function
❑ 𝒇 𝒙 = 𝒘𝒙 + 𝒃

𝑤=0 𝑤 = 0.5 𝑤 = 0.5

𝑏 = 1.5 𝑏=0 𝑏=1

11
Cost Function
❑ Find the values of 𝒘, 𝒃

❑ ෝ𝒊 is close to 𝒚𝒊 for all (𝒙𝒊 , 𝒚𝒊 )

𝒚
12
Squared Error Cost Function
𝒎
𝟏 𝟐
𝒎𝒊𝒏 ෍ 𝒇𝒘,𝒃 𝒙𝒊 − 𝒚𝒊
𝒘,𝒃 𝟐𝒎
𝒊=𝟏

𝟏
❑ Find the values of 𝒘 𝐚𝐧𝐝 𝒃 so that the average, the times
𝟐𝒎
the sum of square errors between our predictions on the
training set minus the actual values of the houses on the
training set is minimized.
❑ Overall objective function for linear regression.
13
Cost Function Intuition
❑ Model
𝒇𝒘,𝒃 𝒙 = 𝒘𝒙 + 𝒃
❑ Parameters
𝒘, 𝒃
❑ Cost Function 𝒎
𝟏 𝟐
𝑱𝒘,𝒃 = ෍ 𝒇𝒘,𝒃 𝒙𝒊 − 𝒚𝒊
𝟐𝒎
𝒊=𝟏
❑ Objective
𝑴𝒊𝒏𝒊𝒎𝒊𝒛𝒆 𝑱𝒘,𝒃
𝒘,𝒃 14
 Simplified Version of Cost Function

❑ Model ❑ Model
𝒇𝒘,𝒃 𝒙 = 𝒘𝒙 + 𝒃 𝒇𝒘 𝒙 = 𝒘𝒙
❑ Parameters ❑ Parameters
𝒘, 𝒃 𝒘
❑ Cost Function ❑ Cost Function
𝒎 𝒎
𝟏 𝟐 𝟏 𝟐
𝑱𝒘,𝒃 = ෍ 𝒇𝒘,𝒃 𝒙𝒊 − 𝒚𝒊 𝑱𝒘 = ෍ 𝒇𝒘 𝒙𝒊 − 𝒚𝒊
𝟐𝒎 𝟐𝒎
𝒊=𝟏 𝒊=𝟏
❑ Objective ❑ Objective
𝑴𝒊𝒏𝒊𝒎𝒊𝒛𝒆 𝑱𝒘,𝒃 𝑴𝒊𝒏𝒊𝒎𝒊𝒛𝒆 𝑱𝒘
𝒘,𝒃 𝒘 15
Cost Function

16
Squared Error Cost Function
❑ Suppose we have training set with 𝒎 = 𝟑 examples, plotted below. Our function
representation is 𝒇𝒘 𝒙 = 𝒘𝒙 with parameter w. The cost function is 𝑱𝒘 is
𝟏 𝒎 𝒊 𝒊 𝟐
𝑱𝒘 = σ𝒊=𝟏 𝒇𝒘 𝒙 − 𝒚 . What is the value of 𝑱𝒘 ?
𝟐𝒎

17
 Squared Error Cost Function
𝒎
𝟏 𝟐
𝑱𝒘 = ෍ 𝒇𝒘 𝒙𝒊 − 𝒚𝒊
𝟐𝒎
𝒊=𝟏
𝒎
𝟏 𝟐
𝑱𝒘 = ෍ 𝒘𝒙𝒊 − 𝒚𝒊
𝟐𝒎
𝒊=𝟏
𝟏
𝑱𝒘 = (𝟏 𝟏 − 𝟏)𝟐 +(𝟏 𝟐 − 𝟐)𝟐 +(𝟏 𝟑 − 𝟑)𝟐
𝟐(𝟑)
𝟏
𝑱𝒘 = (𝟎)𝟐 +(𝟎)𝟐 +(𝟎)𝟐
𝟔
𝑱𝒘 = 𝟎

18
Squared Error Cost Function
❑ 𝑺𝒖𝒑𝒑𝒐𝒔𝒆 𝒎 = 𝟑, 𝒘 = 𝟎. 𝟓, 𝒇𝒊𝒏𝒅 𝑱𝒘

19
Squared Error Cost Function
❑ 𝑺𝒖𝒑𝒑𝒐𝒔𝒆 𝒎 = 𝟑, 𝒘 = 𝟎, 𝒇𝒊𝒏𝒅 𝑱𝒘

20
Squared Error Cost Function
❑ 𝑺𝒖𝒑𝒑𝒐𝒔𝒆 𝒎 = 𝟑, 𝒘 = −𝟎. 𝟓, 𝒇𝒊𝒏𝒅 𝑱𝒘

21
Squared Error Cost Function
❑ 𝑯𝒐𝒘 𝒕𝒐 𝒄𝒉𝒐𝒐𝒔𝒆 𝒘 𝒕𝒐 𝒎𝒊𝒏𝒊𝒎𝒊𝒛𝒆 𝑱𝒘 ?

22
Cost Function Intuition
❑ Model
𝒇𝒘,𝒃 𝒙 = 𝒘𝒙 + 𝒃
❑ Parameters
𝒘, 𝒃
❑ Cost Function 𝒎
𝟏 𝟐
𝑱𝒘,𝒃 = ෍ 𝒇𝒘,𝒃 𝒙𝒊 − 𝒚𝒊
𝟐𝒎
𝒊=𝟏
❑ Objective
𝑴𝒊𝒏𝒊𝒎𝒊𝒛𝒆 𝑱𝒘,𝒃
𝒘,𝒃 23
Visualizing the Cost Function
❑ 𝑬𝒙𝒂𝒎𝒑𝒍𝒆: 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒊𝒏𝒈 𝑯𝒐𝒖𝒔𝒊𝒏𝒈 𝑷𝒓𝒊𝒄𝒊𝒏𝒈

24
Visualizing the Cost Function
❑ 𝑬𝒙𝒂𝒎𝒑𝒍𝒆: 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒊𝒏𝒈 𝑯𝒐𝒖𝒔𝒊𝒏𝒈 𝑷𝒓𝒊𝒄𝒊𝒏𝒈

25
 Data set Link
Kaggle https://www.Kaggle.com/datasets
Data Sets Sources

Amazon https://registry.opendata.aws/

UCI ML Repository https://archieve.ics.uci.edu/ml/index.php

Google’s data search https://datasetsearch.research.google.com/

engine
Micrsoft dataset https://msropendata.com/

Awesome Public https://github.com/awesomedata/awesome-public-datasets

data sets collections
Scikit - learn dataset https://scikit-learn.org/stable/

Computer Vision https://www.visualdata.io/discovery

datasets 26
Government Data Sets

Data set Link

US https://www.data.gov/

Northern Ireland https://www.opendatani.gov.uk/

European Union https://data.Europa.eu/euodp/data/dataset

India https://data.gov.in/

27
Acknowledgment
• Material presented in these lecture slides is obtained from Prof. Andrew
Ng course on Machine Learning
• Dr. Iftikhar Ahmad’s lecture slides were consulted for assistance.
• .

28