CS 4104

APPLIED MACHINE LEARNING

Dr. Hashim Yasin

National University of Computer
and Emerging Sciences,
Faisalabad, Pakistan.
LINEAR REGRESSION
 Linear Regression with one Variable
3

Housing Prices
(Portland, OR)

of dollars)
(in 1000s
Price

Size (feet2)
Supervised Learning Regression Problem
Given the “right answer” for Predict real-valued output
each example in the data.
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Regression Example
4

Training set of Size in feet2 (x) Price ($) in 1000's (y)

housing prices 2104 460
1416 232
1534 315
852 178
… …
Notation:
m = Number of training examples One Training example 𝑥, 𝑦
x’s = “input” variable / features 𝑖𝑡ℎ training example (𝑥 𝑖 , 𝑦 𝑖 )
y’s = “output” variable / “target” variable

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Regression
5

Training Set

Learning Algorithm

Size of Estimated
house (X)
h price (Y)

Question : How to describe h?

ℎ: 𝑋 → 𝑌
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Regression Example
6

Training set of Size in feet2 (x) Price ($) in 1000's (y)

housing prices 2104 460
1416 232
1534 315
852 178
… …

Hypothesis:
‘s: Parameters
How to choose ‘s ?
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Regression
7

 How to choose these parameters , 𝜃 (regression

coefficient)?

 The standard approach is the least square method,

through which parameters are minimized

 The machine learning program optimizes the

parameters, 𝜃, such that the approximation error is
minimized.
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Regression
8

Idea: Choose so that

is close to for our
training examples

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Cost Function
9

Simplified:
Hypothesis:

Parameters:

Cost Function:

Goal:

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Regression
10

𝑚 𝑚

෍ 𝜃1 𝑥 2𝑖 − ෍ 𝑥 𝑖 𝑦 𝑖 = 0
𝑚 𝑖=1 𝑖=1
𝜕 𝐽(𝜃1 ) 𝜕 2
= ෍ ℎ𝜃 𝑥 𝑖 − 𝑦 𝑖 𝑚 𝑚
𝜕𝜃1 𝜕𝜃1
𝑖=1
෍ 𝜃1 𝑥 2𝑖 = ෍ 𝑥 𝑖 𝑦 𝑖
𝑚
𝜕 𝐽(𝜃1 ) 𝜕 𝑖=1 𝑖=1
2
= ෍ 𝜃1 𝑥 𝑖 − 𝑦 𝑖
𝜕𝜃1 𝜕𝜃1 σ𝑚 𝑖 𝑖
𝑖=1 𝑖=1 𝑥 𝑦
𝜃1 = 𝑚 2𝑖
𝑚 σ𝑖=1 𝑥
𝜕 𝐽(𝜃1 ) 𝜕
= 2 ෍ 𝜃1 𝑥 𝑖 − 𝑦 𝑖 𝜃1 𝑥 𝑖 − 𝑦 𝑖
𝜕𝜃1 𝜕𝜃1
𝑖=1
𝑚 𝑐𝑜𝑣𝑎𝑟(𝑋, 𝑌)
𝜕 𝐽(𝜃1 )
= 2 ෍ 𝜃1 𝑥 𝑖 − 𝑦 𝑖 𝑥 𝑖 = 0 𝑣𝑎𝑟(𝑋)
𝜕𝜃1 if 𝑚𝑒𝑎𝑛(𝑋) = 𝑚𝑒𝑎𝑛(𝑌) = 0
𝑖=1

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Regression
11

Price
($)
in
1000’s

Size in feet2 (x)

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
12

Have some function

Want

Outline:
• Start with some
• Keep changing to reduce
until we hopefully end up at a minimum
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Gradient Descent
13

Gradient descent algorithm

Correct: Simultaneous update Incorrect:

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
14

Gradient descent algorithm

Notice : α is the learning rate.

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
15

If α is too small, gradient

descent can be slow.

If α is too large, gradient

descent can overshoot the
minimum. It may fail to
converge, or even diverge.

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
16

Unchange

at local optima

Current value of
Gradient descent can converge to a local minimum, even with the
learning rate α fixed.
As we approach a local minimum, gradient descent will
automatically take smaller steps. So, no need to decrease α over
time.
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Gradient Descent
17

Gradient descent algorithm Linear Regression Model

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
18

J(0,1)

1
0
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Gradient Descent
19

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
20

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
21

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
22

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
23

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
24

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
25

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
26

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
27

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
28

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Acknowledgement
29

Tom Mitchel, Russel & Norvig, Andrew Ng, Alpydin &

Ch. Eick.

Dr. Hashim Yasin Applied Machine Learning (CS4104)