Scribd
Upload
107 views

Linear Regression

This document describes linear regression for predicting house prices based on living area size. It discusses: 1) Using a training dataset of house sizes and prices to learn a linear function that predicts price from size. 2) Representing the linear regression hypothesis as y = θ0 + θ1x, where θ0 and θ1 are parameters learned from the data. 3) Minimizing the cost function, the squared error between predictions and actual prices, through gradient descent to learn the best θ values.

Uploaded by

Kunal Langer
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
107 views

Linear Regression

This document describes linear regression for predicting house prices based on living area size. It discusses: 1) Using a training dataset of house sizes and prices to learn a linear function that predicts price from size. 2) Representing the linear regression hypothesis as y = θ0 + θ1x, where θ0 and θ1 are parameters learned from the data. 3) Minimizing the cost function, the squared error between predictions and actual prices, through gradient descent to learn the best θ values.

Uploaded by

Kunal Langer
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 51

CS 60050

Machine Learning

Linear Regression

Some slides taken from course materials of Andrew Ng


Dataset of living area and price
of houses in a city

This is a training set.

How can we learn to predict the prices of houses of


other sizes in the city, as a function of their living area?
Dataset of living area and price
of houses in a city

Example of supervised learning problem.

When the target variable we are trying to predict is


continuous, regression problem.
Dataset of living area and price
of houses in a city

m = number of training examples


x's = input variables / features
y's = output variables / "target" variables
(x,y) - single training example
(xi, yj) - specific example (ith training example)
i is an index to training set
How to use the training set?

Learn a function h(x), so


that h(x) is a good
predictor for the
corresponding value of y

h: hypothesis function
How to represent hypothesis h?

θi are parameters
- θ0 is zero condition
- θ1 is gradient

θ: vector of all the parameters

We assume y is a linear function of x


Univariate linear regression
Digression:
Multivariate linear regression
How to represent hypothesis h?

θi are parameters
- θ0 is zero condition
- θ1 is gradient

We assume y is a linear function of x


Univariate linear regression
How to learn the values of the parameters θi?
Intuition of hypothesis function

•  We are attempting to fit a straight line to the


data in the training set
•  Values of the parameters decide the equation
of the straight line
•  Which is the best straight line to fit the data?
Intuition of hypothesis function

•  Which is the best straight line to fit the data?


•  How to learn the values of the parameters θi?

•  Choose the parameters such that the


prediction is close to the actual y-value for
the training examples
Cost function

•  Measure of how close the predictions are to


the actual y-values
•  Average over all the m training instances

•  Squared error cost function J(θ)


•  Choose parameters θ so that J(θ) is minimized
Hypothesis:

Parameters:

Cost Function:

Goal:
(for fixed , this is a function of x) (function of the parameters )

500

400

Price ($)
in 1000’s 300

200

100

0
0 1000 2000 3000
Size in feet2 (x)
Contour plot or Contour figure
Minimizing a function

•  For now, let us consider some arbitrary


function (not necessarily an error function)

•  Algorithm called gradient descent

•  Used in many applications of minimizing


functions
Have some function

Want

Outline:
•  Start with some
•  Keep changing to reduce
until we hopefully end up at a minimum
J(θ0,θ1)

θ1
θ0
If the function has multiple local minima, where one starts
can decide which minimum is reached

J(θ0,θ1)

θ1
θ0
Gradient descent algorithm

α is the learning rate – more on this later


Gradient descent algorithm

Correct: Simultaneous update Incorrect:


For simplicity, let us first consider a function of
a single variable
The learning rate
•  Gradient descent can converge to a local
minimum, even with the learning rate α fixed

•  But, value needs to be chosen judiciously


•  If α is too small, gradient descent can be slow
to converge
•  If α is too large, gradient descent can
overshoot the minimum. It may fail to
converge, or even diverge.
Gradient descent for
univariate linear regression
Gradient descent algorithm Linear Regression Model
Gradient descent for univariate linear regression

update
and
simultaneously
“Batch” Gradient Descent

“Batch”: Each step of gradient descent uses


all the training examples.

There are other variations like “stochastic


gradient descent” (used in learning over
huge datasets)
What about multiple local minima?
•  The cost function in linear regression is always a
convex function – always has a single global
minimum

•  So, gradient descent will always converge


Convex cost function
Gradient descent in action
(for fixed , this is a function of x) (function of the parameters )
(for fixed , this is a function of x) (function of the parameters )
(for fixed , this is a function of x) (function of the parameters )
(for fixed , this is a function of x) (function of the parameters )
(for fixed , this is a function of x) (function of the parameters )
(for fixed , this is a function of x) (function of the parameters )
(for fixed , this is a function of x) (function of the parameters )
(for fixed , this is a function of x) (function of the parameters )
(for fixed , this is a function of x) (function of the parameters )
Linear Regression for
multiple variables
Multiple features (variables).

Size (feet2) Number of Number of Age of home Price ($1000)


bedrooms floors (years)

2104 5 1 45 460
1416 3 2 40 232
1534 3 2 30 315
852 2 1 36 178
… … … … …
Multiple features (variables).

Size (feet2) Number of Number of Age of home Price ($1000)


bedrooms floors (years)

2104 5 1 45 460
1416 3 2 40 232
1534 3 2 30 315
852 2 1 36 178
… … … … …

Notation:
= number of features
= input (features) of training example.
= value of feature in training example.
Hypothesis:

Previously:

For multi-variate linear regression:

For convenience of notation, define .


Hypothesis:

Parameters:

Cost function:

Gradient descent:
Repeat

(simultaneously update for every )


New algorithm :
Gradient Descent
Repeat
Previously (n=1):

Repeat

simultaneously update for

(simultaneously update )
New algorithm :
Gradient Descent
Repeat
Previously (n=1):

Repeat

simultaneously update for

(simultaneously update )
Practical aspects of applying
gradient descent
Feature Scaling
Idea: Make sure features are on a similar scale.

E.g. = size (0-2000 feet2) size (feet2)


= number of bedrooms (1-5)
number of bedrooms
Feature Scaling
Idea: Make sure features are on a similar scale.

E.g. = size (0-2000 feet2)


= number of bedrooms (1-5)

Mean normalization:
Replace with to make features have approximately zero
mean (Do not apply to ).

Other types of normalization:


Is gradient descent working properly?
•  Plot how J(θ) changes with every iteration of
gradient descent

•  For sufficiently small learning rate, J(θ) should


decrease with every iteration

•  If not, learning rate needs to be reduced

•  However, too small learning rate means slow


convergence
When to end gradient descent?
•  Example convergence test:

•  Declare convergence if J(θ) decreases by less


than 0.001 in an iteration (assuming J(θ) is
decreasing in every iteration)
Polynomial Regression for
multiple variables
Choice of features

Price
(y)

Size (x)

You might also like