4 linear regeression with multiple variables

4. MULTIVARIATE LINEAR
REGERESSION:
Hypothesis:

X = features vector or design vector
ΘT
= transpose of Θ
Θ = parameter vector
➢In linear regression with ONE VARIABLES: n=1
➔thus n+1 = 2 ➔ for Θ0 and Θ1

For multiple variables: n > 1
➢ Used when all input var. have different range of allowed
values. This makes optimizing slower. Its tedious to find the
local minima.

Example: if diff ranges are used the contours are quite steep type
➢ θ will descend quickly on small ranges and slowly on large
ranges, and so will oscillate inefficiently down to the optimum
when the variables are very uneven.
To solve this: we can change scaling of x1 and x2
This will make the contours more balanced.
This is done to bring approximate values of all xi near a same range.
-1 and 1 are not necessary for all xi …we can work with nearly equal
ranges, like -3 to 3, etc… comparable ranges

➢We can speed up gradient descent by having each of our input
values in roughly the same range.
Ranges that would work:
Ranges that won’t work: if ranges are too larger or too smaller than
± 1
NOTE: x0 = 1 always. Its scaling is not changed.
➢There are two ways to change the ranges of x:
o Feature scaling
o Mean normalization

➔ Feature scaling involves dividing the input values by the
range (i.e. the maximum value minus the minimum value) of
the input variable, resulting in a new range of just 1.
For x1 : average size = 1000
Range = 2000 = upper limit – lower limit
For x2 : average size = 2
Range = 5
➢In mean normalization we try to bring xi in approx. range:

Note that dividing by the range, or dividing by the standard
deviation, give different results
PRICTICAL TIPS: for grad desc.

Making sure gradient descent is working correctly:
The goal is to minimize J
Plot J vs no of iterations, (not J vs Θ): J should decrease after every
iteration. In this curve, J(Θ) is the vertical height of that point.
After a time, the curve flattens – denoting the convergence has
occurred.

If J(θ) ever increases, then you probably need to decrease α.
All these are wrong curves for J(Θ) vs iterations. Solution: use
smaller values of α .
But not too small α as it slows the convergence. And not too large
either: as it may not converge.

DEFINING NEW FEATURES:
POLYNOMIAL REGRESSION: Non-linear hypothesis
➔ We can use different hypothesis equations for a single
dataset. Whichever best fits logically.
➔ For a multivariate: we can convert all features into
functions of each other:

Choice of features: We can convert our linear hypothesis into a
non-linear one
IMPORTANT: if you choose your features this way then feature
scaling becomes very important.
Thus, we find the best fitting curve for h(Θ).

Computing Parameters Analytically:
Up until now, we are using gradient descent Algorithm.. but now
we will use new Algo: NORMAL EQUATIONS
NORMAL EQUATIONS: method to solve for Θ analytically…
unlike grad desc, no need to iterate to minimize the J(Θ).. its
minimized directly in one go.
Intuition: for a single parameter Θ:
For multiple parameters:
Θ is a set of m Θ’s.
For every Θi ➔ we set partial derivative of J wrt to Θi == 0
o Then we find Θ corresponding to that eqn
o This is done for each Θ

m = number of example datas
n = no of features in input
n + 1 => we give an extra feature x0=1 to every example.
To construct the X matrix from xi vectors:
o Transpose them and fill into the X matrix, Such that the x’s
belonging to a single example.. comes in row

In above example : for all m training sets there are only 2 features
x0 and x1.
X’ = transpose of X
Feature scaling is not required in Normal Equations
method(algo)..Unlike in gradient desc => in which its req
WHEN TO USE GRAD DESC v/s NORMAL EQN:

When no. of features is small (upto 105) => use normal eqns.
As for large value of n=> X’ * X will be a n x n matrix: and we have
to find its inverse:
Inverse is of complexity O(n3
) => thus for large no. of input
features, grad desc is better way to converge to minima.
NON INVERTIBILITY PROBLEM IN NORMAL EQN METHOD:
Sometimes X’ * X is not inventible (singular/degenerate)..:
REASONS:
Redundant features – two columns or rows are proportional in the
X’ * X matrix. (i.e. they are linearly dependent)
SOLUTION: delete one of the dependent features.
Too many features – the number of features is too large as
compares to no of examples… (m << n)
SOLUTION: delete some features or Use REGULARIZATION
TECHNIQUE.
$OCTAVE: pinv(X’ * X) * X’ * y
 this would still give the right value of Θ. (Even if X’ * X is non
invertible).
pinv() is pseudo inverse
inv() is just inverse

4 linear regeression with multiple variables

Recommended

More Related Content

What's hot (20)

Similar to 4 linear regeression with multiple variables (20)

Recently uploaded (20)

4 linear regeression with multiple variables