Scribd
Upload
14 views

Linear Regression Analaysis - 21

This document discusses multiple linear regression analysis using matrix notation. It defines the multiple linear regression model components such as the design matrix X, parameter vector β, error vector ε, and response vector Y. It explains how to write the model in matrix form as Y = Xβ + ε and describes how to estimate the parameters using the least squares method, which minimizes the sum of squared residuals. It also covers testing the significance of individual regression coefficients and constructing confidence intervals for the coefficients.

Uploaded by

mhjitu
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
14 views

Linear Regression Analaysis - 21

This document discusses multiple linear regression analysis using matrix notation. It defines the multiple linear regression model components such as the design matrix X, parameter vector β, error vector ε, and response vector Y. It explains how to write the model in matrix form as Y = Xβ + ε and describes how to estimate the parameters using the least squares method, which minimizes the sum of squared residuals. It also covers testing the significance of individual regression coefficients and constructing confidence intervals for the coefficients.

Uploaded by

mhjitu
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 22

Linear Regression Analysis

Multiple Linear Regression in Matrix Notation


• The MLR Model in Scalar Form
Yi = β0 + β1Xi + … + βkXki + i where i ∼iid N(0, σ2 ).
Consider now writing an equation for each observation:
Y1 = β0 + β1X1 + … + βkXk1 + 1
Y2 = β0 + β1X2 + … + βkXk2 + 2
... ... ...
Yn = β0 + β1Xn + … + βkXkn + n
The MLR Model in Matrix Form
𝑌1 0 + 1 𝑋1 + ⋯ + 𝑘 𝑋𝑘1 1
𝑌2 0 + 1 𝑋2 + ⋯ + 𝑘 𝑋𝑘2 2
= + ⋮
⋮ ⋮
𝑌𝑛 0 + 1 𝑋𝑛 + ⋯ + 𝑘 𝑋𝑘𝑛 𝑛

𝑌1 1 𝑋11 𝑋21 … 𝑋𝑘1 0 1


𝑌2 1 𝑋12 𝑋22 … 𝑋𝑘2 1 2
Or , = + ⋮
⋮ ⋮ ⋮ ⋮
𝑌𝑛 1 𝑋1𝑛 𝑋2𝑛 … 𝑋𝑘𝑛 𝑘 𝑛
• X is called the design matrix.
• β is the vector of parameters
•  is the error vector
• Y is the response vector.
The Design Matrix
1 𝑋11 𝑋21 … 𝑋𝑘1
1 𝑋12 𝑋22 … 𝑋𝑘2
𝑋𝑛×(𝑘+1) =
⋮ ⋮
1 𝑋1𝑛 𝑋2𝑛 … 𝑋𝑘𝑛
Vector of Parameters
0
1
β(k+1)×1 =
⋮
𝑘
Vector of Error Terms
1
2
𝑛×1 = ⋮
𝑛
Vector of Responses
𝑌1
𝑌2
𝒀𝑛×1 =
⋮
𝑌𝑛
Multiple Linear Regression Model in Matrix Form
Thus,
Y = Xβ +
Covariance matrix of 
Covariance matrix of Y
Distributional Assumptions in Matrix Form
∼ N(0, σ2I) where I is an n × n identity matrix.
• Ones in the diagonal elements specify that the variance of each i is 1
times σ2.
• Zeros in the off-diagonal elements specify that the covariance
between different i is zero.
• This implies that the correlations are zero.
• Xi’s are independent.
Parameter Estimation
Least Squares
• Residuals are  = Y − Xβ.
• Want to minimize sum of squared residuals.
1
2
𝑖=1 𝜀𝑖 = (1 2 … 𝑛 ) ⋮ = '
𝑛 2
•
𝑛

• We want to minimize ' = (Y−Xβ)’ (Y−Xβ).


• We take the derivative with respect to the vector β.
• This is like a quadratic function: (Y − Xβ)2.
• The derivative works out to 2 times the derivative of (Y − Xβ)’ with respect to β.
• That is, d/dβ ((Y − Xβ)’(Y − Xβ)) = −2X’(Y − Xβ).
• We set this equal to 0 (a vector of zeros), and solve for β.
• So, −2X’(Y − Xβ) = 0.
• Or, X’Y = X’Xβ (the “normal” equations).
• Solving this equation for β gives the least squares solution for β.
• Multiply on the left by the inverse of the matrix X’X.
• The matrix X’X is a (k+1) × (k+1) square matrix for SLR.
• 𝛽 = (𝑋 ′ 𝑋)−1 𝑋 ′ 𝑌.
In simple linear regression, we defined,
1 1 … 1 1 𝑋11 … 𝑋𝑘1
′ 𝑋11 𝑋12 … 𝑋1𝑛 1 𝑋12 … 𝑋𝑘2
•𝑋𝑋=
⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
𝑋𝑘1 𝑋𝑘2 … 𝑋𝑘𝑛 1 𝑋1𝑛 … 𝑋𝑘𝑛
𝑛 𝑛

𝑛 𝑋1𝑖 … 𝑋𝑘𝑖
𝑖=1 𝑖=1
𝑛 𝑛 𝑛
2
= 𝑋1𝑖 𝑋1𝑖 … 𝑋1𝑖 𝑋𝑘𝑖
𝑖=1 𝑖=1 𝑖=1
⋮ ⋮ ⋮ ⋮
𝑛 𝑛 𝑛
2
𝑋𝑘𝑖 𝑋1𝑖 𝑋𝑘𝑖 … 𝑋𝑘𝑖
𝑖=1 𝑖=1 𝑖=1
Distribution of 𝛽
We know that
𝛽 = (X’X)−1X’Y.
The only random variable involved is Y, so the distribution of 𝛽 is based
on the distribution of Y.
Since Y ∼ N(Xβ,σ2I),
we have E(𝛽) = (X’X) −1 X’Xβ = β
σ2𝛽 = Cov(𝛽) = σ2 (X’X) −1
Since σ2 is estimated by the MSE, s2, σ2𝛽 is estimated by s2(X’X)−1.
ANOVA Table
Sources of variation are:
Regression
Error (Residual)
Total

SS and df add as:


SSR + SSE = TSS
dfR + dfe = dfT
ANOVA Table

Source df SS MS F
Regression P-1=k+1 – 1 = k RSS MSR = RSS/(p – 1) MSR/MSE
Error n–p=n–2 SSE MSE = SSE / (n – p)
Total n–1
Inference for Individual Regression Coefficients
• Confidence Interval for 𝛽 1
• We know that 𝛽 1∼ N(β, σ2(X’X)−1)
• Define var{𝛽 1}p×p = 𝜎2(X’X)−1 = MSE × (X’X) −1
• CI for β1:

𝛽 1 ± tn – (k+1), 1 - /2 s.e.{𝛽 1}.


Significance Test for βj
H0 : β j = 0

• test statistic t∗ = 𝛽 1 /s.e.(𝛽 1)


• Where t* follows t-distribution with (n – p) d.f.
• This tests the significance of the independent variable in the model.
• In SLR, the t-tests for β are same as the F-test.
• In MLR, the t-tests are used for testing the significance of each βj.
Example:
Consider a data set, on price of houses.
area (x1) (00 sq.ft) rooms (x2) age (x3) (years) price (y) (in 000)

23 3 8 6562

15 2 7 4569

24 4 9 6897

29 5 4 7562

31 7 6 8234

25 3 10 7485

You might also like