Manchester Metropolitan University

Econometrics:
Topic 2 The General Linear Model
Kevin Albertson
[email protected]
MMUBS 4·12
Office Hours: Tuesday and Thursday at 12:00
 Summary so far

Manchester Metropolitan University

• After lecture 1 you should understand:
– Econometrics is about answering economic questions with data
analysis, usually based on linear regression techniques.
– Regression modelling is about quantifying (measuring) relationships
between variables (estimation of parameters) and assessing statistical
confidence in these estimates (inference).
– Multiple regression is useful to improve the fit of the model and to
control for confounding influences; it gives us ceteris paribus
estimates.
– The conceptual difference between the population model (the
theoretical data generating process) and the sample model. 2
 Overview

Manchester Metropolitan University

• Matrix revision
• The general linear model
• Deriving the OLS estimator
• OLS assumptions
• The variance and standard error of the OLS estimator
• R-Squared – R²
• Summary

3
Manchester Metropolitan University
4
Matrix revision
 Matrix revision

Manchester Metropolitan University

• Matrices are rectangular blocks of numbers or symbols which
represent numbers
• In general, matrices can be added or subtracted (if they are
the same size)
• For matrix multiplication to be possible, the number of
columns in the first matrix must equal the number of rows in
the second

5
 Matrix revision
Matrix multiplication

Manchester Metropolitan University

Am,n × Bn,p = Cm,p
• The i,jth element of C, denoted ci,j, is the product of n
row i of matrix A and column j of matrix B cij   ai ,k bk , j
k 1

   b1 j    
   b2 j  
 
    
ai1 ai 2 ai 3  ain    b3 j      cij  
    
       
    bnj  
  
6
 Matrix revision
The transpose of a matrix

Manchester Metropolitan University

• The transpose of a matrix is created by changing the rows to
the columns and the columns to the rows, e.g.

2 3
- 1 1
2 - 1 7 5
A  A   
3 1 0   7 0
 
5 

7
 Matrix revision
The inverse of a matrix

Manchester Metropolitan University

• An identity matrix, usually denoted I, is one such that for any
other matrices, C and D
CI = C and ID = D
• The inverse of a matrix, A, denoted A-1, is a matrix such that
AA-1 = I and A-1A = I
• For 2 × 2 matrices, you can use Cramer’s rule
a b  1  d - b
A   A 
-1
- c a 
 c d  ad  bc  
8
 Matrix revision
The ABBA rules

Manchester Metropolitan University

• If all products are defined

(AB)’ = B’A’

(AB)⁻¹ = B⁻¹A⁻¹

9
 Matrix revision
Matrix rank

Manchester Metropolitan University

• Linear dependence of column vectors

• A vector A is said to be dependent on a set of vectors, B₁ to Bn

iff
A = k₁B₁ + k₂B₂ + … + knBn for some scalars k₁ … kn

• If we can not write D as a weighted sum of B₁ … Bn we say D

is independent of B₁ … Bn
10
 Matrix revision
Matrix rank

Manchester Metropolitan University

1 1 1
• For example A  2 is dependent on B1  1 and B2  3
     
3 1 5
A = ½B₁ + ½B₂
1 1 1
• For example C  2 is independent of B1  1 and B2  3
 
5 1 5
11
 Matrix revision
Matrix rank

Manchester Metropolitan University

• The Rank of a matrix, A is defined as the maximum number of
independent columns that can be taken from A

• Equivalently, the Rank of a matrix, A is defined as the

maximum number of independent rows that can be taken
from A

12
 Matrix revision
Matrix rank

Manchester Metropolitan University

1 1 1
_2 1 3_has a rank of 2
3 1 5

1 1 1
 
_2 1 3_has a rank of 3
5 1 5
13
 Matrix revision
Properties of matrix rank

Manchester Metropolitan University

• Rank(0) = 0
• Rank(Apq)  min(p, q)
• If AB = C, Rank(C)  min(Rank(A), Rank(B))
• App is invertible iff Rank (A) = p
• Rank(A’) = Rank(A)
• For any scalar k  0, Rank(kA) = Rank(A)

14
 Random vectors

Manchester Metropolitan University

 x
 y
• Consider the n × 1 vector L   

 
z
where each element of L is a random
variable with the following properties:
– E(x) = μx &c.
– var(x) = σx² &c.
– covar(x,y) = σx,y &c.
15
 Random vectors

Manchester Metropolitan University

• The mean of the random vector L is also a vector
– The ith element of E(L) is E(Li)

  x   E x   x 
     
 y  E y 
EL   E         y
      
  
 z  E z   
     z

16
 Random vectors

Manchester Metropolitan University

• The covariance of the random vector L is a matrix
Σ(L) = E((L – E(L)(L – E(L))’)

 2x  x, y   x,z 
 
  x,y  y   y ,z 
2

 L  
     
 2 
 x , z  y , z   z 

17
 Random vectors

Manchester Metropolitan University

• If each element of the n×1 random vector L is normally
distributed, we say L is Multivariate Normal
• L ~ MVN(E(L), Σ(L))
• Let Y and X be two independent n×1 random vectors, Kn,1
and Am,n are matrices of constants
• If Y ~ MVN(μY, ΣY), X ~ MVN(μX, ΣX)
– (Y + K) ~ MVN(K + μY, ΣY)
– (Y + X) ~ MVN(μY + μX , ΣY + ΣX)
– (AY ) ~ MVN(AμY , AΣY A’)
18
Manchester Metropolitan University
19
The general linear model
 The general linear model

Manchester Metropolitan University

• The general linear model is
yᵢ = β₀ + β₁xᵢ + β₂wᵢ + … + βK–1zᵢ + εᵢ; i = 1,…,n
where:
– xᵢ, wᵢ, …, zᵢ represent each of K – 1 regressors
– we have K parameters in total to estimate with n observations on
each variable
– β₀ is an intercept term
– each of β₁ , … , βK–1 is a slope term (partial differential)
– εᵢ is an unobserved random element
20
 The general linear model
Matrix notation

Manchester Metropolitan University

• If we write each observation’s equations one above the other
we have:
y1 = β0 + β1x1 + β2w1 + … + βK-1z1 + ε1
y2 = β0 + β1x2 + β2w2 + … + βK-1z2 + ε2
y3 = β0 + β1x3 + β2w3 + … + βK-1z3 + ε3
: : : : : :
yn = β0 + β1xn + β2wn + … + βK-1zn + εn

21
 The general linear model
Matrix notation

Manchester Metropolitan University

• According to the rules of matrix multiplication we can write:

y₁ = [ 1 x₁ w₁ … z₁] + ε₁
 0 
y₂ = [ 1 x₂ w₂ … z₂]    + ε₂
y₃ = [ 1 x₃ w₃ … z₃]  1  + ε₃
  
:  
yn = [ 1 xn wn … zn] 
 K 1  + εn

22
 The general linear model
Matrix notation

Manchester Metropolitan University

• According to the rules of matrix multiplication we can write:

y₁ = [ 1 x₁ w₁ … z₁] + ε₁
 0 
y₂ = [ 1 x₂ w₂ … z₂]    + ε₂
y₃ = [ 1 x₃ w₃ … z₃]  1  + ε₃
  
:  
yn = [ 1 xn wn … zn] 
 K 1  + εn

23
 The general linear model
Matrix notation

Manchester Metropolitan University

• According to the rules of matrix multiplication we can write:

 y1  1 x1 w1  z1   1 
 y  1   0   
x2 w2  z2  
 2   2   2 
 y3   1 x3 w3  z3      3 
       
          
 K 1 
 yn  1 xn wn  z n   n 

y = Xβ + ε
24
 The general linear model
Matrix notation

Manchester Metropolitan University

• The general linear model is written
y = Xβ + ε
where
y is a n × 1 vector of n observations on the dependent
variable
X is a n × K matrix of n observations of K – 1 independent
variables and a constant
β is an unobserved K × 1 vector of parameters
ε is an unobserved n × 1 random vector 25
 The general linear model

Manchester Metropolitan University

• In the general linear model
y = Xβ + ε
we wish to estimate the β parameter vector
• Let the estimate be denoted b
• The fitted model is given by
y = Xb + e
• Because the data is random, the estimated coefficients will
not always equal the true coefficients
26
Manchester Metropolitan University
27
Deriving the OLS estimator
 The general linear model
Deriving the OLS estimator

Manchester Metropolitan University

• The fitted model is given by
y = Xb + e
• For any estimate b we measure the distance between the
actual data and the estimated relationship
• This is reflected in the residual
e = y – Xb
• The “best” b will minimise the sum of squared residuals
SSR = e’e = (y – Xb)’(y – Xb)
28
 The general linear model
Deriving the OLS estimator

Manchester Metropolitan University

SSR = e’e
= (y – Xb)’(y – Xb)
= (y’ – b’X’)(y – Xb)
= y’y – b’X’y – y’Xb + b’X’Xb
= y’y – 2b’X’y + b’X’Xb
• Note that SSR is a scalar (not a matrix)
• To minimise SSR we differentiate and set the first differential
equal to zero
29
 The general linear model
Deriving the OLS estimator

Manchester Metropolitan University

SSR = y’y – 2b’X’y + b’X’Xb

SSR  yy  2bX y  bX Xb 

  2 X y  2 X Xb
b b
• We require b such that
2X’y + 2X’Xb = 0
• The Ordinary Least Squares Estimator of β is
b = (X’X)⁻¹X’y
30
Manchester Metropolitan University
31
OLS assumptions
 The general linear model
Assumptions

Manchester Metropolitan University

• In the general linear model
y = Xβ + ε
we generally assume:
– E(εᵢ) = 0
– The regressors are non-random (or at least independent of the errors)
– The error has a constant variance var(εᵢ) = σ²
– The error in any one observation is independent of the errors in all
other observations
– The regressors are not linearly related to each other
– The error εᵢ has a normal distribution 32
 The general linear model
Assumptions

Manchester Metropolitan University

• In the general linear model
y = Xβ + ε
we generally assume:
– X’X is invertible
• this condition will hold if rank(X) = K
– X is non random or at least independent of ε
– ε ~ N(0, σ²I), where I is an n × n identity matrix

33
 The general linear model
Properties of estimates

Manchester Metropolitan University

• In the general linear model
y = Xβ + ε
given our assumptions, Ordinary Least Squares estimates are
BLU
Best (minimum variance) of all Linear Unbiased estimators
• If one or more of the least squares assumptions is invalid, we
should consider a different estimation method

34
 Manchester Metropolitan University
The variance and standard error of the OLS estimator

35
 The general linear model
Properties of estimates

Manchester Metropolitan University

• We have y = Xβ + ε, where ε ~ N(0, σ²I)

Therefore y ~ N(Xβ, σ²I)

• As b = (X’X)⁻¹X’y
b ~ N(β, σ²(X’X)⁻¹)

SSR
• An estimator of var(εᵢ) = σ² is s 
2

nK
36
 The general linear model
Properties of estimates

Manchester Metropolitan University

• Let bk denote the estimate of βk in
yᵢ = β₀ + β₁xᵢ + β₂wᵢ + … + βK-1zᵢ + εᵢ
• Let Sk denote the kth diagonal element of (X’X)⁻¹

• Given our assumptions bk ~ N(βk, σ²Sk)

• An estimate of the standard deviation of bk is given by the

standard error
sebk   s 2 S k
37
Manchester Metropolitan University
38
R-Squared – R²
 The general linear model
R-Squared – R²

Manchester Metropolitan University

• Goodness-of-Fit
– “How well does the explanatory variable explain the dependent
variable?”

• Measures of Variation
n n n
SST    yi  y  2
SSE    yˆi  y 2 SSR    yi  yˆi 2
i 1 i 1 i 1

Total sum of squares, Explained sum of squares, Residual sum of squares,

represents total variation represents variation represents variation not
in dependent variable explained by regression explained by regression 39
 The general linear model
R-Squared – R²

Manchester Metropolitan University

• Goodness-of-Fit
– Measures of Variation: SST = SSE + SSR
• R² measures the fraction of the total variation that is
explained by the regression
SSE SSR SSR
R 
2
1 1 n
SST SST y  y
2
i
i 1

– R² lies in the range [0,1]

40
 The general linear model
R-Squared – R²

Manchester Metropolitan University

• R² will always increase if we add more regressors to the
model
– If n = K, R² =1, even if the regressors are irrelevant
• The adjusted R² adjusted for the degrees of freedom in the
model
SSR
R  1  n n 2K
2
  yi  y 
i 1
n 1

41
Manchester Metropolitan University
42
Summary
Summary

Manchester Metropolitan University

• We can conveniently frame any multiple linear regression
model using matrix notation
• This allows a single equation “Least Squares” estimator to be
derived
• The least squares estimates are themselves random variables
• Applying (so-called) least squares assumptions allows us to
describe the statistical properties of estimators
• R 2 and R 2 allow us to describe how well the model fits the
data.
43