APPENDIX

E
The Linear Regression
Model in Matrix Form

T
his appendix derives various results for ordinary least squares estimation of the multiple
linear regression model using matrix notation and matrix algebra (see Appendix D for
a summary). The material presented here is much more advanced than that in the text.

E.1 The Model and Ordinary Least Squares Estimation

Throughout this appendix, we use the t subscript to index observations and an n to denote
the sample size. It is useful to write the multiple linear regression model with k parameters
as follows:

yt 5 b0 1 b1xt1 1 b2xt2 1 … 1 bk xtk 1 ut, t 5 1, 2, …, n, [E.1]

where yt is the dependent variable for observation t, and xtj, j 5 1, 2, …, k, are the indepen-
dent variables. As usual, b0 is the intercept and b1, …, bk denote the slope parameters.
For each t, define a 1 (k 1 1) vector, xt 5 (1, xt1, …, xtk), and let 5 (b0, b1, …,
bk) be the (k 1 1) 1 vector of all parameters. Then, we can write (E.1) as

yt 5 xt 1 ut, t 5 1, 2, …, n. [E.2]

[Some authors prefer to define x t as a column vector, in which case x t is replaced

with xt in (E.2). Mathematically, it makes more sense to define it as a row vector.]
We can write (E.2) in full matrix notation by appropriately defining data vectors and
matrices. Let y denote the n 1 vector of observations on y: the t th element of y is yt.
Let X be the n (k 1 1) vector of observations on the explanatory variables. In other
words, the t th row of X consists of the vector xt. Written out in detail,

 
x1 1 x11 x12 ... x1k
x2 1 x21 x22 ... x2k
X .
. 5 .. .
n (k 1 1) . .
xn 1 xn1 xn2 ... xnk

807
Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 808 APPENDICES

Finally, let u be the n 1 vector of unobservable errors or disturbances. Then, we can

write (E.2) for all n observations in matrix notation:

y 5 X 1 u. [E.3]
Remember, because X is n (k 1 1) and is (k 1 1) 1, X is n 1.
Estimation of proceeds by minimizing the sum of squared residuals, as in Section 3.2.
Define the sum of squared residuals function for any possible (k 1 1) 1 parameter
vector b as
n
SSR(b) ∑ ( y 2 x b) .
t51
t t
2

The (k 1 1) 1 vector of ordinary least squares estimates, ˆ 5 (b ˆ0, b

ˆ1, …, b
ˆk) , minimizes
SSR(b) over all possible (k 1 1) 1 vectors b. This is a problem in multivariable calculus. For
ˆ to minimize the sum of squared residuals, it must solve the first order condition

∂SSR( ˆ)/∂b 0. [E.4]

Using the fact that the derivative of ( yt 2 xtb)2 with respect to b is the 1 (k 1 1)
vector 22( yt 2 xtb)xt, (E.4) is equivalent to
n

∑ x ( y 2 x ˆ)
t51
t t t 0. [E.5]

(We have divided by 22 and taken the transpose.) We can write this first order condition as
n

∑ ( y 2 bˆ
t51
t 0
ˆ1xt1 2 … 2 b
2b ˆk xtk) 5 0

∑ x ( y 2 bˆ
t51
t1 t 0
ˆ1xt1 2 … 2 b
2b ˆk xtk) 5 0

.
.
.
n

∑ x ( y 2 bˆ
t51
tk t 0
ˆ1xt1 2 … 2 b
2b ˆk xtk) 5 0,

which is identical to the first order conditions in equation (3.13). We want to write these
in matrix form to make them easier to manipulate. Using the formula for partitioned
multiplication in Appendix D, we see that (E.5) is equivalent to

X (y 2 Xˆ) 5 0 [E.6]

or
(X X)ˆ 5 X y. [E.7]

It can be shown that (E.7) always has at least one solution. Multiple solutions do not help
us, as we are looking for a unique set of OLS estimates given our data set. Assuming that
the (k 1 1) (k 1 1) symmetric matrix X X is nonsingular, we can premultiply both
sides of (E.7) by (X X)21 to solve for the OLS estimator ˆ:

​ ˆ 5 (X X)21X y. [E.8]

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 APPENDIX E The Linear Regression Model in Matrix Form 809

This is the critical formula for matrix analysis of the multiple linear regression model.
The assumption that X X is invertible is equivalent to the assumption that rank(X) 5
(k 1 1), which means that the columns of X must be linearly independent. This is the ma-
trix version of MLR.3 in Chapter 3.
Before we continue, (E.8) warrants a word of warning. It is tempting to simplify the
formula for ˆ as follows:

​ ˆ 5 (X X)21X y 5 X21(X )21X y 5 X21y.

The flaw in this reasoning is that X is usually not a square matrix, so it cannot be inverted.
In other words, we cannot write (X X)21 5 X21(X )21 unless n 5 (k 1 1), a case that vir-
tually never arises in practice.
The n 1 vectors of OLS fitted values and residuals are given by

y 5 X ˆ, u
ˆ y 5 y 2 X ˆ, respectively.
ˆ5y2ˆ

u, we can see that the first order condition for ˆ is the

From (E.6) and the definition of ˆ
same as

Xˆ
u 5 0. [E.9]

Because the first column of X consists entirely of ones, (E.9) implies that the OLS re-
siduals always sum to zero when an intercept is included in the equation and that the
sample covariance between each independent variable and the OLS residuals is zero. (We
discussed both of these properties in Chapter 3.)
The sum of squared residuals can be written as
n
SSR 5 ∑ uˆ
t51
2
t 5ˆ
uˆu 5 (y 2 X ˆ) (y 2 X ˆ). [E.10]

All of the algebraic properties from Chapter 3 can be derived using matrix algebra. For ex-
ample, we can show that the total sum of squares is equal to the explained sum of squares
plus the sum of squared residuals [see (3.27)]. The use of matrices does not provide a sim-
pler proof than summation notation, so we do not provide another derivation.
The matrix approach to multiple regression can be used as the basis for a geometrical
interpretation of regression. This involves mathematical concepts that are even more ad-
vanced than those we covered in Appendix D. [See Goldberger (1991) or Greene (1997).]

E.2 Finite Sample Properties of OLS

Deriving the expected value and variance of the OLS estimator ˆ is facilitated by matrix
algebra, but we must show some care in stating the assumptions.

Assumption E.1 Linear in Parameters

The model can be written as in (E.3), where y is an observed n 1 vector, X is an n (k 1 1)
observed matrix, and u is an n 1 vector of unobserved errors or disturbances.

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 810 APPENDICES

Assumption E.2 No Perfect Collinearity

The matrix X has rank (k 1 1).

This is a careful statement of the assumption that rules out linear dependencies among the
explanatory variables. Under Assumption E.2, X X is nonsingular, so ˆ is unique and can
be written as in (E.8).

Assumption E.3 Zero Conditional Mean

Conditional on the entire matrix X, each error ut has zero mean: E(utX) 5 0, t 5 1, 2, …, n.

In vector form, Assumption E.3 can be written as

E(uX) 5 0. [E.11]

This assumption is implied by MLR.4 under the random sampling assumption, MLR.2.
In time series applications, Assumption E.3 imposes strict exogeneity on the explana-
tory variables, something discussed at length in Chapter 10. This rules out explanatory
variables whose future values are correlated with ut; in particular, it eliminates lagged
dependent variables. Under Assumption E.3, we can condition on the xtj when we compute
the expected value of ˆ.

THEOREM UNBIASEDNESS OF OLS

E.1 Under Assumptions E.1, E.2, and E.3, the OLS estimator ˆ is unbiased for .
PROOF: Use Assumptions E.1 and E.2 and simple algebra to write

ˆ 5 (X X)21X y 5 (X X)21X (X 1 u)
5 (X X)21(X X) 1 (X X)21X u 5 1 (X X)21X u, [E.12]

where we use the fact that (X X)21(X X) 5 Ik 1 1. Taking the expectation conditional on X gives

E( ˆX) 5 1 (X X)21X E(uX)

5 1 (X X)21X 0 5 ,

because E(uX) 5 0 under Assumption E.3. This argument clearly does not depend on the
value of , so we have shown that ˆ is unbiased.

To obtain the simplest form of the variance-covariance matrix of ˆ, we impose the

assumptions of homoskedasticity and no serial correlation.

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 APPENDIX E The Linear Regression Model in Matrix Form 811

Assumption E.4 Homoskedasticity and No Serial Correlation

(i) Var(utX) 5 2, t 5 1, 2, …, n. (ii) Cov(ut,usX) 5 0, for all t s. In matrix form, we can
write these two assumptions as

Var(uX) 5 2In, [E.13]

where In is the n n identity matrix.

Part (i) of Assumption E.4 is the homoskedasticity assumption: the variance of ut cannot
depend on any element of X, and the variance must be constant across observations, t. Part
(ii) is the no serial correlation assumption: the errors cannot be correlated across observa-
tions. Under random sampling, and in any other cross-sectional sampling schemes with
independent observations, part (ii) of Assumption E.4 automatically holds. For time series
applications, part (ii) rules out correlation in the errors over time (both conditional on X
and unconditionally).
Because of (E.13), we often say that u has a scalar variance-covariance matrix
when Assumption E.4 holds. We can now derive the variance-covariance matrix of the
OLS estimator.

THEOREM VARIANCE-COVARIANCE MATRIX OF THE OLS ESTIMATOR

E.2 Under Assumptions E.1 through E.4,

Var( ˆX) 5 2(X X)21. [E.14]

PROOF: From the last formula in equation (E.12), we have

Var( ˆX) 5 Var[(X X)21X uX] 5 (X X)21X [Var(uX)]X(X X)21.

Now, we use Assumption E.4 to get

Var( ˆX) 5 (X X)21X (2In)X(X X)21

5 2(X X)21X X(X X)21 5 2(X X)21.

Formula (E.14) means that the variance of bˆj (conditional on X) is obtained by multiply-
2 th
ing ​ by the j diagonal element of (X X)21. For the slope coefficients, we gave an
interpretable formula in equation (3.51). Equation (E.14) also tells us how to obtain the
covariance between any two OLS estimates: multiply 2 by the appropriate off-diagonal
element of (X X)21. In Chapter 4, we showed how to avoid explicitly finding covari-
ances for obtaining confidence intervals and hypothesis tests by appropriately rewriting
the model.

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 812 APPENDICES

The Gauss-Markov Theorem, in its full generality, can be proven.

THEOREM GAUSS-MARKOV THEOREM

E.3 Under Assumptions E.1 through E.4, ˆ is the best linear unbiased estimator.
PROOF: Any other linear estimator of can be written as
˜ 5 A y,
[E.15]
where A is an n (k 1 1) matrix. In order for ˜ to be unbiased conditional on X, A can
consist of nonrandom numbers and functions of X. (For example, A cannot be a function
of y.) To see what further restrictions on A are needed, write
˜ 5 A (X 1 u) 5 (A X) 1 A u.
[E.16]
Then,

E( ˜ X) 5 A X 1 E(A uX)

5 A X 1 A E(uX) because A is a function of X
5 A X because E(uX) 5 0.

For ˜ to be an unbiased estimator of , it must be true that E( ˜ X) 5 for all (k 1 1) 1

vectors , that is,

AX 5 for all (k 1 1) 1 vectors . [E.17]

Because A X is a (k 1 1) (k 1 1) matrix, (E.17) holds if, and only if, A X 5 Ik 1 1.
Equations (E.15) and (E.17) characterize the class of linear, unbiased estimators for .
Next, from (E.16), we have

Var( ˜ X) 5 A [Var(uX)]A 5 2A A,

by Assumption E.4. Therefore,

Var( ˜ X) 2 Var( ˆX) 5 2[A A 2 (X X)21]

5 2[A A 2 A X(X X)21X A] because A X 5 Ik 1 1
5 2A [In 2 X(X X)21X ]A
2A MA,
where M In 2 X(X X)21X . Because M is symmetric and idempotent, A MA is positive
semi-definite for any n (k 1 1) matrix A. This establishes that the OLS estimator ˆ is
BLUE. Why is this important? Let c be any (k 1 1) 1 vector and consider the linear
combination c 5 c0 b0 1 c1b1 1 … 1 ck bk, which is a scalar. The unbiased estimators
of c are c ˆ and c ˜. But

Var(c ˜X) 2 Var(c ˆX) 5 c [Var( ˜ X) 2 Var(b​

ˆX)]c  0,

because [Var( ˜ X) 2 Var(b​ ˆX)] is p.s.d. Therefore, when it is used for estimating any
linear combination of , OLS yields the smallest variance. In particular, Var( b ˆjX) 
Var( b̃jX) for any other linear, unbiased estimator of bj.

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 APPENDIX E The Linear Regression Model in Matrix Form 813

The unbiased estimator of the error variance 2 can be written as

​ ˆ2 5 u
​ ˆuˆ/(n 2 k 2 1),

which is the same as equation (3.56).

THEOREM UNBIASEDNESS OF 2
E.4 ˆ2 is unbiased for ​2: E(​
Under Assumptions E.1 through E.4, ​ ˆ2X) 5 ​2 for all ​2 . 0.
ˆ 5 y 2 Xˆ 5 y 2 X(X X)21X y 5 My 5 Mu, where M 5 In 2 X(X X)21X ,
PROOF: Write u
and the last equality follows because MX 5 0. Because M is symmetric and idempotent,

ˆu
u ˆ 5 u M Mu 5 u Mu.

Because u Mu is a scalar, it equals its trace. Therefore,

E(u MuX) 5 E[tr(u Mu)X] 5 E[tr(Muu )X]

5 tr[E(Muu |X)] 5 tr[ME(uu |X)]
5 tr(M2In) 5 2tr(M) 5 2(n 2 k 2 1).

The last equality follows from tr(M) 5 tr(In) 2 tr[X(X X)21X ] 5 n 2 tr[(X X)21X X] 5 n 2
tr (Ik 1 1) 5 n 2 (k 1 1)5 n 2 k 2 1. Therefore,

ˆ2X) 5 E(u MuX)/(n 2 k 2 1) 5 2.

E(​

E.3 Statistical Inference

When we add the final classical linear model assumption, ˆ has a multivariate normal
distribution, which leads to the t and F distributions for the standard test statistics covered
in Chapter 4.

Assumption E.5 Normality of Errors

Conditional on X, the ut are independent and identically distributed as Normal(0, 2).
Equivalently, u given X is distributed as multivariate normal with mean zero and variance-
covariance matrix 2In: u ~ Normal(0,2In).

Under Assumption E.5, each ut is independent of the explanatory variables for all t. In a
time series setting, this is essentially the strict exogeneity assumption.

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 814 APPENDICES

THEOREM 
NORMALITY OF b
E.5 Under the classical linear model Assumptions E.1 through E.5, ˆ conditional on
X is distributed as multivariate normal with mean and variance-covariance
matrix 2(X X)21.

Theorem E.5 is the basis for statistical inference involving . In fact, along with the prop-
erties of the chi-square, t, and F distributions that we summarized in Appendix D, we can
use Theorem E.5 to establish that t statistics have a t distribution under Assumptions E.1
through E.5 (under the null hypothesis) and likewise for F statistics. We illustrate with a
proof for the t statistics.

THEOREM DISTRIBUTION OF t STATISTIC

E.6 Under Assumptions E.1 through E.5,

ˆj 2 bj)/se( b
(b ˆj) ~ tn 2 k 2 1, j 5 0, 1, …, k.

PROOF: The proof requires several steps; the following statements are initially conditional on
ˆj 2 bj)/sd(b
X. First, by Theorem E.5, (b ˆj) 5 ​ c__jj , and cjj is the jth
ˆj) ~ Normal(0,1), where sd(b

diagonal element of (X X)21. Next, under Assumptions E.1 through E.5, conditional on X,

ˆ2/2 ~
(n 2 k 2 1) ​ 2
n 2 k 2 1. [E.18]

This follows because (n 2 k 2 1) ​ˆ2/​2 5 (u/) M(u/), where M is the n n symmetric,

idempotent matrix defined in Theorem E.4. But u/ ~ Normal(0,In) by Assumption E.5. It
follows from Property 1 for the chi-square distribution in Appendix D that (u/) M(u/) ~
2
n2k21 (because M has rank n 2 k 2 1).
We also need to show that ˆ and ​ ˆ2 are independent. But ˆ 5 1 (X X)21X u, and
2
ˆ 5 u Mu/(n 2 k 2 1). Now, [(X X) X ]M 5 0 because X M 5 0. It follows, from Prop-
​ 21

erty 5 of the multivariate normal distribution in Appendix D, that ˆ and Mu are indepen-
ˆ2 is a function of Mu, ˆ and ​
dent. Because ​ ˆ2 are also independent.

ˆj 2 bj)/se( b
(b ˆj) 5 [( b
ˆj 2 bj)/sd(b
ˆj)]/(​
ˆ2/2)1/2,

which is the ratio of a standard normal random variable and the square root of a n2 2 k 2 1 /
(n 2 k 2 1) random variable. We just showed that these are independent, so, by definition
of a t random variable, ( bˆj 2 bj )/se( b
ˆj ) has the tn 2 k 2 1 distribution. Because this distri-
bution does not depend on X, it is the unconditional distribution of ( b ˆj 2 bj)/se( b
ˆj) as well.

From this theorem, we can plug in any hypothesized value for bj and use the t statistic for
testing hypotheses, as usual.
Under Assumptions E.1 through E.5, we can compute what is known as the Cramer-
Rao lower bound for the variance-covariance matrix of unbiased estimators of (again
conditional on X) [see Greene (1997, Chapter 4)]. This can be shown to be 2(X X)21,

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 APPENDIX E The Linear Regression Model in Matrix Form 815

which is exactly the variance-covariance matrix of the OLS estimator. This implies that
ˆ is the minimum variance unbiased estimator of (conditional on X): Var( ˜ X) 2
Var( ˆX) is positive semi-definite for any other unbiased estimator ˜ ; we no longer have
to restrict our attention to estimators linear in y.
It is easy to show that the OLS estimator is in fact the maximum likelihood estimator
of under Assumption E.5. For each t, the distribution of yt given X is Normal(xt ,​2).
Because the yt are independent conditional on X, the likelihood function for the sample is
obtained from the product of the densities:
n
t51
(2 ​2)21/2exp[2(yt 2 xt )2/(22)],

where denotes product. Maximizing this function with respect to and 2 is the same
as maximizing its natural logarithm:
n

∑ [2(1/2)log(2
t51
2) 2 (yt 2 xt )2/(22)].

For obtaining ˆ, this is the same as minimizing

n
t51
(yt 2 xt )2—the division by 22 ∑
does not affect the optimization—which is just the problem that OLS solves. The estimator
of 2 that we have used, SSR/(n 2 k), turns out not to be the MLE of 2; the MLE is SSR/n,
which is a biased estimator. Because the unbiased estimator of 2 results in t and F statistics
with exact t and F distributions under the null, it is always used instead of the MLE.
That the OLS estimator is the MLE under Assumption E.5 implies an interesting
robustness property of the MLE based on the normal distribution. The reasoning is simple.
We know that the OLS estimator is unbiased under Assumptions E.1 to E.3; normality of
the errors is used nowhere in the proof, and neither is Assumption E.4. As the next section
shows, the OLS estimator is also consistent without normality, provided the law of large
numbers holds (as is widely true). These statistical properties of the OLS estimator imply
that the MLE based on the normal log-likelihood function is robust to distributional speci-
fication: the distribution can be (almost) anything and yet we still obtain a consistent (and,
under E.1 to E.3, unbiased) estimator. As discussed in Section 17.3, a maximum likeli-
hood estimator obtained without assuming the distribution is correct is often called a quasi-
maximum likelihood estimator (QMLE).
Generally, consistency of the MLE relies on having a correct distribution in order to
conclude that it is consistent for the parameters. We have just seen that the normal distribu-
tion is a notable exception. There are some other distributions that share this property, includ-
ing the Poisson distribution—as discussed in Section 17.3. Wooldridge (2010, Chapter 18)
discusses some other useful examples.

E.4 Some Asymptotic Analysis

The matrix approach to the multiple regression model can also make derivations of
asymptotic properties more concise. In fact, we can give general proofs of the claims in
Chapter 11.
We begin by proving the consistency result of Theorem 11.1. Recall that these
assumptions contain, as a special case, the assumptions for cross-sectional analysis under
random sampling.

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 816 APPENDICES

Proof of Theorem 11.1. As in Problem E.1 and using Assumption TS.1 , we write the
OLS estimator as

n n n n

 ∑ x x   ∑ x y  5  ∑ x x   ∑ x (x
21 21
ˆ5
t51
t t
t51
t t
t51
t t
t51
t t 1 ut) 
n n

∑x x  ∑x u 
21
5 1 t t t t [E.19]
t51 t51

n n

 ∑   ∑ x u .
21
5 1 n21 xt xt n21 t t
t51 t51

Now, by the law of large numbers,

n n
n21 ∑t51
xt xt →
p
and n21 ∑x u
t51
p
t t → 0, [E.20]

where A 5 E(xt xt) is a (k 1 1) (k 1 1) nonsingular matrix under Assumption TS.2 and

we have used the fact that E(xt ut) 5 0 under Assumption TS.3 . Now, we must use a ma-
trix version of Property PLIM.1 in Appendix C. Namely, because A is nonsingular,
n

 ∑ xx
21
p
n21 t t → A21. [E.21]
t51

[Wooldridge (2010, Chapter 3) contains a discussion of these kinds of convergence re-

sults.] It now follows from (E.19), (E.20), and (E.21) that

plim( ˆ) 5 1 A21  0 5 .

This completes the proof.

Next, we sketch a proof of the asymptotic normality result in Theorem 11.2.

Proof of Theorem 11.2. From equation (E.19), we can write

 ∑ x x  n ∑ x u 
21 n
__
n ( ˆ 2
21 21/2
)5 n t t t t
t51 t51
n
5 A21 n21/2  ∑ x u  1 o (1),
t51
t t p [E.22]

where the term “op(1)” is a remainder term that converges in probability to zero. This
term is equal to  n21 xt xt  2 A21  n21/2 t51 xt ut . The term in brackets con-
∑ ∑
n 21 n
t51
verges in probability to zero (by the same argument used in the proof of Theorem 11.1),
n
∑
while  n21/2 t51 xt ut  is bounded in probability because it converges to a multivariate
normal distribution by the central limit theorem. A well-known result in asymptotic theory
__
is that the product of such terms converges in probability to zero. Further,  n (ˆ 2 )
n
inherits its asymptotic distribution from A21  n21/2 t51 xt ut . See Wooldridge (2010, ∑
Chapter 3) for more details on the convergence results used in this proof.

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 APPENDIX E The Linear Regression Model in Matrix Form 817

By the central limit theorem, n21/2 t51 ∑ n

xt ut has an asymptotic normal distribution
__
with mean zero and, say, (k 1 1) (k 1 1) variance-covariance matrix B. Then, n ( ˆ 2 )
has an asymptotic multivariate normal distribution with mean zero and variance-
covariance matrix A21BA21. We now show that, under Assumptions TS.4 and TS.5 ,
B 5 2A. (The general expression is useful because it underlies heteroskedasticity-
robust and serial-correlation robust standard errors for OLS, of the kind discussed in
Chapter 12.) First, under Assumption TS.5 , xt ut and xs us are uncorrelated for t  s. Why?
Suppose s , t for concreteness. Then, by the law of iterated expectations, E(xt utusxs) 5
E[E(utusxt xs) xt xs] 5 E[E(utus xt xs)xt xs] 5 E[0  xt xs] 5 0. The zero covariances imply
that the variance of the sum is the sum of the variances. But Var(xt ut) 5 E(xt ututxt) 5
E(u2t xt xt). By the law of iterated expectations, E(u2t xt xt) 5 E[E(u2t xt xt xt)] 5 E[E(u2t xt)xt xt] 5
E(2xt xt) 5 2E(xt xt) 5 2A, where we use E(u2t xt) 5 2 under Assumptions TS.3 and
TS.4 . This shows that B 5 2A, and so, under Assumptions TS.1 to TS.5 , we have

__
 n ( ˆ 2 ) ~a Normal (0,2A21). [E.23]

This completes the proof.

From equation (E.23), we treat ˆ as if it is approximately normally distributed with
mean and variance-covariance matrix 2A21/n. The division by the sample size, n, is
expected here: the approximation to the variance-covariance matrix of ˆ shrinks to zero
ˆ2 5 SSR/(n 2 k 2 1),
at the rate 1/n. When we replace 2 with its consistent estimator, 
n
and replace A with its consistent estimator, n21 t51 xt xt 5X X/n, we obtain an estimator ∑
for the asymptotic variance of ˆ:

!
Avar(ˆ) 5 ​
ˆ 2(X X)21. [E.24]

Notice how the two divisions by n cancel, and the right-hand side of (E.24) is just the
usual way we estimate the variance matrix of the OLS estimator under the Gauss-Markov
assumptions. To summarize, we have shown that, under Assumptions TS.1 to TS.5 —
which contain MLR.1 to MLR.5 as special cases—the usual standard errors and t statistics
are asymptotically valid. It is perfectly legitimate to use the usual t distribution to obtain
critical values and p-values for testing a single hypothesis. Interestingly, in the general
setup of Chapter 11, assuming normality of the errors—say, ut given xt, ut21, xt21, ..., u1,
x1 is distributed as Normal(0,2)—does not necessarily help, as the t statistics would not
generally have exact t statistics under this kind of normality assumption. When we do not
assume strict exogeneity of the explanatory variables, exact distributional results are dif-
ficult, if not impossible, to obtain.
If we modify the argument above, we can derive a heteroskedasticity-robust, variance-
covariance matrix. The key is that we must estimate E(u2t xt xt) separately because this matrix
no longer equals 2E(xt xt). But, if the uˆt are the OLS residuals, a consistent estimator is

n
(n 2 k 2 1)21 ∑ uˆ x x ,
t51
2
t t t [E.25]

where the division by n 2 k 2 1 rather than n is a degrees of freedom adjustment that typi-
cally helps the finite sample properties of the estimator. When we use the expression in
equation (E.25), we obtain

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 818 APPENDICES

n
!
Avar(ˆ) 5 [n/(n 2 k 2 1)](X X)21  ∑ uˆ x x  (X X)
t51
2
t t t
21
. [E.26]

The square roots of the diagonal elements of this matrix are the same heteroskedasticity-
robust standard errors we obtained in Section 8.2 for the pure cross-sectional case. A
matrix extension of the serial correlation- (and heteroskedasticity-) robust standard er-
rors we obtained in Section 12.5 is also available, but the matrix that must replace (E.25)
is complicated because of the serial correlation. See, for example, Hamilton (1994,
Section 10.5).

Wald Statistics for Testing Multiple Hypotheses

Similar arguments can be used to obtain the asymptotic distribution of the Wald statistic
for testing multiple hypotheses. Let R be a q (k 1 1) matrix, with q  (k 1 1). Assume
that the q restrictions on the (k 1 1) 1 vector of parameters, , can be expressed as
H0R 5 r, where r is a q 1 vector of known constants. Under Assumptions TS.1 to
TS.5 , it can be shown that, under H0,
__ __
[n (R ˆ 2 r)] (2RA21R )21[n (R ˆ 2 r)] ~a 2
q, [E.27]

where A 5 E(xt xt), as in the proofs of Theorems 11.1 and 11.2. The intuition behind
__
equation (E.25) is simple. Because  n ( ˆ 2 ) is roughly distributed as Normal(0,2A21),
__ ˆ __
R[  n ( 2 )] 5  n R( ˆ 2 ) is approximately Normal(0,2RA21R ) by Property 3
of the multivariate normal distribution in Appendix D. Under H 0 , R 5 r, so
__
 n (R ˆ 2 r) ~ Normal(0, RA
2 21
R ) under H0. By Property 3 of the chi-square distri-
bution, z ( RA R ) z ~ q if z ~ Normal(0,2RA21R ). To obtain the final result
2 21 21 2

formally, we need to use an asymptotic version of this property, which can be found in
Wooldridge (2010, Chapter 3).
Given the result in (E.25), we obtain a computable statistic by replacing A and 2
with their consistent estimators; doing so does not change the asymptotic distribution. The
result is the so-called Wald statistic, which, after cancelling the sample sizes and doing a
little algebra, can be written as

W 5 (R ˆ 2 r) [R(X X)21R ]21(R ˆ 2 r)yˆ

​2. [E.28]

Under H0,W ~a 2q, where we recall that q is the number of restrictions being tested. If
​2 5 SSR/(n 2 k 2 1), it can be shown that W/q is exactly the F statistic we obtained in
ˆ
Chapter 4 for testing multiple linear restrictions. [See, for example, Greene (1997, Chap-
ter 7).] Therefore, under the classical linear model assumptions TS.1 to TS.6 in Chapter 10,
W/q has an exact Fq,n 2 k 2 1 distribution. Under Assumptions TS.1 to TS.5 , we only have
the asymptotic result in (E.26). Nevertheless, it is appropriate, and common, to treat the
usual F statistic as having an approximate Fq,n 2 k 2 1 distribution.
A Wald statistic that is robust to heteroskedasticity of unknown form is obtained by
using the matrix in (E.26) in place of ˆ ​2(X X)21, and similarly for a test statistic robust
to both heteroskedasticity and serial correlation. The robust versions of the test statistics
cannot be computed via sums of squared residuals or R-squareds from the restricted and
unrestricted regressions.

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 APPENDIX E The Linear Regression Model in Matrix Form 819

Summary
This appendix has provided a brief treatment of the linear regression model using matrix no-
tation. This material is included for more advanced classes that use matrix algebra, but it is
not needed to read the text. In effect, this appendix proves some of the results that we ei-
ther stated without proof, proved only in special cases, or proved through a more cumbersome
method of proof. Other topics—such as asymptotic properties, instrumental variables estima-
tion, and panel data models—can be given concise treatments using matrices. Advanced texts
in econometrics, including Davidson and MacKinnon (1993), Greene (1997), Hayashi (2000),
and Wooldridge (2010), can be consulted for details.

Key Terms
First Order Condition Scalar Variance-Covariance Wald Statistic
Matrix Notation Matrix Quasi-Maximum Likelihood
Minimum Variance Unbiased Variance-Covariance Matrix Estimator (QMLE)
Estimator of the OLS Estimator

Problems
1 Let xt be the 1 (k 1 1) vector of explanatory variables for observation t. Show that the
OLS estimator ˆ can be written as
n n

 ∑ x x   ∑ x y .
21
​ ˆ5 t t t t
t51 t51

Dividing each summation by n shows that ˆ is a function of sample averages.

2 Let ˆ be the (k 1 1) 1 vector of OLS estimates.
(i) Show that for any (k 1 1) 1 vector b, we can write the sum of squared residuals as

ˆu
SSR(b) 5 u ˆ 1 ( ˆ 2 b) X X( ˆ 2 b).

{Hint: Write (y 2 Xb) (y 2 Xb) 5 [u ˆ 1 X( ˆ 2 b)] [u

ˆ 1 X( ˆ 2 b)] and use the
ˆ 5 0.}
fact that X u
(ii) Explain how the expression for SSR(b) in part (i) proves that ˆ uniquely minimizes
SSR(b) over all possible values of b, assuming X has rank k 1 1.
3 Let ˆ be the OLS estimate from the regression of y on X. Let A be a (k 1 1)
(k 1 1) nonsingular matrix and define zt xtA, t 5 1, …, n. Therefore, zt is 1 (k 1 1)
and is a nonsingular linear combination of xt. Let Z be the n (k 1 1) matrix with rows zt.
Let ˜ denote the OLS estimate from a regression of y on Z.
(i) Show that ˜ 5 A21ˆ.
(ii) Let yˆt be the fitted values from the original regression and let y˜t be the fitted values
from regressing y on Z. Show that y˜t 5 yˆt, for all t 5 1, 2, …, n. How do the residuals
from the two regressions compare?
(iii) Show that the estimated variance matrix for ˜ is ​ ˆ2A21(X X)21A21 , where ​ ˆ2 is the
usual variance estimate from regressing y on X.

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.
 820 APPENDICES

ˆj be the OLS estimates from regressing yt on 1, xt1, …, xtk, and let the b̃j be
(iv) Let the b
the OLS estimates from the regression of yt on 1, a1xt1, …, ak xtk, where aj 0,
j 5 1, …, k. Use the results from part (i) to find the relationship between the b̃j
ˆj.
and the b
ˆj)/aj.
(v) Assuming the setup of part (iv), use part (iii) to show that se(b̃j) 5 se(b
(vi) Assuming the setup of part (iv), show that the absolute values of the t statistics for b̃j
ˆj are identical.
and b
4 Assume that the model y 5 X 1 u satisfies the Gauss-Markov assumptions, let G be
a (k 1 1) (k 1 1) nonsingular, nonrandom matrix, and define d 5 G , so that d is
also a (k 1 1) 1 vector. Let ˆ be the (k 1 1) 1 vector of OLS estimators and define dˆ
5 G ˆ as the OLS estimator of d.
(i) Show that E(dˆX) 5 d.
(ii) Find Var(dˆX) in terms of 2, X, and G.
(iii) Use Problem E.3 to verify that dˆ and the appropriate estimate of Var(dˆX) are
obtained from the regression of y on XG21.
(iv) Now, let c be a (k 1 1) 1 vector with at least one nonzero entry. For concreteness,
assume that ck 0. Define  5 c , so that  is a scalar. Define j 5 j, j 5 0, 1, ...,
k 2 1 and k 5 . Show how to define a (k 1 1) (k 1 1) nonsingular matrix G so
that d 5 G . (Hint: Each of the first k rows of G should contain k zeros and a one.
What is the last row?)
(v) Show that for the choice of G in part (iv),

 
1 0 0 . . . 0
0 1 0 . . . 0
.
G
21
5 . .
.
0 0 . . . 1 0
2c0 /ck 2c1/ck . . . 2ck21/ck 1/ck

ˆ and its standard error are ob-

Use this expression for G21 and part (iii) to conclude that ​
tained as the coefficient on xtk /ck in the regression of

yt on [1 2 (c0/ck)xtk], [xt1 2 (c1/ck)xtk], ..., [xt,k21 2 (ck21/ck)xtk], xtk/ck, t 5 1, ..., n.

This regression is exactly the one obtained by writing bk in terms of  and b0, b1, ..., bk21,
plugging the result into the original model, and rearranging. Therefore, we can formally
justify the trick we use throughout the text for obtaining the standard error of a linear com-
bination of parameters.
5 Assume that the model y 5 X 1 u satisfies the Gauss-Markov assumptions and let ˆ be
the OLS estimator of . Let Z 5 G(X) be an n (k 1 1) matrix function of X and assume
that Z X [a (k 1 1) (k 1 1) matrix] is nonsingular. Define a new estimator of by ˜ 5
(Z X)21Z y.
(i) Show that E( ˜ X) 5 , so that ˜ is also unbiased conditional on X.
(ii) Find Var( ˜ X). Make sure this is a symmetric, (k 1 1) (k 1 1) matrix that depends
on Z, X, and 2.
(iii) Which estimator do you prefer, ˆ or ˜ ? Explain.

Copyright 2012 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has
deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it.