Variance-Covariance Matrix

Assembled by Ben Danforth June 1, 2009

Covariance

Covariance measures the degree to which two variables change or vary together (i.e. co-vary). On the one hand, the covariance of two variables is positive if they vary together in the same direction relative to their expected values (i.e. if one variable moves above its expected value, then the other variable also moves above its expected value). On the other hand, if one variable tends to be above its expected value when the other is below its expected value, then the covariance between the two variables is negative. If there is no linear dependency between the two variables, then the covariance is 0. The covariance of two random variables, Xi and Xj , can be mathematically represented as cov (Xi , Xj ) = E [(Xi i )(Xj j )] (1) where i = E (Xi ) and j = E (Xj ).1 This relationship can be further generalized to a s) using matrix notation multivariate situation and for estimated coecients (i.e. ) = ( )( )T = E ( T ) T cov ( is a vector of estimated coecients and = where more explicitly represented as 1 , 1 ) cov ( 1 , 2 ) cov ( ) = cov (2 , 1 ) cov (2 , 2 ) cov ( ... ... k , 2 ) cov (k , 1 ) cov ( (2)

) = . This relationship can be E ( 1 , k ) ... cov ( 2 , k ) ... cov ( , ... ... k , k ) ... cov (

(3)

with k elements. where the diagonal of this matrix represents the variance of the vector Therefore, the matrix can be rewritten as 1 ) 1 , 2 ) ... cov ( 1 , k ) var( cov ( 2 ) ... cov ( 2 , k ) var( ) = cov (2 , 1 ) cov ( . (4) ... ... ... ... k , 1 ) cov ( k , 2 ) ... var( k ) cov ( This essentially represents the covariance or variance-covariance matrix.
1

Sometimes cov (Xi , Xj ) is denoted as i,j .

Deriving the Variance-Covariance Matrix

First, begin with the OLS estimator in matrix notation

Next, substitute y = X + u into the preceding equation to arrive at = (X T X )1 X T (X + u) Then distribute the rst set of terms = (X T X )1 X T X + (X T X )1 X T u And simplify, recalling that (X T X )1 (X T X ) = 1 = + (X T X )1 X T u Finally, subtract from both sides to get = (X T X )1 X T u (6)

Recalling Eq. 2, with substituted for ) = E ( )( )T cov ( Insert Eq. 6 into Eq. 7 and then reorder the terms (remember (AB )T = B T AT ) ) = E ((X T X )1 X T u)((X T X )1 X T u)T cov ( ) = E (X T X )1 X T uuT X (X T X )1 cov ( Take the expectation (the X s are non-stochastic, and it is assumed that E (uuT ) = 2 I ) ) = (X T X )1 X T E (uuT )X (X T X )1 cov ( ) = (X T X )1 X T 2 IX (X T X )1 cov ( And simplify, producing the nal equation for the variance-covariance matrix ) = 2 (X T X )1 cov ( (8) (7)

In this last equation, 2 is the homoskedastic variance of ui and (X T X )1 is the inverse matrix appearing in the OLS estimator. ) = 2 (X T X )1 cov (

Estimating the Variance-Covariance Matrix

To estimate the variance-covariance matrix, the variance is needed. Since the true variance is unknown, it must be estimated. An unbiased estimator of 2 for the k -th variable case is 2 = which is equivalent to 2 = u T u nk (10) u 2 i nk (9)

where u is the estimated residuals, n is the number of observations, and k the number of parameters being estimated. Although 2 can be computed directly from the estimated residuals, there is a second approach to calculating it. First, recall that T SS = ESS + RSS (11)

where T SS is the total sum of squares, ESS is the explained sum of squares, and RSS is the residual sum of squares. This equation can be rewritten as RSS = T SS ESS Next, remember that RSS = u T u 2 T SS = y T y nY T X T y T nY 2 ESS = (13) (14) (15) (12)

2 is known as the correction for mean. Therefore, substituting Eqs. 13-15 where the term nY into Eq. 12 produces T X T y T u T u = yT y (16)

Covariance vs. Correlation

Covariance and correlation are related but not equivalent statistical measures. In particular, the correlation of two variables, Xi and Xj , is their normalized covariance, which is dened as E [(Xi i )(Xj j )] , (17) i,j = i j where i,j is the correlation coecient, i = E (Xi ), j = E (Xj ), i is the standard deviation of Xi , and j is the standard deviation of Xj . Because the correlation is normalized, it is dimensionless (i.e. it is a pure number without a unit of measure). By contrast, covariance does have a unit of measurethe product of the units of two variables.

Excercise
n = 15 Y = 68.7333 = 5.5027 4.1535 3.3787 8.2903 y T y = 259651 1031 11370 XT y = 35981 10176 1. Compute the variance-covariance matrix. You can also compute R2 . 2. Use the variance-covariance matrix to make inferences about the estimated coecients. 3. Is there any evidence of violations of the OLS assumptions given the above information and your calculations? 4. The covariance matrix of the three variables of interest (X1 , X2 , X3 ) is X1 X2 X3 X1 X2 X3 53.5523 167.2238 4.1667 167.2238 522.9238 12.5238 4.16667 12.5238 12.3810

You are given the following pieces of information:

Given that the standard deviations between X1 and X2 are 7.3179 and 22.8675, respectively, what is the correlation between these two variables?