CS395T Computational Statistics with Application to Bioinformatics

Prof. William H. Press Spring Term, 2010 The University of Texas at Austin Unit 6: Multivariate Normal Distributions and Chi Square

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

(Let me explain where were going here)

Building up prerequisites to do a fairly sophisticated treatment of model fitting
Bayes parameter estimation p-value tail tests really understand multivariate normal and covariance really understand chi-square

Then, we get to appreciate the actual model fitting stuff

fitted parameters their uncertainty expressed in several different ways goodness-of-fit

And it will in turn be a nice platform for learning some other things
bootstrap resampling

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

Multivariate Normal Distributions Generalizes Normal (Gaussian) to M-dimensions Like 1-d Gaussian, completely defined by its mean and (co-)variance Mean is a M-vector, covariance is a M x M matrix

N (x|, ) =

1 T 1 1 exp[ ( x ) (x )] 2 M/ 2 1 / 2 (2 ) det()

The mean and covariance of r.v.s from this distribution are*

= hxi

= (x )(x )

In the one-dimensional case is the standard deviation, which can be visualized as error bars around the mean. In more than one dimension can be visualized as an error ellipsoid around the mean in a similar way.

1 = (x )T 1 (x )
*really?
The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

Because mean and covariance are easy to estimate from a data set, it is easy perhaps too easy to fit a multivariate normal distribution to data.

1 X = hxi xi N i

I.e., estimate by sample averages.

1 X T = (x )(x ) (xi )(xi )T N i

But back to really? The mean follows from the symmetry argument Z Z 1 T 1 M 1 0 = (x ) exp[ ( x ) ( x )] d x 2 (2)M/2 det()1/2 Its not obvious that the covariance in fact obtains from the definition of the multivariate Normal. One has to do the multidimensional (and tensor) integral: M2 = Z Z (x)(x)T 1 T 1 M 1 exp[ ( x ) ( x )] d x 2 (2)M/2 det()1/2

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

The only way I know how to do this integral is by trickery involving the Cholesky decomposition (square root of a positive definite matrix):
were setting to 0 for convenience

y p(y) = p(x) x

Jacobian determinant. The transformation law for multivariate probability distributions.

This is the distribution of N independent univariate Normals N(0,1)!

Ha!

(I dont know an elementary proof, i.e., without some matrix decomposition. Can you find one?)

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

Reduced dimension properties of multivariate normal

1 T 1 1 N (x|, ) = exp[ ( x ) (x )] 2 (2 )M/2 det()1/2

1. Any slice through a m.v.n. is a m.v.n (constraint or conditioning) 2. Any projection of a m.v.n. is a m.v.n (marginalization)

You can prove both assertions by completing the square in the exponential, producing an exponential in (only) the reduced dimension times an exponential in (only) the lost dimensions. Then the second exponential is either constant (slice case) or can be integrated over (projection case).

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

How to generate multivariate normal deviates N():

Cholesky:

= LLT y = {yi } N(0, 1)

Fill y with independent Normals: Transform: Proof:

x = Ly +
T

Thats it! x is the desired m.v.n.

T T (x )(x ) = (Ly)(Ly) T T T T = L(yy )L = L yy L = LLT =

Even easier: MATLAB has a built-in function mvnrnd(MU,SIGMA). But be sure you get a bunch of m.v.n.s all in one call, because it (probably) re-does the Cholesky decomposition on each call!
Notice that the proof never used Normality. You can fill y with anything with zero mean and variance one, and youll reproduce . But the result wont be Normal!
The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

yy

=1

So, easy operations are: 1. Fitting a multivariate normal to a set of points (just compute the sample mean and covariance!) 2. Sampling from the fitted m.v.n.
mu = mean([len1 len2]) sig = cov(len1,len2)

mu = 3.2844 3.2483 sig = 0.6125 0.2476

In MATLAB, for example, these are one-line operations.

0.2476 0.5458

Example:

rsamp = mvnrnd(mu,sig,1000);

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

A related, useful, Cholesky trick is to draw error ellipses (ellipsoids, )

= LLT
So, locus of points at 1 standard deviation is

1 = (x ) x = Lz +

(x )

So, if z is on the unit circle (sphere, ) then

1 L (x ) = 1

will be on the error ellipse.

my coding of this idea looks like this

function [x y] = errorellipse(mu,sigma,stdev,n) L = chol(sigma,'lower'); circle = [cos(2*pi*(0:n)/n); sin(2*pi*(0:n)/n)].*stdev; ellipse = L*circle + repmat(mu,[1,n+1]); x = ellipse(1,:); y = ellipse(2,:);

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

The distribution we have been looking at has some interesting biology in it!

file genestats.dat (on course web site) contains 20694 lines like this:
number of exons N N exon lengths N-1 intron lengths

ENST00000341866 17470 3262 0.00002 4 1290 349 1412 211 169 678 13361 <EOL> ENST00000314348 22078 1834 0.00001 7 100 166 113 178 165 262 850 5475 385 3273 1149 2070 7892 ENST00000313081 13858 1160 0.00001 6 496 150 107 85 151 171 2068 76 2063 674 7817 ENST00000298622 80000 6487 0.00001 24 135 498 216 120 147 132 36 60 129 129 84 63 99 99 54 66 69 78 204 66 73 1081 397 2452 12133 15737 1513 769 942 103 829 2272 1340 3058 327 2371 1361 471 2922 735 85 9218 1257 2247 897 822 12104

total length gene name

ignore for now

total length of exons

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

10

Log10 of size of 1st and 2nd introns for 1000 genes:

This is kind of fun, because its not just the usual featureless scatter plot
notice the hard edges this is biology!

log10(second intron length)

log10(first intron length) Is there a significant correlation here? If the first intron is long, does the second one also tend to be? Or is our eye being fooled by the non-Gaussian shape?

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

11

Biology: The hard lower bounds on intron length are because the intron has to fit around the big spliceosome machinery! Its all carefully arranged to allow exons of any length, even quite small. Why? Could the spliceosome have evolved to require a minimum exon length, too? Are we seeing chance early history, or selection?
credit: Alberts et al. Molecular Biology of the Cell

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

12

The covariance matrix is a more general idea than just for multivariate Normal. You can compute the covariances of any set of random variables. Its the generalizaton to M-dimensions of the (centered) second moment Var.

For multiple r.v.s, all the possible covariances form a (symmetric) matrix:

Notice that the diagonal elements are the variances of the individual variables.

The variance of any linear combination of r.v.s is a quadratic form in C :

This also shows that C is positive definite, so it can still be visualized as an ellipsoid in the space of the r.v.s., where the directions are the different linear combinations.
The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

13

The covariance matrix is closely related to the linear correlation matrix.

rij = p

Cij Cii Cjj

more often seen written out as

When the null hypothesis is that X and Y are independent r.v.s, then r is useful as a p-value statistic (test for correlation), because 1. For large numbers of data points N, it is normally distributed,

r N(0, N 1/2 ) so r N is a normal t-value

2. Even with small numbers of data points, if the underlying distribution is multivariate normal, there is a simple form for the pvalue (comes from a Student t distribution).

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

14

For the exon length data, we can easily now show that the correlation is highly significant.

r = sig ./ sqrt(diag(sig) * diag(sig)') tval = sqrt(numel(len1))*r

r = 1.0000 0.3843 tval = 31.6228 12.1511 rr = 1.0000 0.3843 p = 1.0000 0.0000 0.0000 1.0000
not clear why Matlab reports 1 on the diagonals. Id call it 0!

0.3843 1.0000 12.1511 31.6228

statistical significance of the correlation in standard deviations (but note: uses CLT) Matlab has built-ins

[rr p] = corrcoef(i1llen,i2llen)

0.3843 1.0000

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

15

Lets talk more about chi-square. Recall that a t-value is (by definition) a deviate from 2 is a statistic defined as the sum of the squares of n independent t-values.

2 =

X xi i 2
i

xi N(i , i )

is a distribution (special case of Gamma), defined as

The important theorem is that 2 is in fact distributed as Chisquare. Lets prove it.

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

16

Prove first the case of =1: Suppose and

1 1 x2 pX (x) = e 2 2

x N(0, 1)

y = x2

pY (y ) dy = 2pX (x) dx

So, pY (y) = y

1/2

pX (y

1/2

)=

1 1 2y e 2 y

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

17

To prove the general case for integer , compute the characteristic function

Since we already proved that =1 is the distribution of a single t2-value, this proves that the general case is the sum of t2-values.
The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

18

Question: What is the generalization of

2 =

X xi i 2
i

xi N(i , i )

to the case where the xis are normal, but not independent? I.e., x comes from a multivariate Normal distribution? Answer:

2 = (x )T 1 (x ),

x N(, )

Proof is one of those Cholesky things,

= LLT ,

show that y is product of independent N(0,1)s, as we did before, and that

Ly = x , X
2 yi

=y y=

The University of Texas at Austin, CS 395T, Spring 2010, Prof. William H. Press

19