Chi-Squared Distribution
Chi-Squared Distribution
Definitions
If Z 1, ..., Z k are independent, standard normal Notation or
random variables, then the sum of their squares,
Parameters (known as "degrees of
freedom")
Support if , otherwise
Mean
Median
Mode
Variance
Skewness
The chi-squared distribution has one parameter: a Ex.
positive integer k that specifies the number of kurtosis
degrees of freedom (the number of random
Entropy
variables being summed, Zi s).
Introduction MGF
It is also a component of the definition of the t-distribution and the F-distribution used in t-tests, analysis of
variance, and regression analysis.
The primary reason for which the chi-squared distribution is extensively used in hypothesis testing is its
relationship to the normal distribution. Many hypothesis tests use a test statistic, such as the t-statistic in a t-
test. For these hypothesis tests, as the sample size, n , increases, the sampling distribution of the test statistic
approaches the normal distribution (central limit theorem). Because the test statistic (such as t) is
asymptotically normally distributed, provided the sample size is sufficiently large, the distribution used for
hypothesis testing may be approximated by a normal distribution. Testing hypotheses using a normal
distribution is well understood and relatively easy. The simplest chi-squared distribution is the square of a
standard normal distribution. So wherever a normal distribution could be used for a hypothesis test, a chi-
squared distribution could be used.
Suppose that is a random variable sampled from the standard normal distribution, where the mean is
and the variance is : . Now, consider the random variable . The distribution of the
random variable is an example of a chi-squared distribution: . The subscript 1 indicates that
this particular chi-squared distribution is constructed from only 1 standard normal distribution. A chi-
squared distribution constructed by squaring a single standard normal distribution is said to have 1 degree
of freedom. Thus, as the sample size for a hypothesis test increases, the distribution of the test statistic
approaches a normal distribution. Just as extreme values of the normal distribution have low probability
(and give small p-values), extreme values of the chi-squared distribution have low probability.
An additional reason that the chi-squared distribution is widely used is that it turns up as the large sample
distribution of generalized likelihood ratio tests (LRT).[6] LRTs have several desirable properties; in
particular, simple LRTs commonly provide the highest power to reject the null hypothesis (Neyman–
Pearson lemma) and this leads also to optimality properties of generalised LRTs. However, the normal and
chi-squared approximations are only valid asymptotically. For this reason, it is preferable to use the t
distribution rather than the normal approximation or the chi-squared approximation for a small sample size.
Similarly, in analyses of contingency tables, the chi-squared approximation will be poor for a small sample
size, and it is preferable to use Fisher's exact test. Ramsey shows that the exact binomial test is always more
powerful than the normal approximation.[7]
Lancaster shows the connections among the binomial, normal, and chi-squared distributions, as follows.[8]
De Moivre and Laplace established that a binomial distribution could be approximated by a normal
distribution. Specifically they showed the asymptotic normality of the random variable
where is the observed number of successes in trials, where the probability of success is , and
.
The expression on the right is of the form that Karl Pearson would generalize to the form
where
In the case of a binomial outcome (flipping a coin), the binomial distribution may be approximated by a
normal distribution (for sufficiently large ). Because the square of a standard normal distribution is the chi-
squared distribution with one degree of freedom, the probability of a result such as 1 heads in 10 trials can
be approximated either by using the normal distribution directly, or the chi-squared distribution for the
normalised, squared difference between observed and expected value. However, many problems involve
more than the two possible outcomes of a binomial, and instead require 3 or more categories, which leads
to the multinomial distribution. Just as de Moivre and Laplace sought for and found the normal
approximation to the binomial, Pearson sought for and found a degenerate multivariate normal
approximation to the multinomial distribution (the numbers in each category add up to the total sample size,
which is considered fixed). Pearson showed that the chi-squared distribution arose from such a multivariate
normal approximation to the multinomial distribution, taking careful account of the statistical dependence
(negative correlations) between numbers of observations in different categories.[8]
where denotes the gamma function, which has closed-form values for integer .
For derivations of the pdf in the cases of one, two and degrees of freedom, see Proofs related to chi-
squared distribution.
where is the lower incomplete gamma function and is the regularized gamma function.
which can be easily derived by integrating directly. The integer recurrence of the
gamma function makes it easy to compute for other small, even .
Tables of the chi-squared cumulative distribution function are widely available and the function is included
in many spreadsheets and all statistical packages.
Letting , Chernoff bounds on the lower and upper tails of the CDF may be obtained.[9] For the
cases when (which include all of the cases when this CDF is less than half):
For another approximation for the CDF modeled after the cube of a Gaussian, see under Noncentral chi-
squared distribution.
Properties
Cochran's theorem
If are independent identically distributed (i.i.d.), standard normal random variables, then
where
and the all ones vector. has one eigenvector with eigenvalue , and eigenvectors
(all orthogonal to ) with eigenvalue , which can be chosen so that is an
orthogonal matrix. Since also , we have
Additivity
It follows from the definition of the chi-squared distribution that the sum of independent chi-squared
variables is also chi-squared distributed. Specifically, if are independent chi-squared variables
with , degrees of freedom, respectively, then is chi-squared distributed
with degrees of freedom.
Sample mean
The sample mean of i.i.d. chi-squared variables of degree is distributed according to a gamma
distribution with shape and scale parameters:
Asymptotically, given that for a scale parameter going to infinity, a Gamma distribution converges
towards a normal distribution with expectation and variance , the sample mean
converges towards:
Note that we would have obtained the same result invoking instead the central limit theorem, noting that for
each chi-squared variable of degree the expectation is , and its variance (and hence the variance of
the sample mean being ).
Entropy
The chi-squared distribution is the maximum entropy probability distribution for a random variate for
which and are fixed. Since the chi-squared is in the family of
gamma distributions, this can be derived by substituting appropriate values in the Expectation of the log
moment of gamma. For derivation from more basic principles, see the derivation in moment-generating
function of the sufficient statistic.
Noncentral moments
The moments about zero of a chi-squared distribution with degrees of freedom are given by[10][11]
Cumulants
The cumulants are readily obtained by a (formal) power series expansion of the logarithm of the
characteristic function:
Concentration
The chi-squared distribution exhibits strong concentration around its mean. The standard Laurent-
Massart[12] bounds are:
Asymptotic properties
variance [16] This is known as the Wilson–Hilferty transformation, see (18.24), p. 426
of Johnson.[4]
This normalizing transformation leads directly to the commonly used median
Related distributions
As , (normal distribution)
(noncentral chi-squared distribution with non-centrality parameter )
If then has the chi-squared distribution
If then
[17]
A chi-squared variable with degrees of freedom is defined as the sum of the squares of independent
standard normal random variables.
If is a -dimensional Gaussian random vector with mean vector and rank covariance matrix , then
is chi-squared distributed with degrees of freedom.
The sum of squares of statistically independent unit-variance Gaussian variables which do not have mean
zero yields a generalization of the chi-squared distribution called the noncentral chi-squared distribution.
If is a vector of i.i.d. standard normal random variables and is a symmetric, idempotent matrix
with rank , then the quadratic form is chi-square distributed with degrees of freedom.
If is a positive-semidefinite covariance matrix with strictly positive diagonal entries, then for
and a random -vector independent of such that and
then
[15]
The chi-squared distribution is also naturally related to other distributions arising from the Gaussian. In
particular,
statistically independent.
If and are statistically independent, then . If and
are not independent, then is not chi-square distributed.
Generalizations
The chi-squared distribution is obtained as the sum of the squares of k independent, zero-mean, unit-
variance Gaussian random variables. Generalizations of this distribution can be obtained by summing the
squares of other types of Gaussian random variables. Several such distributions are described below.
Linear combination
If are chi square random variables and , then a closed expression for the
distribution of is not known. It may be, however, approximated efficiently using the
Chi-squared distributions
The noncentral chi-squared distribution is obtained from the sum of the squares of independent Gaussian
random variables having unit variance and nonzero means.
The generalized chi-squared distribution is obtained from the quadratic form z'Az where z is a zero-mean
Gaussian vector having an arbitrary covariance matrix, and A is an arbitrary matrix.
using the rate parameterization of the gamma distribution (or using the scale
Because the exponential distribution is also a special case of the gamma distribution, we also have that if
, then is an exponential distribution.
The Erlang distribution is also a special case of the gamma distribution and thus we also have that if
with even , then is Erlang distributed with shape parameter and scale parameter .
Following are some of the most common situations in which the chi-squared distribution arises from a
Gaussian-distributed sample.
if are i.i.d. random variables, then where
Name Statistic
chi-squared distribution
chi distribution
Computational methods
The p-value is the probability of observing a test statistic at least as extreme in a chi-squared distribution.
Accordingly, since the cumulative distribution function (CDF) for the appropriate degrees of freedom (df)
gives the probability of having obtained a value less extreme than this point, subtracting the CDF value
from 1 gives the p-value. A low p-value, below the chosen significance level, indicates statistical
significance, i.e., sufficient evidence to reject the null hypothesis. A significance level of 0.05 is often used
as the cutoff between significant and non-significant results.
The table below gives a number of p-values matching to for the first 10 degrees of freedom.
Degrees of freedom (df) value[20]
1 0.004 0.02 0.06 0.15 0.46 1.07 1.64 2.71 3.84 6.63 10.83
2 0.10 0.21 0.45 0.71 1.39 2.41 3.22 4.61 5.99 9.21 13.82
3 0.35 0.58 1.01 1.42 2.37 3.66 4.64 6.25 7.81 11.34 16.27
4 0.71 1.06 1.65 2.20 3.36 4.88 5.99 7.78 9.49 13.28 18.47
5 1.14 1.61 2.34 3.00 4.35 6.06 7.29 9.24 11.07 15.09 20.52
6 1.63 2.20 3.07 3.83 5.35 7.23 8.56 10.64 12.59 16.81 22.46
7 2.17 2.83 3.82 4.67 6.35 8.38 9.80 12.02 14.07 18.48 24.32
8 2.73 3.49 4.59 5.53 7.34 9.52 11.03 13.36 15.51 20.09 26.12
9 3.32 4.17 5.38 6.39 8.34 10.66 12.24 14.68 16.92 21.67 27.88
10 3.94 4.87 6.18 7.27 9.34 11.78 13.44 15.99 18.31 23.21 29.59
p-value (probability) 0.95 0.90 0.80 0.70 0.50 0.30 0.20 0.10 0.05 0.01 0.001
These values can be calculated evaluating the quantile function (also known as "inverse CDF" or "ICDF")
of the chi-squared distribution;[21] e. g., the χ2 ICDF for p = 0.05 and df = 7 yields 2.1673 ≈ 2.17 as
in the table above, noticing that 1 – p is the p-value from the table.
History
This distribution was first described by the German geodesist and statistician Friedrich Robert Helmert in
papers of 1875–6,[22][23] where he computed the sampling distribution of the sample variance of a normal
population. Thus in German this was traditionally known as the Helmert'sche ("Helmertian") or "Helmert
distribution".
The distribution was independently rediscovered by the English mathematician Karl Pearson in the context
of goodness of fit, for which he developed his Pearson's chi-squared test, published in 1900, with computed
table of values published in (Elderton 1902), collected in (Pearson 1914, pp. xxxi–xxxiii, 26–28, Table
XII). The name "chi-square" ultimately derives from Pearson's shorthand for the exponent in a multivariate
normal distribution with the Greek letter Chi, writing −½χ2 for what would appear in modern notation as
−½xTΣ−1x (Σ being the covariance matrix).[24] The idea of a family of "chi-squared distributions",
however, is not due to Pearson but arose as a further development due to Fisher in the 1920s.[22]
See also
Mathematics
portal
Chi distribution
Scaled inverse chi-squared distribution
Gamma distribution
Generalized chi-squared distribution
Noncentral chi-squared distribution
Pearson's chi-squared test
Reduced chi-squared statistic
Wilks's lambda distribution
Modified half-normal distribution[25] with the pdf on is given as
References
1. M.A. Sanders. "Characteristic function of the central chi-square distribution" (https://web.arch
ive.org/web/20110715091705/http://www.planetmathematics.com/CentralChiDistr.pdf#)
(PDF). Archived from the original (http://www.planetmathematics.com/CentralChiDistr.pdf)
(PDF) on 2011-07-15. Retrieved 2009-03-06.
2. Abramowitz, Milton; Stegun, Irene Ann, eds. (1983) [June 1964]. "Chapter 26" (http://www.m
ath.ubc.ca/~cbm/aands/page_940.htm). Handbook of Mathematical Functions with
Formulas, Graphs, and Mathematical Tables. Applied Mathematics Series. Vol. 55 (Ninth
reprint with additional corrections of tenth original printing with corrections (December 1972);
first ed.). Washington D.C.; New York: United States Department of Commerce, National
Bureau of Standards; Dover Publications. p. 940. ISBN 978-0-486-61272-0. LCCN 64-
60036 (https://lccn.loc.gov/64-60036). MR 0167642 (https://mathscinet.ams.org/mathscinet-g
etitem?mr=0167642). LCCN 65-12253 (https://lccn.loc.gov/65012253).
3. NIST (2006). Engineering Statistics Handbook – Chi-Squared Distribution (http://www.itl.nis
t.gov/div898/handbook/eda/section3/eda3666.htm)
4. Johnson, N. L.; Kotz, S.; Balakrishnan, N. (1994). "Chi-Square Distributions including Chi
and Rayleigh". Continuous Univariate Distributions. Vol. 1 (Second ed.). John Wiley and
Sons. pp. 415–493. ISBN 978-0-471-58495-7.
5. Mood, Alexander; Graybill, Franklin A.; Boes, Duane C. (1974). Introduction to the Theory of
Statistics (Third ed.). McGraw-Hill. pp. 241–246. ISBN 978-0-07-042864-5.
6. Westfall, Peter H. (2013). Understanding Advanced Statistical Methods. Boca Raton, FL:
CRC Press. ISBN 978-1-4665-1210-8.
7. Ramsey, PH (1988). "Evaluating the Normal Approximation to the Binomial Test". Journal of
Educational Statistics. 13 (2): 173–82. doi:10.2307/1164752 (https://doi.org/10.2307%2F116
4752). JSTOR 1164752 (https://www.jstor.org/stable/1164752).
8. Lancaster, H.O. (1969), The Chi-squared Distribution, Wiley
9. Dasgupta, Sanjoy D. A.; Gupta, Anupam K. (January 2003). "An Elementary Proof of a
Theorem of Johnson and Lindenstrauss" (http://cseweb.ucsd.edu/~dasgupta/papers/jl.pdf)
(PDF). Random Structures and Algorithms. 22 (1): 60–65. doi:10.1002/rsa.10073 (https://doi.
org/10.1002%2Frsa.10073). S2CID 10327785 (https://api.semanticscholar.org/CorpusID:10
327785). Retrieved 2012-05-01.
10. Chi-squared distribution (http://mathworld.wolfram.com/Chi-SquaredDistribution.html), from
MathWorld, retrieved Feb. 11, 2009
11. M. K. Simon, Probability Distributions Involving Gaussian Random Variables, New York:
Springer, 2002, eq. (2.35), ISBN 978-0-387-34657-1
12. Laurent, B.; Massart, P. (2000-10-01). "Adaptive estimation of a quadratic functional by
model selection" (https://dx.doi.org/10.1214/aos/1015957395). The Annals of Statistics. 28
(5). doi:10.1214/aos/1015957395 (https://doi.org/10.1214%2Faos%2F1015957395).
ISSN 0090-5364 (https://www.worldcat.org/issn/0090-5364). S2CID 116945590 (https://api.s
emanticscholar.org/CorpusID:116945590).
13. Box, Hunter and Hunter (1978). Statistics for experimenters (https://archive.org/details/statisti
csforexp00geor/page/118). Wiley. p. 118 (https://archive.org/details/statisticsforexp00geor/pa
ge/118). ISBN 978-0-471-09315-2.
14. Bartlett, M. S.; Kendall, D. G. (1946). "The Statistical Analysis of Variance-Heterogeneity and
the Logarithmic Transformation". Supplement to the Journal of the Royal Statistical Society.
8 (1): 128–138. doi:10.2307/2983618 (https://doi.org/10.2307%2F2983618).
JSTOR 2983618 (https://www.jstor.org/stable/2983618).
15. Pillai, Natesh S. (2016). "An unexpected encounter with Cauchy and Lévy". Annals of
Statistics. 44 (5): 2089–2097. arXiv:1505.01957 (https://arxiv.org/abs/1505.01957).
doi:10.1214/15-aos1407 (https://doi.org/10.1214%2F15-aos1407). S2CID 31582370 (https://
api.semanticscholar.org/CorpusID:31582370).
16. Wilson, E. B.; Hilferty, M. M. (1931). "The distribution of chi-squared" (https://www.ncbi.nlm.ni
h.gov/pmc/articles/PMC1076144). Proc. Natl. Acad. Sci. USA. 17 (12): 684–688.
Bibcode:1931PNAS...17..684W (https://ui.adsabs.harvard.edu/abs/1931PNAS...17..684W).
doi:10.1073/pnas.17.12.684 (https://doi.org/10.1073%2Fpnas.17.12.684). PMC 1076144 (htt
ps://www.ncbi.nlm.nih.gov/pmc/articles/PMC1076144). PMID 16577411 (https://pubmed.ncb
i.nlm.nih.gov/16577411).
17. Bäckström, T.; Fischer, J. (January 2018). "Fast Randomization for Distributed Low-Bitrate
Coding of Speech and Audio" (https://aaltodoc.aalto.fi/handle/123456789/33466).
IEEE/ACM Transactions on Audio, Speech, and Language Processing. 26 (1): 19–30.
doi:10.1109/TASLP.2017.2757601 (https://doi.org/10.1109%2FTASLP.2017.2757601).
S2CID 19777585 (https://api.semanticscholar.org/CorpusID:19777585).
18. Bausch, J. (2013). "On the Efficient Calculation of a Linear Combination of Chi-Square
Random Variables with an Application in Counting String Vacua". J. Phys. A: Math. Theor.
46 (50): 505202. arXiv:1208.2691 (https://arxiv.org/abs/1208.2691).
Bibcode:2013JPhA...46X5202B (https://ui.adsabs.harvard.edu/abs/2013JPhA...46X5202B).
doi:10.1088/1751-8113/46/50/505202 (https://doi.org/10.1088%2F1751-8113%2F46%2F5
0%2F505202). S2CID 119721108 (https://api.semanticscholar.org/CorpusID:119721108).
19. den Dekker A. J., Sijbers J., (2014) "Data distributions in magnetic resonance images: a
review", Physica Medica, [1] (https://dx.doi.org/10.1016/j.ejmp.2014.05.002)
20. Chi-Squared Test (http://www2.lv.psu.edu/jxm57/irp/chisquar.html) Table B.2. Dr. Jacqueline
S. McLaughlin at The Pennsylvania State University. In turn citing: R. A. Fisher and F. Yates,
Statistical Tables for Biological Agricultural and Medical Research, 6th ed., Table IV. Two
values have been corrected, 7.82 with 7.81 and 4.60 with 4.61
21. "Chi-squared Distribution | R Tutorial" (https://www.r-tutor.com/elementary-statistics/probabili
ty-distributions/chi-squared-distribution). www.r-tutor.com.
22. Hald 1998, pp. 633–692, 27. Sampling Distributions under Normality.
23. F. R. Helmert, "Ueber die Wahrscheinlichkeit der Potenzsummen der Beobachtungsfehler
und über einige damit im Zusammenhange stehende Fragen (http://gdz.sub.uni-goettingen.d
e/dms/load/img/?PPN=PPN599415665_0021&DMDID=DMDLOG_0018)", Zeitschrift für
Mathematik und Physik 21 (http://gdz.sub.uni-goettingen.de/dms/load/toc/?PPN=PPN59941
5665_0021), 1876, pp. 192–219
24. R. L. Plackett, Karl Pearson and the Chi-Squared Test, International Statistical Review,
1983, 61f. (https://www.jstor.org/stable/1402731?seq=3) See also Jeff Miller, Earliest Known
Uses of Some of the Words of Mathematics (http://jeff560.tripod.com/c.html).
25. Sun, Jingchao; Kong, Maiying; Pal, Subhadip (22 June 2021). "The Modified-Half-Normal
distribution: Properties and an efficient sampling scheme" (https://www.tandfonline.com/doi/
abs/10.1080/03610926.2021.1934700?journalCode=lsta20). Communications in Statistics -
Theory and Methods. 52 (5): 1591–1613. doi:10.1080/03610926.2021.1934700 (https://doi.o
rg/10.1080%2F03610926.2021.1934700). ISSN 0361-0926 (https://www.worldcat.org/issn/0
361-0926). S2CID 237919587 (https://api.semanticscholar.org/CorpusID:237919587).
Further reading
Hald, Anders (1998). A history of mathematical statistics from 1750 to 1930. New York:
Wiley. ISBN 978-0-471-17912-2.
Elderton, William Palin (1902). "Tables for Testing the Goodness of Fit of Theory to
Observation" (https://zenodo.org/record/1431595). Biometrika. 1 (2): 155–163.
doi:10.1093/biomet/1.2.155 (https://doi.org/10.1093%2Fbiomet%2F1.2.155).
"Chi-squared distribution" (https://www.encyclopediaofmath.org/index.php?title=Chi-squared
_distribution), Encyclopedia of Mathematics, EMS Press, 2001 [1994]
Pearson, Karl (1914). "On the probability that two independent distributions of frequency are
really samples of the same population, with special reference to recent work on the identity
of Trypanosome strains". Biometrika. 10: 85–154. doi:10.1093/biomet/10.1.85 (https://doi.or
g/10.1093%2Fbiomet%2F10.1.85).
External links
Earliest Uses of Some of the Words of Mathematics: entry on Chi squared has a brief history
(https://jeff560.tripod.com/c.html)
Course notes on Chi-Squared Goodness of Fit Testing (http://www.stat.yale.edu/Courses/19
97-98/101/chigf.htm) from Yale University Stats 101 class.
Mathematica demonstration showing the chi-squared sampling distribution of various
statistics, e. g. Σx², for a normal population (https://demonstrations.wolfram.com/StatisticsAss
ociatedWithNormalSamples/)
Simple algorithm for approximating cdf and inverse cdf for the chi-squared distribution with a
pocket calculator (https://www.jstor.org/stable/2348373)
Values of the Chi-squared distribution (https://www.medcalc.org/manual/chi-square-table.ph
p)