Math 19b: Linear Algebra with Probability Oliver Knill, Spring 2011

0.15

Lecture 22: Distributions

0.10
A random variable is a function from a probability space Ω to the real line R. There are two
important classes of random variables:

1) For discrete random variables, the random variable X takes a discrete set of values. This 0.05
means that the random variable takes values xk and the probabilities P[X = xk ] = pk add up to
1.

2) For continuous random variables, there is a probability density function f (x) such that 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
P[X ∈ [a, b]] = ab f (x) dx and −∞ f (x) dx = 1.
R R∞

3 The probability distribution on N = {0, 1, 2, ... }

λk e−λ
Discrete distributions P[X = k ] =
k!
is called the Poisson distribution. It is used to describe the number of radioactive decays
1 We throw a dice and assume that each side appears with the same probability. The random in a sample, the number of newborns with a certain defect. You show in the homework that
variable X which gives the number of eyes satisfies the mean and standard deviation is λ Poisson distribution is the most important distribution
on {0, 1, 2, 3, . . .}. It is a limiting case of Binomial distributions
1
P[X = k ] = .
6 4 An epidemiology example from Cliffs notes: the UHS sees X=10 pneumonia cases each
winter. Assuming independence and unchanged conditions, what is the probability of there
This is a discrete distribution called the uniform distribution on the set {1, 2, 3, 4, 5, 6 }.
being 20 cases of pneumonia this winter? We use the Poisson distribution with λ = 10 to
see P[X = 20] = 1020 e−10 /20! = 0.0018.
0.4

The Poisson distribution is the n → ∞ limit of the binomial distribution if we chose

0.3 for each n the probability p such that λ = np is fixed.

Proof. Setting p = λ/n gives

0.2
!
n
P[X = k] = pk (1 − p)n−k
k
n! λ λ
0.1 = ( )k (1 − )n−k
k!(n − k)! n n
n! λk λ λ
= ( k )( )(1 − )n (1 − )k .
n (n − k)! k! n n
1 2 3 4 5 6
This is a product of four factors. The first factor n(n − 1)...(n − k + 1)/nk converges to 1.
Leave the second factor aλk /k! as it is. The third factor converges to e−λ by the definition
2 Throw n coins for which head appears with probability p. Let X denote the number of of the exponential. The last factor (1 − nλ )k converges to 1 for n → ∞ since k is kept fixed.
heads. This random variable takes values 0, 1, 2, . . . , n and We see that P[X = k] → λk e−λ /k!.
!
n
P[X = k ] = pk (1 − p)n−k .
k
Continuous distributions
This is the Binomial distribution we know.
5 A random variable is called a uniform distribution if P[X ∈ [a, b]] = (b − a) for all 8 The probability density function
0 ≤ a < b ≤ b. We can realize this random variable on the probability space Ω = [a, b] with f (x) = λe−λx .
the function X(x) = x, where P[I] is the length of an interval I. The uniform distribution
is called the exponential distribution. The exponential distribution is the most natural
is the most natural distribution on a finite interval.
distribution on the positive real line.
√
6 The random variable X on [0, 1] where P[[a, b]] = b − a is given by X(x) = x. We have Remark. The statements ”most natural” can be made more precise. Given a subset X ⊂ R
√ of the real line and the mean and standard deviation we can look at the distribution f on X
P[X ∈ [a, b]] = P[ x ∈ [a, b]] = P[x ∈ [a2 , b2 ]] = b2 − a2 . for which the entropy − X f log(f ) dx is maximal. The uniform, exponential and normal
R

distributions extremize entropy on an interval, half line or real line. The reason why they
We have f (x) = 2x because ab f (x) dx = x2 |ba = b2 −a2 . The function f (x) is the probability
R
appear so often is that adding independent random variables increases entropy. If different
density function of the random variable. processes influence an experiment then the entropy becomes large. Nature tries to maximize
entropy. Thats why these distributions are ”natural”.

9 The distribution on the positive real axis with the density function

1 (log(x)−m)2
f (x) = √ e− 2
2πx2
is called the log normal distribution with mean m. Examples of quantities which have
log normal distribution is the size of a living tissue like like length or height of a population
or the size of cities. An other example is the blood pressure of adult humans. A quantity
which has a log normal distribution is a quantity which has a logarithm which is normally
distributed.

a b Homework due March 30, 2011

7 A random variable with normal distribution with mean 1 and standard deviation 1 has the
1 a) Find the mean of the exponential distribution.
probability density
1 2
b) Find the variance and standard deviation of the exponential distribution.
f (x) = √ e−x /2 . c) Find the entropy − 0∞ f (x) log(f (x)) dx in the case λ = 1.
R
2π
This is an example of a continuous distribution. The normal distribution is the most natural 2 Here is a special case of the Students t distribution
distribution on the real line.
2
f (x) = (1 + x2 )−2 .
π
a) Verify that it is a probability distribution.
b) Find the mean. (No computation needed, just look at the symmetry).
c) Find the standard deviation.
To compute the integrals in a),c), you can of course use a computer algebra system if
needed.

3 a) Verify that the Poisson distribution is a probability distribution:

P∞
k=0 P[X = k] = 1.
b) Find the mean m = ∞ k=0 kP[X = k].
P
2
c) Find the standard deviation ∞ k=0 (k − m) P[X = k].
P

a b