1

SETS, SAMPLE SPACES AND EVENTS

There are no little events in life.

– AMELIA E. BARR, All the Days of My Life (1913)

Mathematical Models
What is a mathematical model?

 
 
 
Observation

Application

 


Deterministic models

For specified values of an experiment’s input parameters, the result is a known

constant.
Example: V = IR
Established experimentally by George Simon Ohm in 18271

Nondeterministic models

Also called probabilistic models or stochastic models.

Examples:
1
To place this date in perspective, University at Buffalo was founded in 1846.

1
 • the number of α-particles emitted from a piece of radioactive material in one
minute.
• the outcome of a single toss of a fair die.
• the number of students who will earn an A in EAS305.

Sets
Basic definitions

We will need to use set theory to describe possible outcomes and scenarios in our
model of uncertainty.
Definition 1.1. A set is a collection of elements.

Notation: Sets are denoted by capital letters (A, B, Ω, . . .) while their

elements are denoted by small letters (a, b, ω, . . .). If x is an element of
a set A, we write x ∈ A. If x is not an element of A, we write x ∈ A.

A set can be described

• by listing the set’s elements

A = {1, 2, 3, 4}

• by describing the set in words

“A is the set of all real numbers between 0 and 1, inclusive.”
• by using the notation {ω : specification for ω}

A = {x : 0 ≤ x ≤ 1}

or
A = {0 ≤ x ≤ 1}

There are two sets of special importance:

Definition 1.2. The universe is the set containing all points under consideration
and is denoted by Ω.
Question: What is the difference (if any) between Ø and {Ø}?
Definition 1.3. The empty set (the set containing no elements) is denoted by Ø.
 Definition 1.4. If every point of set A belongs to set B , then we say that A is a
subset of B (B is a superset of A). We write

A⊆B B⊇A

Now we have a way to say A is identical to B . . .

Definition 1.5. A = B if and only if A ⊆ B and B ⊆ A.
Example: Let Ω = R, the set of all real numbers. Define the sets

A = {x : x2 + x − 2 = 0}
B = {x : (x − 3)(x2 + x − 2) = 0}
C = {−2, 1, 3}

Then A ⊆ B and B = C .
Definition 1.6. A is a proper subset of B if A ⊆ B and A = B . We write
A⊆/ B.
Note: Some authors use the symbol ⊂ for subset and other use ⊂ for proper subset.
We avoid the ambiguity here by using ⊆ for subset and ⊆/ for proper subset.

Venn diagrams

Set operations

Complementation

Ac = {ω ∈ Ω : ω ∈ A}
 Example: Here is the region representing Ac in a Venn diagram:
Ω

A Ac

Union

A ∪ B = {ω ∈ Ω : ω ∈ A or ω ∈ B (or both)}

Example: Consider the following Venn diagram:

A B

then the region representing A ∪ B is:

Intersection

A ∩ B = {ω ∈ Ω : ω ∈ A and ω ∈ B}
 Example: Using the same Venn diagram as above the region representing A ∩ B
is given by:

Note that some textbooks use the notation AB instead of A ∩ B to denote the
intersection of two sets.
Definition 1.7. Two sets A and B are disjoint (or mutually exclusive) if A∩B =
Ø.
These basic operations can be extended to any finite number of sets.

A ∪ B ∪ C = A ∪ (B ∪ C) = (A ∪ B) ∪ C

and
A ∩ B ∩ C = A ∩ (B ∩ C) = (A ∩ B) ∩ C

You can show that

(a) A ∪ B = B ∪ A
(b) A ∩ B = B ∩ A
(c) A ∪ (B ∪ C) = (A ∪ B) ∪ C
(d) A ∩ (B ∩ C) = (A ∩ B) ∩ C
Note: (a) and (b) are the commutative laws
Note: (c) and (d) are the associative laws

Set identities

(e) A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C)
(f) A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C)
(g) A ∩ Ø = Ø
 (h) A ∪ Ø = A
(i) (A ∪ B)c = Ac ∩ B c
(j) (A ∩ B)c = Ac ∪ B c
(k) (Ac )c = A

Note: (e) and (f) are called the Distributive Laws

Note: (i) and (j) are called DeMorgan’s Laws

Proving statements about sets

Venn diagrams can only illustrate set operations and basic results. You cannot
prove a true mathematical statement regarding sets using Venn diagrams. But they
can be used to disprove a false mathematical statement.
Here is an example of a simple set theorem and its proof.
Theorem 1.1. If A ⊆ B and B ⊆ C , then A ⊆ C .
Proof.
Discussion Reasons
(1) If x ∈ A then x ∈ B Definition of A ⊆ B
(2) Since x ∈ B then x ∈ C Definition of B ⊆ C
(3) If x ∈ A then x ∈ C By statements (1) and (2)
(4) Therefore A ⊆ C Definition of A ⊆ C

Note: A proof is often concluded with the symbol (as in the above proof) or the
letters QED.
Many mathematicians prefer to write proofs in paragraph form. For example, the
proof of Theorem 1.1 would become:
Proof. Choose an x ∈ A. Since A ⊆ B , x ∈ A implies x ∈ B , from the definition
of subset. Furthermore, since B ⊆ C , x ∈ B implies x ∈ C . Therefore, every
element x ∈ A is also an element of C . Hence A ⊆ C .
A proof of the distributive law for sets

Here is a proof of one of the distributive laws for sets. The proof is from a textbook
on set theory by Flora Dinkines2
Theorem 1.2. A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C).
Proof.

Discussion Reasons
(1) If x ∈ A ∩ (B ∪ C), then Definition of ∩.
x ∈ A and x ∈ B ∪ C .
(2) Since x ∈ B ∪ C , then Definition of ∪.
x ∈ B or x ∈ C .
(3) Therefore x ∈ A and x ∈ B , By (1) and (2).
or x ∈ A and x ∈ C .
(4) Hence x ∈ A ∩ B or Definition of ∩.
x ∈ A ∩ C.
(5) x ∈ (A ∩ B) ∪ (A ∩ C). Definition of ∪.

Therefore every element of A ∩ (B ∪ C) is also an element of (A ∩ B) ∪ (A ∩ C),

giving us
A ∩ (B ∪ C) ⊆ (A ∩ B) ∪ (A ∩ C).

We now prove the inclusion in the opposite direction:

2
Dinkines, F., Elementary Theory of Sets, Meredith Publishing, 1964.
 (6) If y ∈ (A ∩ B) ∪ (A ∩ C), Definition of ∪.
then y ∈ (A ∩ B) or
y ∈ (A ∩ C).
(7) If y ∈ A ∩ B then Definition of ∩.
y ∈ A and y ∈ B .
(8) If y ∈ A ∩ C then Same as (7).
y ∈ A and y ∈ C .
(9) In either case (7) or (8), Statements (7) and (8)
y ∈ A and y is an
element of one of the
sets B or C .
(10) Therefore y ∈ A and Since y is in B or C ,
y ∈ B ∪ C. it is in the union.
(11) Therefore y ∈ A ∩ (B ∪ C). Definition of ∩.

Therefore every element of (A ∩ B) ∪ (A ∩ C) is also an element of A ∩ (B ∪ C),

giving us
A ∩ (B ∪ C) ⊇ (A ∩ B) ∪ (A ∩ C).

Therefore, using the definition of set equality, we have

A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C).

Elementary sets

Let Ω be a universe with three subsets A, B , and C . Consider the following subsets
of Ω:
A B C
0 0 0 Ac ∩ B c ∩ C c = S0
0 0 1 Ac ∩ B c ∩ C = S1
0 1 0 Ac ∩ B ∩ C c = S2
0 1 1 Ac ∩ B ∩ C = S3
1 0 0 A ∩ Bc ∩ C c = S4
1 0 1 A∩B ∩C c = S5
1 1 0 A ∩ B ∩ Cc = S6
1 1 1 A∩B∩C = S7
The sets S0 , S1 , . . . , S7 are called the elementary sets of A, B and C .
 Ω

B
A

A Bc Cc A B Cc Ac B Cc

A B C
Ac B C
A Bc C

Ac Bc Cc Ac Bc C C

Figure 1.1: An example of elementary sets for three subsets

For example, the binary sequence (0,1,1) corresponds to S3 = Ac ∩ B ∩ C . This

is the subset of elements that are
0: not in A and
1: in B and
1: in C
Figure 1.1 illustrates all of the 23 = 8 elementary sets for A, B and C using a Venn
diagram.
Elementary sets have two very useful properties:
• Elementary sets are pairwise disjoint. That is, Si ∩ Sj = Ø for any i = j .
• Any set operation involving A, B , and C can always be represented as a
union of the elementary sets for A, B and C . For example
(A ∪ B) ∩ C = S3 ∪ S5 ∪ S7

Cardinality of sets

The cardinality of a set A (denoted by |A|) is the number of elements in A.

For some (but not all) experiments, we have
 • |Ω| is finite.
• Each of the outcomes of the experiment are equally likely.
In such cases, it is important to be able to enumerate (i.e., count) the number of
elements of subsets of Ω.

Sample spaces and events

Using set notation, we can now introduce the first components of our probability
model.
Definition 1.8. A sample space, Ω, is the set of all possible outcomes of an
experiment.
Definition 1.9. An event is any subset of a sample space.
Example: Consider the experiment of rolling a die once.
3 What should be Ω for this experiment?
 
Ω=

If you thought the answer was Ω = {1, 2, 3, 4, 5, 6}, then you

just (re)invented the concept of a random variable. This will be
discussed in a later chapter.
3 Identify the subsets of Ω that represent the following events:
1. The die turns up even.
 
A=

2. The die turns up less than 5.

 
B=

3. The die turns up 6.  

C=

Example: Consider the experiment of burning a light bulb until it burns out. The
outcome of the experiment is the life length of the bulb.
 3 What should be Ω for this experiment?

Ω = {x ∈ R : x ≥ 0}

For the lack of a better representation, we resorted to assigning a

nonnegative real number to each possible outcome of the experi-
ment.
3 Identify the subsets of Ω that represent the following events:
1. The bulb burns out within one and one-half hours.

A = {x ∈ R : 0 ≤ x < 1.5}

2. The bulb lasts at least 5 hours.

B = {x ∈ R : x ≥ 5}

3. The bulb lasts exactly 24 hours.

C = {24}

Sets of events

It is sometimes useful to be able to talk about the set of all possible events F. The
set F is very different than the set of all possible outcomes, Ω. Since an event is a
set, the set of all events is really a set of sets. Each element of F is a subset of Ω.
For example, suppose we toss a two-sided coin twice. The sample space is

Ω = {HH, HT, T H, T T }.

If you were asked to list all of the possible events for this experiment, you would
need to list all possible subsets of Ω, namely

A1 = {{HH}}
A2 = {{HT }}
A3 = {{T H}}
A4 = {{T T }}
A5 = {{HH}, {HT }}
A6 = {{HH}, {T H}}
 A7 = {{HH}, {T T }}
A8 = {{HT }, {T H}}
A9 = {{HT }, {T T }}
A10 = {{T H}, {T T }}
A11 = {{HH}, {HT }, {T H}}
A12 = {{HH}, {HT }, {T T }}
A13 = {{HH}, {T H}, {T T }}
A14 = {{HT }, {T H}, {T T }}
A15 = {{HH}, {HT }, {T H}, {T T }} = Ω
A16 = {} = Ø

In this case
F = {A1 , A2 , . . . , A16 }

Definition 1.10. For a given set A, the set of all subsets of A is called the power
set of A, and is denoted by 2A .
Theorem 1.3. If A is a finite set, then |2A | = 2|A|
Proof. Left to reader.
If Ω is a sample space, then we often use the set of all possible events F = 2Ω .
When Ω contains a finite number of outcomes, this often works well. However, for
some experiments, Ω contains an infinite number of outcomes (either countable or
uncountable) and F = 2Ω is much too large. In those cases, we select a smaller
collection of sets, F, to represent those events we actually need to consider. How
to properly select a smaller F, is found in many graduate level probability courses.

Self-Test Exercises for Chapter 1

For each of the following multiple-choice questions, choose the best response
among those provided. Answers can be found in Appendix B.
S1.1 The event that corresponds to the statement, "at least one of the events A, B
and C occurs," is
(A) A ∪ B ∪ C
(B) A ∩ B ∩ C
(C) (A ∩ B ∩ C)c
 (D) (A ∪ B ∪ C)c
(E) none of the above
S1.2 The event that corresponds to the statement that exactly one of the events A,
B or C occurs is
(A) (A ∩ B ∩ C)
(B) (A ∩ B ∩ C)c
(C) (A ∪ B ∪ C)
(D) (A ∩ B c ∩ C c ) ∪ (Ac ∩ B ∩ C c ) ∪ (Ac ∩ B c ∩ C)
(E) none of the above.
S1.3 The set (A ∩ B) ∪ (A ∩ B c ) is equivalent to
(A) Ø
(B) Ω
(C) A
(D) B
(E) none of the above.
S1.4 The set (A ∩ B) ∪ A is equivalent to
(A) (A ∪ B) ∩ A
(B) B
(C) A ∩ B
(D) Ø
(E) none of the above.
S1.5 For any sets A and B ,
(A) (A ∩ B) ⊆ B
(B) A ⊆ (A ∪ B)
(C) Ac ⊆ (A ∩ B)c
(D) all of the above are true.
 (E) none of the above are true.
S1.6 For any events A and B , the event (A ∪ B c ) ∩ B equals
(A) Ø
(B) Ω
(C) A ∪ B
(D) A ∩ B
(E) none of the above.
S1.7 If A ∩ B = Ø then (Ac ∪ B c )c equals
(A) Ø
(B) A
(C) B
(D) Ω
(E) none of the above.
2
ENUMERATION

Do not count your chickens before they are hatched.

– AESOP, Fables (600 BC)

This section is based on Chapter 3 of a text by K. L. Chung1

So you think you know how to count? Consider the following problem: Suppose
you are the chief engineer at the Ellicott Dominoes domino factory. Dominoes are
rectangular tiles used in some parlor games and they look like Figure 2.1. Typically,

Figure 2.1: A single domino

dominoes have each half of a tile painted with 1 to 9 dots (called “pips”) or a blank
for a total of 10 possible choices.
Of course, a domino painted (1,4) can be turned 180 degrees to look like a domino
painted (4,1). So there is no need to have one assembly line producing (1,4)
dominoes and another producing (4,1) dominoes.2 How many different dominoes
does your factory actually produce? (Try this problem on your friends and see how
many different answers you get.)
1
Chung, K. L., Elementary probability theory with stochastic processes, Springer-Verlag, New
York, 1975.
2
Just have a person at the end of the assembly line rotating half of the (4,1) the dominoes by 180
degrees.

19
 In this chapter, we will present a systematic way to approach counting problems.
We will classify counting problems into four categories, and solve each type. In
practice, most counting problems can be approached using one or a combination
of these four basic types.

Fundamental Rule
The following rule forms the foundation for solving any counting problem:
Multiplication Principle: Suppose there are several multiple choices to be made.
There are m1 alternatives for the first choice, m2 alternatives for the second, m3
for the third, etc. The total number of alternatives for the whole set of choices is
equal to
(m1 )(m2 )(m3 ) · · · .

Example: Suppose a engineering class has five women and four men. You would
like to select a research team consisting of one man and one woman. There are
four choices for selecting the man and five choices for selecting the woman. The
total number of possible research teams is, therefore, 4 · 5 = 20

Sampling
The Urn Model: Imagine an urn that contains n balls marked 1 to n. There are
four disciplines under which the balls can be drawn from the urn and examined.
I. Sampling with replacement and with ordering.
A total of m balls are drawn sequentially. After a ball is drawn, it is replaced
in the urn and may be drawn again. The order of the balls is noted. Hence the
sequence 1, 3, 2, 3 is distinguished from 3, 3, 1, 2. Since there are n possible
ways to draw the first ball, n possible ways to draw the second, etc., the total
number of possible sequences is

nm .

Example: Suppose you give a quiz with 5 questions, each with a possible
answer of True or False. Therefore, there are two ways to answer the first
question, two ways to answer the second question, and so on. The total
number of possible ways to answer all of the questions on the exam is

2 · 2 · 2 · 2 · 2 = 25 = 32.
II. Sampling without replacement and with ordering.
A total of m balls are drawn sequentially with m ≤ n. After a ball is drawn,
it is not replaced in the urn and hence cannot be drawn again. The order of
the balls is noted. Since there are n possible ways to draw the first ball, n − 1
possible ways to draw the second, n − 2 possible ways to draw the third, etc.,
the total number of possible sequences is

n(n − 1)(n − 2) · · · (n − m + 1).

For the special case when m = n, the total number of possible ways to draw
n balls is
n(n − 1)(n − 2) · · · (2)(1) ≡ n!.
Note that when m = n, this is equivalent to the problem of counting the
number of ways you can arrange n different balls in a row.
Computational Hint: We can compute
n!
n(n − 1)(n − 2) · · · (n − m + 1) = .
(n − m)!
By definition, 0! ≡ 1 so that when n = m, we have
n! n!
n(n − 1)(n − 2) · · · (n − n + 1) = = = n !.
(n − n)! 0!

Example: Suppose you have five different birthday cards, and you have
three friends (named Bill, Mary, and Sue) with birthdays this week. You
want to send one card to each friend. The number of possible choices for
Bill’s card is 5. The card for Mary’s must come from the remaining 4 cards.
Finally, the number of possible choices for Sue’s card is 3. Hence the total
number of ways you can assign five birthday cards to your three friends is

5 · 4 · 3 = 60.

It’s interesting to note that the sequence of friends (Bill, then Mary, then Sue)
is arbitrary and does not affect the answer.
Here is another way to look at this problem: Consider the empty mailboxes
of your three friends,
? ? ?
B M S
 If the cards are labelled a,b,c,d,e, then one possible assignment is

c d a
B M S

There are 5 ways to fill the first position (i.e., Bill’s mailbox), 4 ways to fill
the second position and 3 ways to fill the third.
Example: Using the above example, we can compute the number of ways
one can arrange the three letters x, y, z. This time we have 3 positions to fill
with 3 different items.
? ? ?
1 2 3
So a possible arrangement is

y z x
1 2 3

and there are 3 · 2 · 1 = 3! = 6 possible assignments.

III. Sampling without replacement and without ordering.
A total of m balls (m ≤ n) are drawn from the urn. The balls are not replaced
after each drawing and the order of the drawings is not observed. Hence,
this case can be thought of as reaching into the urn and drawing m balls
simultaneously. The total number of ways a subset of m items can be drawn
from a set of n elements is
 
n! n
(1) ≡ .
m!(n − m)! m

This number is often referred to as the binomial coefficient.

To derive the above formula, let x denote the number of ways to choose m
items from n items without replacement and without ordering. We can find
the value of x by considering Case II as a two-stage process:
Stage 1 Grab m items from n without replacement and without ordering.
Stage 2 Given the m items selected in Stage 1, arrange the m items in order.
Combined, the two stages implement Case II (sampling without replacement
and with ordering). Stage 1 (selecting the m balls) can be performed x ways,
our unknown quantity. Stage 2 (ordering the m balls) can be performed m!
ways (using Case II for m items drawn from m items).
Furthermore, we know, using Case II (with m items drawn from n items),
that the entire process of Stage 1 followed by Stage 2 can be performed in
n!
(n−m)! ways. Hence, we must have

n!
x · m! =
(n − m)!
In other words,
[Stage 1] · [Stage 2] = [Case II].
Solving for x then yields our result.
Computational Hint: Note that
   
n n! n! n
= = = .
m m!(n − m)! (n − m)!m! n−m
So, for example,
       
5 5 9 9
= and = .
3 2 4 5
Also
n
note that if n is large, brute force computation of all three factorial terms
of m can be a challenge. To overcome this, note that
 
n n! n(n − 1)(n − 2) · · · (n − m + 1)
= = .
m m!(n − m)! m!
with only m terms in both the numerator and the denominator. Hence,
 
10 10! 10 · 9
= = = 45
2 2!8! 2!
and    
100 100 100! 100 · 99 · 98
= = = = 161700.
97 3 3!97! 3!

Example: A consumer survey asks: Select exactly two vegetables from the
following list:
◦ peas ◦ carrots ◦ lima beans ◦ corn
4
The total number of ways you can select two items from the four is 2 = 6.
IV. Sampling with replacement and without ordering.
This case is perhaps the most difficult to perceive as a sampling discipline,
yet it has important applications. A total of m balls are drawn sequentially
from the urn. Each ball is replaced after being drawn. However, the sequence
of the draws is ignored. Therefore, drawing 3, 3, 1, 2 is the same as drawing
1, 3, 2, 3. The number of ways that m balls can be drawn from an urn
containing n balls in this fashion is
 
m+n−1
.
m

Let’s prove that this formula works. First, note that to describe a possible
outcome from this scenario, we only need to know how many times each
ball was drawn. For example, to describe 3, 3, 1, 2 or 1, 3, 2, 3 we only need
to know that Ball 1 was drawn once, Ball 2 was drawn once, and Ball 3 was
drawn twice. This arrangement can be represented as follows:

1 2 3
√ √ √√
1 2 3 3

For another example using this scheme, the arrangements 2, 2, 2, 3 or 2, 3, 2, 2

and their counterparts would be represented as

1 2 3
√√√ √
2 2 2 3

Now notice that any of these outcomes can be represented by only specifying
√
the arrangement of the four check marks ( ) among the inner two vertical
lines (|). For example,

Coded
1 2 3 Representation
√ √ √√ √ √ √√
1 2 3 3 | |
√√√ √ √√√ √
2 2 2 3 | |

So, for example,

√√√√
||
would represent 3, 3, 3, 3.
To count the total number of possible outcomes, note that our coded repre-
sentation has six positions

? ? ? ? ? ?
√
Any four of these can be selected for the check marks ( ) and the remaining
two will be filled with the two vertical lines

(|). The number of ways we
can select 4 positions from 6 positions is 64 = 15. Therefore, there are 15
possible ways we can select 4 balls from an urn containing 3 balls if we allow
replacement and disregard ordering.
In general, if there are n balls and m drawings, we will need m check marks
√
( ). To represent all possible outcomes, our table will have n columns.

Coded
1 2 ··· n Representation
√√ √ √ √√ √ √
k1 k2 · · · kn ··· | |···|

Therefore, our coded representation will have m check marks and n − 1

vertical lines. So we need (n − 1) + m positions for both check marks and
vertical lines
? ? ? ··· ?


(n − 1) + m
positions

The number of ways we can select m positions for check marks from (n −
1) + m positions is  
n+m−1
m

Example: Mary wants to buy a dozen cookies. She likes peanut butter
cookies, chocolate chip cookies and sugar cookies. The baker will allow her
to select any number of each type to make a dozen. In this case, our table
has three columns with twelve check marks. So the following arrangement:

Peanut Chocolate Coded

Butter Chip Sugar Representation
√√ √√√ √√√√√√√ √√ √√√ √√√√√√√
| |

would represent the outcome when Mary selects 2 peanut butter, 3 chocolate
chip and 7 sugar cookies. In this case, we are arranging 14 symbols in a row:
 2 vertical lines and 12 check marks. So the number of possible different
boxes of one dozen cookies is
     
12 + 3 − 1 14 14 14 · 13
= = = = 91.
12 12 2 2·1

Permutation of n items distinguished by groups

An extension of case III is the situation where one has n balls with n1 of them
painted with color number 1, n2 of them painted with color number 2,. . . ,nk of
them painted with color number k . We have
k

ni = n.
i=1

If the balls are arranged in a row, the number of distinguishable arrangements is

given by
n!
.
n1 !n2 ! · · · nk !
This quantity is a generalization of the binomial coefficient and is called the multi-
nomial coefficient. Note that when k = 2, this situation is the same as case
III.
Example: Consider the seven characters, A, A, A, B, B, C, C . The number of
distinct arrangements of these seven letters in a row is
7!
= 210.
3!2!2!

Occupancy
The Token Model: In sampling problems, we think of n balls placed in an urn from
which m balls are drawn. A direct analogy can be made with so-called occupancy
problems. In such models, we are given n boxes marked 1 to n, and m tokens
numbered 1 to m. There are four disciplines for placing the tokens in the boxes.
Each of these four scenarios correspond to one of the four sampling cases.
The reason for introducing occupancy problems becomes clear as one encounters
different counting problems. Some may be easier to conceptualize as occupancy
problems rather than sampling problems. Nevertheless, the number of distinguish-
able ways in which the tokens may occupy the boxes is computed in precisely the
same manner as in the sampling models.
The connection between the two formulations is easy to remember if you note the
following relationships:
Sampling Problems Occupancy Problems
ball box
number of drawing number on token
j th draw yields ball k j th token is placed in box k
For example, suppose we have a sampling problem with 3 balls (labelled a, b, c)
being drawn 4 times. This would correspond to an occupancy problem with 3
boxes and 4 tokens (See Figure 2.2). Moreover, the following outcomes from each

Tokens Tokens

1 2 3 4 1 2 3 4

2 1 4
3

a b c a b c

(a) Token labels observed (b) Token labels not observed

Figure 2.2: The occupancy problem

model would be equivalent:

Sampling
Occupancy
Drawing Ball
Box Token(s)
1 b
a 2
2 a
b 1,3
3 b
c 4
4 c

Sampling cases I to IV can now be viewed as

I*. Each box may contain any number of tokens and the labels on the tokens are
observed.
II*. No box may contain more than one token and the labels on the tokens are
observed.
III*. No box may contain more than one token and the labels on the tokens are not
observed.
IV*. Each box may contain any number of tokens and the labels on the tokens are
not observed.
The scheme with check marks and vertical bars that we used for Case IV is now
readily apparent. Each column of the table represents a box, and each check mark
represents a token.
Example: Suppose a school has five classrooms (with different room numbers)
and three teachers, Ms. Smith, Mr. Thomas and Ms. Warren. For some courses,
more than one teacher might be assigned to the same classroom.

I*. If more than one teacher can be assigned to a classroom, and it matters which
teacher teaches in a particular room, then the number of ways you can assign
teachers to classrooms is

nm = 53 = 125.

II*. If only one teacher can be assigned to a classroom, and it matters which teacher
teaches in a particular room, then the number of ways you can assign teachers
to classrooms is

n(n − 1) · · · (n − m + 1) = 5 · 4 · 3· = 60.

III*. If only one teacher can be assigned to a classroom, and it does not matter
which teacher teaches in a particular room, then the number of ways you can
assign teachers to classrooms is
   
n 5
= = 10.
m 3

IV*. If more than one teacher can be assigned to a classroom, and it does not
matter which teacher teaches in a particular room, then the number of ways
 you can assign teachers to classrooms is
   
m+n−1 3+5−1
= = 35.
m 3

Self-Test Exercises for Chapter 2

For each of the following multiple-choice questions, choose the best response
among those provided. Answers can be found in Appendix B.
S2.1 Bill’s Donut Shop bakes 10 varieties of doughnuts. How many different
boxes of 12 doughnuts can a customer select from the 10 varieties? Note:
Assume that Bill’s Shop has at least 12 of each type of donut.
(A) 12!
21
(B) 12

(C) 10!
(D) 1210
(E) none of the above.
S2.2 A teacher has a class of 30 students. Each day, she selects a team of 6
students to present their homework solutions to the rest of the class. How
many different teams of 6 students can she select?
30
(A) 6

(B) 306
(C) 30!/24!
(D) 630
(E) none of the above
S2.3 A box is partitioned into 3 numbered cells. Two indistinguishable balls are
dropped into the box. Each cell may contain at most one ball. The total
number of distinguishable arrangements of the balls in the numbered cells is
(A) 1
(B) 3
(C) 6
 (D) 9
(E) none of the above.
S2.4 A professor wants to use three colors (red, green and blue) to draw an
equilateral triangle on a sheet of paper. Each edge of the triangle is a single
color, but any color can be used more than once. How many different
triangles can be drawn?
(A) 33
(B) 3!
5
(C) 3
5
(D) 3 +1
(E) none of the above.
S2.5 A professor has an equilateral triangle cut from a sheet of cardboard. He
wants to use three colors (red, green and blue) to color the edges. Each edge
of the triangle is a single color, but any color can be used more than once.
How many different triangles can be produced?
(A) 33
(B) 3!
5
(C) 3
5
(D) 3 +1
(E) none of the above.
S2.6 Two white balls are dropped randomly into four labelled boxes, one at a
time. A box may contain more than one ball. The number of distinguishable
arrangements of the balls in the boxes is
4
(A) 2
7
(B) 2

(C) 4
(D) 42
(E) none of the above.
S2.7 One white ball and one black ball are dropped randomly into four labelled
boxes, one at a time. A box may contain more than one ball. The number of
distinguishable arrangements of the balls in the boxes is
4
(A) 2
7
(B) 2

(C) 4
(D) 42
(E) none of the above.
S2.8 A class has ten students. The teacher must select three students to form a
committee. How many different possible committees can be formed?
(A) 103
(B) 310
(C) 10!/7!
10
(D) 3

(E) none of the above

S2.9 A student club has ten members. The club must elect a president, vice-
president and treasurer from the membership. How many ways can these
three positions be filled?
(A) 103
(B) 310
(C) 10!/7!
10
(D) 3

(E) none of the above

S2.10 A candy company is introducing a new, bite-sized, chocolate candy called
"XYZ’s". Each candy is covered with a sugar coat that is colored a single
color: either red, green, or blue. Also each candy is marked with a single
letter: either “X,” “Y” or “Z’.
How many different possible candies are there?
 (A) 9
(B) 33
9
(C) 3

(D) 3!
(E) none of the above.

S2.11 An ice cream shop offers 20 different flavors of ice cream. Their menu says
that their “Big Dish” contains five scoops of ice cream. You may order the
five scoops in any way, ranging from five different flavors to five scoops of
the same flavor. How many different “Big Dishes” are possible?
(A) 205
20
(B) 5
24
(C) 5

(D) 20!/5!
(E) none of the above
S2.12 There are 4 machines in a factory. If each is worker assigned to one machine,
how many ways can 4 workers be assigned to the machines?
(A) 42
(B) 4!
7
(C) 4

(D) 44
(E) none of the above

Questions for Chapter 2

3
PROBABILITY

The scientific imagination always restrains itself within the limits of probability.
– THOMAS HENRY HUXLEY, Reflection (c 1880)

Probability Spaces
In Chapter 1, we showed how you can use sample spaces Ω and events A ⊆ Ω
to describe an experiment. In this chapter, we will define what we mean by
the probability of an event, P (A) and the mathematics necessary to consistently
compute P (A) for all events A ⊆ Ω.
For infinite sample spaces, Ω, (for example, Ω = R) we often cannot reasonably
compute P (A) for all A ⊆ Ω. In those cases, we restrict our definition of P (A) to
some smaller subset of events, F, that does not necessarily include every possible
subset of Ω. The mathematical details are usually covered in advanced courses in
probability theory.

Measuring sets

Definition 3.1. The sample space, Ω, is the set of all possible outcomes of an
experiment.
Definition 3.2. An event is any subset of the sample space.
How would you measure a set?

• finite sets
• countable sets
• uncountable sets

Definition 3.3. A probability measure, P (·), is a function, defined on subsets of

Ω, that assigns to each event E a real number P (E) such that the following three
probability axioms hold:

35
 1. P (E) ≥ 0 for all events E
2. P (Ω) = 1
3. If E ∩ F = Ø (i.e., if the events E and F are mutually exclusive ), then
P (E ∪ F ) = P (E) + P (F )

The three specifications in the above definition are often called the Kolmogorov
axioms for probability measures. However, a great deal of probability theory has
its foundations in a mathematical field called measure theory with important results
from many mathematicians, including Emile Borel, Henri Lebesgue and Eugene
Dynkin.
Note that every event A ⊆ Ω is assigned one and only one number P (A). In
section 3, we will show that 0 ≤ P (A) ≤ 1 for every A ⊆ Ω. In mathematical
terms, P (·) is a function that assigns to each member of F, the set of events, a
single number from the interval [0, 1]. That is
P : F → [0, 1].

In calculus, you worked with real-valued functions f (·). That is,

f : R → R.

Such functions assign each number in R a single number from R. For example,
f (x) = x3 , maps x = 2 into f (2) = 8.
Here, the function, P (·), is a function whose argument is a subset of Ω (namely
an event) rather than a number. In mathematics we call functions that assign
real numbers to subsets, measures. That’s why we refer to P (·) as a probability
measure.

Interpretation of probability

How do you assign the real numbers in Definition 3.3?

Logical probability This can be done when Ω is a finite set

Ω = {ω1 , ω2 , . . . , ωn }
and each of the outcomes in Ω is equally likely. That is
1
P ({ωi }) = for i = 1, . . . , n
n
 In that case, for any A ⊆ Ω assign

|A|
P (A) =
n
For example, consider the outcome of rolling a fair six-sided die once, In
that case
Ω = {ω1 , ω2 , . . . , ω6 }
where
ωi = the die shows i pips
for i = 1, 2, . . . , 6. So we can assign, for example,
1
P ({ωi }) = for i = 1, . . . , 6
6
3
P ({The die is even}) = P ({ω2 , ω4 , ω6 }) =
6
and, in general,
|A|
P (A) = for any A ⊆ Ω
6
Logical probability is only useful when the mechanism producing the random
outcomes has symmetry so that one could argue that there is no reason for one
outcome to occur rather than other. Some examples where logical probability
can be used are:
tossing a fair coin The coin is symmetrical and there is no reason for heads
to turn up rather than tails.
rolling a fair die The die is symmetrical and there is no reason for one side
to turn up rather than another.
drawing a card randomly from a deck of playing cards Each card is phys-
ically the same, and there is no reason for one card to be drawn rather
than another.
As you can see, the notion of logical probability has limited applications.
Experimental probability This interpretation requires that the experiment can
be identically reproduced over and over again, indefinitely. Each replicate
is called an experimental trial. We also must be able to assume that the
outcome of any one of the trials does not affect the probabilities associated
with the outcomes of any of the other trials. As an example, suppose we
examine transistors being produced by an assembly line, with each transistor
an experimental trial. The outcome of one of our experimental trials is
whether that transistor is either good or bad. Consider the sequence of
outcomes for the repeated experiment, with

Ti = 0 if transistor i is bad
Ti = 1 if transistor i is good

In this case, Ω = {bad, good}. Let A = {good}. To assign P (A) using

logical probability, consider the infinite sequence of experimental outcomes

T 1 , T2 , . . .

and assign n
i=1 Ti
P (A) = lim
n→∞ n
There are some important mathematical issues associated with defining P (A)
in this way. In particular, the type of limit that you defined in calculus cannot
be used. In calculus, there always existed some K , determined in advance,
such that for all n > K the quantity
n
i=1 Ti
n
was arbitrarily close to P (A). That concept is not suitable here.
Suppose, for instance, that P (A) equaled 12 . It is possible, although not
likely, that you could examine
 billions of transistors and find that all of them
n
Ti
are bad. In that case, i=1 n would be zero, and quite far away from the
1
eventual limit of 2 as n gets bigger than billions.
In fact, it can be proven that for a prespecified small number δ > 0, there
does exist some integer K such that for all n > K
 n 
 i=1 Ti 
 − P (A) < δ.
 n
The value of K is, itself, random and cannot be predetermined. We can,
however, say that P (K < ∞) = 1. The proof requires measure theory, and
is typically covered in a graduate-level probability course.
Some examples where experimental probability can be used are:
 quality assurance Specifying the probability that a single microprocessor
from a semiconductor fabrication facility will be defective.
interarrival times for queues Stating the probability that the time between
any two consecutive customers is greater than five minutes.
hitting a home run Specifying the probability that a baseball player will hit
a home run on any attempt.
Subjective probability There are some cases where we would like to use probabil-
ity theory, but it is impossible to argue that the experiment can be replicated.
For example, consider the experiment of taking EAS305, with the outcome
being the grade that you get in the course. Let

A = {you get an “A” in EAS305}

The procedure of determining the subjective probability of A, is to construct

an equivalent lottery consisting of an urn, and an unlimited quantity of red
and white balls. The decision maker is asked to place in the urn any number
of red balls and white balls he or she desires, with the goal of making the
event of drawing a red ball equivalent to getting an “A” in EAS305.
There is a mathematical theory (complete with axioms and deductions) re-
garding the existence of such equivalent lotteries.
Once the decision maker has decided how many balls of each type to place
in the urn, we simply count the balls in the urn and assign
number of red balls in the urn
P (A) =
total number of balls in the urn
Not surprisingly, subjective probability has a lot of critics. But sometimes it
is the only way we can give meaning to the number P (A).
Some examples where subjective probability can be used are:
weather Specifying the probability that it will be warm and sunny tomorrow.
auctions Stating the probability that painting by Monet will sell for one
million dollars at an art auction.
football Specifying the probability that the Buffalo Bills will win the Super
Bowl next year.
Examples using logical probability

Example: A box contains three light bulbs, and one of them is bad. If two light
bulbs are chosen at random from the box, what is the probability that one of the
two bulbs is bad?
Solution 1: Assume that the bulbs are labelled A, B , and C and that light bulb A
is bad. If the two bulbs are selected one at a time, there are (3)(2) = 6 possible
outcomes of the experiment:
AB BA
AC CA
CB BC
Note that 4 of the 6 outcomes result in choosing A, the bad light bulb. If these 6
outcomes are equally likely (are they?), then the probability of selecting the bad
bulb is 46 = 23
Solution 2:
3
If the two bulbs are selected simultaneously (i.e., without ordering),
there are 2 = 3 possible outcomes of the experiment:

AB AC CB

and 2 of the 3 outcomes result in choosing bad bulb A. If these 3 outcomes are
equally likely (are they?), then the probability of selecting the bad bulb is 23
Example: A box contains two light bulbs, and one of them is bad. Suppose a
first bulb is chosen, tested and then placed back in the box. Then a second bulb is
chosen, and tested. Note that the same bulb might be selected twice. What is the
probability that exactly one of the two tested bulbs is bad?
“Solution:” Suppose the two bulbs are labelled A and B and bulb A is the bad
light bulb, There are three possible outcomes:

Both bulbs are good

Both bulbs are bad
Exactly one bulb is bad

So the probability that exactly one bulb is bad is 13 . (Actually, this solution is
wrong. Why it is wrong?)
Relationships between English and set notation

• P (not E) = P (E c )
• P (E or F ) = P (E ∪ F )
• P (E and F ) = P (E ∩ F )

Deductions from the axioms

Theorem 3.1. P (Ac ) = 1 − P (A)

Proof.

P (Ω) = 1 from Axiom 2

P (A ∪ Ac ) = 1 since Ω = A ∪ Ac
P (A) + P (Ac ) = 1 since A ∩ Ac = Ø
P (Ac ) = 1 − P (A)

Theorem 3.2. P (A) ≤ 1 for all A ⊆ Ω

Proof.

P (A) = 1 − P (Ac ) from Theorem 3.1

P (A) ≤ 1 since P (Ac ) ≥ 0 (Axiom 1)

Theorem 3.3. A ⊆ B ⇒ P (A) ≤ P (B)

Proof.

A∪B = B since A ⊆ B
(A ∪ B) ∩ (A ∪ Ac ) = B since A ∪ Ac = Ω
A ∪ (B ∩ Ac ) = B DeMorgan’s law
P (A ∪ (B ∩ Ac )) = P (B) See Question 3.31
P (A) + P (B ∩ Ac ) = P (B) A ∩ (B ∩ Ac ) = Ø
P (A) ≤ P (B) since P (B ∩ Ac ) ≥ 0 (Axiom 1)

Example: In a class of 100 students it is known that 80 of the students like

hamburger. Consider the experiment of picking a student at random from the class
of 100. Let A be the event that the selected student is one of those who likes
hamburger. In this case |A| = 80 and |Ω| = 100 and we can use logical probability
to determine P (A) = 80/100 = 0.80.
Let B be the event that the selected student is one of those who likes broccoli.
Suppose |B| is unknown.

3 Describe the set A ∩ B .

Solution: A ∩ B is the event that the student selected likes hamburger and
broccoli.
3 Show that A ∩ B ⊆ A.
Solution: Pick an element ω ∈ A ∩ B . That implies that ω ∈ A and ω ∈ B .
Hence, ω ∈ A. Therefore, A ∩ B ⊆ A.
3 Even if |B| is unknown we know that P (A ∩ B) ≤ 0.80. Why?
Solution: For any events E and F , E ⊆ F implies P (E) ≤ P (F ). Since
A ∩ B ⊆ A, we have P (A ∩ B) ≤ P (A) = 0.80.

Theorem 3.4. For any finite number of pairwise disjoint sets A1 , A2 , . . . , An we

have
P (A1 ∪ A2 ∪ · · · ∪ An ) = P (A1 ) + P (A2 ) + · · · + P (An )

Proof. (By induction)

Is it true for n = 2?
Yes, that’s Axiom 3.
Assume it’s true for n = k , is it true for n = k + 1?
Assume for k pairwise disjoint sets A1 , A2 , . . . , Ak we have

P (A1 ∪ · · · ∪ Ak ) = P (A1 ) + · · · + P (Ak )

Let Ak+1 be an event such that Ai ∩ Ak+1 = Ø for all i = 1, . . . , k . Then

(A1 ∪ · · · ∪ Ak ) ∩ Ak+1 = Ø
since Ai ∩ Ak+1 = Ø for all i
P ((A1 ∪ · · · ∪ Ak ) ∪ Ak+1 ) = P (A1 ∪ · · · ∪ Ak ) + P (Ak+1 )
since (A1 ∪ · · · ∪ Ak ) ∩ Ak+1 = Ø
P ((A1 ∪ · · · ∪ Ak ) ∪ Ak+1 ) = (P (A1 ) + . . . + P (Ak )) + P (Ak+1 )
from induction hypothesis
P (A1 ∪ · · · ∪ Ak ∪ Ak+1 ) = P (A1 ) + . . . + P (Ak+1 )
Associative law

Theorem 3.5. For any two events A and B

P (A ∪ B) = P (A) + P (B) − P (A ∩ B)

Proof.

A ∪ B = A ∪ (Ac ∩ B)
Distributive law for sets
and
B = (A ∩ B) ∪ (Ac ∩ B)
Distributive law for sets
Therefore
P (A ∪ B) = P (A) + P (Ac ∩ B)
from Axiom 3
P (B) = P (A ∩ B) + P (Ac ∩ B)
from Axiom 3
Subtracting the second equation from the first produces the result.
Example: A hydraulic piston can have only two types of defects:
1. a surface defect, and/or
2. a structural defect
A single piston may have one, both or neither defect. From quality control records
we see that
• The probability that a piston has a surface defect is 0.5.
 • The probability that a piston has a structural defect is 0.4.
• The probability that a piston has some type of defect is 0.6
To find the probability that a piston has both a structural and a surface defect, first
define the following events:

S = {a piston has a surface defect}

T = {a piston has a structural defect}

We want to find P (S ∩ T ).
We know that

P (S) = 0.5
P (T ) = 0.4
P (S ∪ T ) = 0.6

Therefore, using Theorem 3.5,

P (S ∪ T ) = P (S) + P (T ) − P (S ∩ T )
0.6 = 0.4 + 0.5 − P (S ∩ T )
P (S ∩ T ) = 0.4 + 0.5 − 0.6 = 0.3

Theorem 3.6. P (Ø) = 0

Proof.

P (Ø) = 1 − P (Øc ) from Theorem 3.1

P (Ø) = 1 − P (Ω) since Øc = Ω
P (Ø) = 1−1 from Axiom 2
P (Ø) = 0

In order to allow the extension of Axiom (3) in Definition 3.3 to apply to a countable
collection of disjoint sets (rather than just a finite collection), the third probability
axiom is replaced by
3.* If A1 , A2 , . . . is a countable sequence of disjoint sets, then
∞  ∞
 
P Ai = P (Ai )
i=1 i=1
 Example: Consider the experiment of counting the number of accidents in a factory
in a month. Let Ei be the event that exactly i accidents occur in a month with
i = 0, 1, 2, . . .. Suppose that

e−1
P (Ei ) = for i = 0, 1, 2, . . .
i!
and we want to find the probability that the factory has at least one accident in a
month.

That is, we want to compute P ∪∞
i=1 Ei .
∞ α i α
Since i=0 i! = e for any α, we have

∞
 ∞ −1
 e
P (Ei ) =
i=0 i=0
i!
∞
 1
= e−1
i=0
i!
−1 1
= e e =1

Using axiom 3.* and the fact that Ei ∩ Ej = Ø for any i = j , we get
∞  ∞
 
P Ei = P (Ei ) = 1
i=0 i=0
 ∞ 

P E0 ∪ Ei = 1
i=1
∞ 
  
P (E0 ) + P Ei = 1 Since E0 ∩ ∪∞
i=1 Ei = Ø
i=1
∞ 

P Ei = 1 − P (E0 ) = 1 − e−1
i=1

THE BIRTHDAY PROBLEM

Suppose k randomly selected individuals are seated in a room. What is the proba-
bility that at least two of them will have the same birthday? How large must k be
so that the probability is 0.50 that at least two of them will have the same birthday?
To solve this problem, we will make two assumptions:
 1. there are n = 365 days in a year
2. each person is assigned a birthday with any of the n = 365 days being equally
likely.
To illustrate the issues, we’ll first solve the problem for k = 3 and n = 365. Later,
we’ll extend our solution for any k , and any n.
Define the events
E = {at least 2 birthdays are the same}
F = {all of the birthdays are different}
and notice that E = F c . The question asks us to compute P (E). But P (E) =
1 − P (F ) and it turns out that P (F ) is easier to compute.
To compute P (F ), we’ll use an approach that is common for most problems
involving logical probability:
Step 1 Count the number of equally likely outcomes in Ω.
Step 2 Count the number of elements in F ⊆ Ω.
Step 3 Divide the number obtained in Step 2 by the number in Step 1.
It sounds simple, that is until you begin identifying and counting equally likely
elements in Ω. Some textbooks1 say something like “If we imagine the k people
to be ordered, there are (365)k corresponding permutations of their k birthdays.”
Why ordered? Why not?
Suppose k = 3. Let’s suppose the individuals are ordered. Then every element of
the sample space can be written as an ordered triple ω = (d1 , d2 , d3 ), where d1 , d2
and d3 are the birthdays of the three individuals. We can begin to write down all of
the elements of Ω as follows:
Ω = {(Jan01,Jan01,Jan01),
(Jan01,Jan01,Jan02),
(Jan01,Jan01,Jan03),
(Jan01,Jan01,Jan04), . . .}
The assignment of birthdays with ordered individuals can be represented as a
sampling problem. The number of ways we can draw three balls from 365 with
replacement and with ordering is (365)3 .
1
For just one example of many, see Richard J. Larsen and Morris Marx, An introduction to
probability and its applications, Prentice-Hall, 1985.
Continuing with this reasoning, the number of ways that we can assign birthdays
to individuals with everyone getting a different birthday is equal to the number of
ways we can draw three balls from 365 without replacement and with ordering.
And that number is (365)!/(365 − 3)!.
So for k = 3 we get
(365)!/(365 − 3)!
P (E) = 1 − P (F ) = 1 −
(365)3
This turns out to be the correct answer. Furthermore, if we perform the same
computations for any n and k ≤ n, the general correct answer is
(n)!/(n − k)!
P (E) = 1 − P (F ) = 1 −
(n)k
To answer the question “how large must k be?,” the following table lists P (E) for
several different values of k with n = 365:
k P (E)
15 0.253
22 0.475
23 0.507
40 0.891
50 0.970
70 0.999

and we see that we need at least 23 persons to insure that P (E) is at least 0.50.
But why would we get the wrong answer if we don’t order the k individuals? Let’s
see what this would mean for k = 3.
Identifying the elements of Ω without ordering the individuals is equivalent to
listing all of the ways we can draw 3 balls from an urn containing 365 balls, with
replacement and without ordering. We can do this by creating a table with 365
columns (one for each calendar day) and asking the question, “how many ways can
we arrange 3 check marks among the 365 columns?”

Jan01 Jan02 Jan03 ...

√√√
√√ √
√√ √
. . . etc.
 Using this approach, each element of Ω is distinguishable by the number of times
someone in the group is assigned the birthday Jan01, Jan02, etc., not who in the
group is assigned Jan01, Jan02, etc.. There is nothing wrong with identifying
the elements of the sample space Ω in this way. But we would wrong to assume
that such elements are equally likely.
We can again compute P (E), but this time, using the unordered individual assump-
tion. In this case, the total number of elements in Ω is
 
n+k−1 (n + k − 1)!
=
k k !(n − 1)!

and the number of elements in F is

 
n n!
=
k k !(n − k)!

So that
(n)!(n − 1)!
P (E) = 1 − P (F ) = 1 −
(n + k − 1)!(n − k)!
To show why the unordered individual assumption is wrong, let’s use all of our
formulas to solve a slightly modified version of the birthday problem:

The birthday problem on the planet Yak

There are two aliens from outer space in a room. They are from a planet called Yak.
Yak has only two days in its calendar year. The names of the days are “Heads” and
“Tails”.
Once again, assume that birthdays are randomly assigned, with each day equally
likely. Assigning birthdays at random to the aliens is the same as each alien tossing
a fair coin to determine it’s birthday. So we can use our formulas with values n = 2
and k = 2.
Under the ordered individual model there are four elements in Ω, that is

Ω = {HH, HT, T H, T T },

and
(2)!/(2 − 2)! 1
P (E) = 1 − =
(2)2 2
 (1, 1) (1, 2) (1, 3) (1, 4) (1, 5) (1, 6)
(2, 1) (2, 2) (2, 3) (2, 4) (2, 5) (2, 6)
(3, 1) (3, 2) (3, 3) (3, 4) (3, 5) (3, 6)
(4, 1) (4, 2) (4, 3) (4, 4) (4, 5) (4, 6)
(5, 1) (5, 2) (5, 3) (5, 4) (5, 5) (5, 6)
(6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6)

Figure 3.1: Sample Space for Rolling a Die Twice

Under the unordered individual model there are three elements in Ω,

Ω = {2H, 2T, 1H1T}

that is, “two heads”, “two tails” and “one head one tail.” And the corresponding
computation for P (E) yields

(2)!(2 − 1)! 2
P (E) = 1 − P (F ) = 1 − =
(2 + 2 − 1)!(2 − 2)! 3

And we see that the second (and faulty) formulation results in the same mistake
that Jean d‘Alembert2 made in 1754.
Moral of the story: Sampling Case IV (with replacement and without ordering) usu-
ally cannot be used to enumerate equally likely outcomes resulting from multiple,
repeated random assignments.3

Conditional Probability and Independence

Conditional probability

Example: Roll a die twice. The sample space can be represented as in Figure 3.1.
Define the events

A = {first roll is less than or equal to the second roll}

B = {the sum of the two rolls is 4}
2
Jean d’Alembert Encyclopédie ou dictionnaire raisonné des sciences, des arts and des métiers,
Volume 4, 1754.
3
For an important exception, see the work of Bose and Einstein in statistical mechanics.
 Since each of the outcomes is equally likely, we can use logical probability to get
P (B) = 1/12. Suppose you are told that the first roll is less than or equal to the
second. With this additional information, what is the probability that the two rolls
sum to 4?
Definition 3.4. The probability of an event A given the event B , denoted
P (A | B), is the value that solves the equation

P (A ∩ B) = P (A | B)P (B)

Note: If P (B) > 0, this is equivalent to

P (A ∩ B)
P (A | B) =
P (B)

Notation: Sometimes we write P (A ∩ B) = P (AB).

Theorem 3.7. For any events A1 , A2 , . . . , An

P (A1 A2 · · · An ) = P (A1 )P (A2 | A1 )P (A3 | A1 A2 ) · · · P (An | A1 A2 · · · An−1 )

P (· | B) is a probability measure

Remember that P (·) is a probability measure if and only if

1. P (E) ≥ 0 for all events E
2. P (Ω) = 1
3. If E ∩ F = Ø then P (E ∪ F ) = P (E) + P (F )
Suppose P (B) > 0 and consider P (· | B)
• For any event E ⊆ Ω

P (E ∩ B)
P (E | B) =
P (B)
[something ≥ 0]
= ≥0
[something > 0]

Therefore P (E | B) ≥ 0 for all events E

 • Also, we have
P (Ω ∩ B)
P (Ω | B) =
P (B)
P (B)
= =1
P (B)

Therefore P (Ω | B) = 1
• And for any two events E and F with E ∩ F = Ø

P ((E ∪ F ) ∩ B)
P (E ∪ F | B) =
P (B)
P ((E ∩ B) ∪ (F ∩ B)
=
P (B)
P (E ∩ B) + P (F ∩ B)
=
P (B)
P (E ∩ B) P (F ∩ B)
= +
P (B) P (B)
= P (E | B) + P (F | B)

Therefore if E ∩ F = Ø then P (E ∪ F | B) = P (E | B) + P (F | B)

• Hence P (· | B) is a probability measure.

• We can think of P (· | B) as the updated version of P (·) given B .
• In fact, every event A ⊆ Ω produces an updated probability measure P (· | A).
• When interpreting P (E | A) we are asking the “what if” question, “what is
the probability of E if the event A occurs.”
• We can evaluate and use P (E | A) and P (E | B) simultaneously for different
events, A and B .

Independence

Definition 3.5. The events A and B are independent if and only if

P (A ∩ B) = P (A)P (B).
 Note: If P (B) > 0 this is equivalent to saying that A and B are
independent if and only if
P (A | B) = P (A).

Theorem 3.8. If A and B are independent events then

• A and B c are independent events,
• Ac and B are independent events, and
• Ac and B c are independent events.
Proof. Suppose A and B are independent events. Then
P (A ∩ (B ∪ B c )) = P (A)
P ((A ∩ B) ∪ (A ∩ B c )) = P (A)
P (A ∩ B) + P (A ∩ B c ) = P (A)
P (A)P (B) + P (A ∩ B c ) = P (A)
P (A ∩ B c ) = P (A) − P (A)P (B)
P (A ∩ B c ) = P (A)(1 − P (B))
P (A ∩ B c ) = P (A)P (B c )

Therefore A and B c are independent events.

The other two results follow by interchanging the roles of A and B .

Bayes’ theorem

Definition 3.6. A complete set of alternatives is a collection of events

B1 , B2 , . . . , Bn

such that
1. B1 ∪ B2 ∪ · · · ∪ Bn = Ω, and
2. Bi ∩ Bj = Ø for any i = j .
Theorem 3.9. (Law of Total Probability) Given an event A and a complete set
of alternatives B1 , B2 , . . . , Bn then
n
 n

P (A) = P (A ∩ Bi ) = P (A | Bi )P (Bi )
i=1 i=1
Example: (from Meyer4 ) Suppose that among six bolts, two are shorter than a
specified length. If two bolts are chosen at random, what is the probability that the
two short bolts are picked? Let

Ai = {the ith chosen bolt is short}.

The proper solution is obtained by writing

1 2 1
P (A1 ∩ A2 ) = P (A2 | A1 )P (A1 ) = 5 · 6 = 15 .

The common but incorrect solution is obtained by writing

1 2 1
P (A1 ∩ A2 ) = P (A2 )P (A1 ) = 5 · 6 = 15 .

Although the answer is correct, the value of P (A2 ) is not 15 . But the value of
P (A2 | A1 ) is 15 .
To evaluate P (A2 ) properly, we note that {A1 , Ac1 } is a complete set of alternatives.
We can the use the law of total probability, and write

P (A2 ) = P (A1 )P (A2 | A1 ) + P (Ac1 )P (A2 | Ac1 )

1 2 2 4
= 5 · 6 + 5 · 6 = 13 .

Theorem 3.10. (Bayes’ Theorem) Let B1 , B2 , . . . , Bn be a complete set of

alternatives, then
P (A | Bk )P (Bk )
P (Bk | A) = n
i=1 P (A | Bi )P (Bi )

Example: Thousands of little snails live in the Snail Garden between the University
at Buffalo Natural Sciences Complex and Talbert Hall. A concrete stairway divides
the garden into two sections.
When it rains, the snails bravely travel from one section of the garden to the other by
crawling across the steps of the concrete stairway. It is a treacherous journey. Many
of the snails are crushed by the feet of unthinking students and faculty walking up
and down the stairway.
4
Meyer, P., Introductory probability theory and statistical applications, Addison-Wesley, Reading
MA, 1965.
 There are two types of snails, smart snails and not-so-smart snails. A smart snail
crawls along the vertical portion of a step to avoid being crushed. A not-so-smart
snail crawls on the horizontal part of a step.
Suppose that half of the snails are smart snails, and the other half are not-so-smart
snails. The probability that a smart snail will be crushed is 0.10. The probability
that a not-so-smart snail will be crushed is 0.40. The event that any one snail will
be crushed is independent from all others.

3 To find the probability that a single snail (chosen at random) will successfully
cross the concrete steps uncrushed define the events

S = {The snail is a smart snail}

S c = {The snail is a not-so-smart snail}
A = {The snail is successful}

Note that S ∪ S c = Ω and S ∩ S c = Ø. Therefore, using the Law of Total

Probability

P (A) = P (A | S)P (S) + P (A | S c )P (S c )

= (0.90)(0.50) + (0.60)(0.50) = 0.75

3 Suppose that after the end of a rainstorm, you select, at random, a snail that
has successfully crossed the concrete steps. What is the probability that it is
a smart snail? In this case, we want to find
P (A | S)P (S)
P (S | A) =
P (A | S)P (S) + P (A | S c )P (S c )
(0.90)(0.50)
= = 0.60
(0.90)(0.50) + (0.60)(0.50)

The probability of selecting a smart snail before the rainstorm was 0.50. The
probability after the rainstorm is 0.60.
What is the probability of finding a smart snails after two rainstorms?

Independence of more than two events

The definition of independence can be extended to more than two events.

Definition 3.7. A finite collection of events A1 , A2 , . . . , An are independent if
and only if
P (Ak1 ∩ · · · ∩ Akj ) = P (Ak1 ) · · · P (Akj )
for all 2 ≤ j ≤ n and all 1 ≤ k1 < · · · < kj ≤ n.
Note: For n = 3, the events A1 , A2 , A3 are independent if and only if
(1) P (A1 ∩ A2 ) = P (A1 )P (A2 )
(2) P (A1 ∩ A3 ) = P (A1 )P (A3 )
(3) P (A2 ∩ A3 ) = P (A2 )P (A3 )
(4) P (A1 ∩ A2 ∩ A3 ) = P (A1 )P (A2 )P (A3 )
All of the above must hold for A1 , A2 , A3 to be independent. If only the first three
conditions hold, then we say that A1 , A2 , A3 are pairwise independent.
Three events A, B and C can be pairwise independent, but not independent, as
shown by the following example:
1
Example: Toss a fair coin twice, and assign probability 4 to each of the outcomes
HH, HT, TH, TT. Let
A = {first toss is a head}
B = {second toss is a head}
C = {first toss equals second toss}
Then
1
P (A ∩ B) = 4 = P (A)P (B)
1
P (A ∩ C) = 4 = P (A)P (C)
1
P (B ∩ C) = 4 = P (B)P (C)
1
P (A ∩ B ∩ C) = 4 = P (A)P (B)P (C)
Hence A, B, C are pairwise independent but not independent.
It is also possible to have three events A1 , A2 and A3 where equation (4) holds, but
equations (1), (2) and (3) all fail. This is demonstrated by the following example:
Example: Suppose that two dice are tossed, and let Ω be all ordered pairs (i, j)
1
for i, j = 1, 2, . . . , 6 with probability of 36 assigned to each point. Let
A = {second die is 1, 2 or 5}
B = {second die is 4, 5 or 6}
C = {the sum of the faces is 9}
Then
P (A ∩ B) = 1
6 = P (A)P (B) = 1
4
P (A ∩ C) = 1
36 = P (A)P (C) = 1
18
P (B ∩ C) = 1
12 = P (B)P (C) = 1
18
P (A ∩ B ∩ C) = 1
36 = P (A)P (B)P (C)

Example: Suppose you roll a fair die repeatedly until a six appears. Assume all
the outcomes from each roll are independent. Let

Sk = {toss k is a six}
An = {the first six occurs on toss n}

Then
P (Sk ) = 1
6 P (Skc ) = 5
6

for k = 1, 2, . . ..
Furthermore,
An = S1c ∩ S2c ∩ · · · ∩ Sn−1
c
∩ Sn
for n = 1, 2, . . ..

P (An ) = P (S1c )P (S2c ) · · · P (Sn−1

c
)P (Sn )
 n−1
5 1
= 6 6

for n = 1, 2, . . ..
We can also compute,

P (first 6 occurs after toss k)

∞

= P (An )
n=k+1
∞  n−1
5 1
= 6 6
n=k+1
letting n = n − (k + 1)
∞  
 k+n
5 1
= 6 6
n =0
  k   n
1 5 ∞ 5
= 6 6 n =0 6
 k
1 5
= 6 6 ·6
 k
5
= 6

for k = 0, 1, 2, . . .. Which makes sense, since the first 6 occurs after toss k if and
only if you get k non-6’s in a row. So

{first 6 occurs after toss k} = S1c ∩ S2c ∩ · · · ∩ Skc

P (first 6 occurs after toss k) = P (S1c )P (S2c ) · · · P (Skc )
 k
5
= 6

for k = 0, 1, 2, . . ..

Self-Test Exercises for Chapter 3

For each of the following multiple-choice questions, choose the best response
among those provided. Answers can be found in Appendix B.
S3.1 If P (A) = 0.3, P (B) = 0.3 and P (A | B) = 0.3 then
(A) A and B are mutually exclusive
(B) P (A ∩ B) = 0
(C) A and B are independent
(D) all of the above
(E) none of the above.
S3.2 Let P (A) = P (B) = 2p and let P (A ∪ B) = 3p. Which of the following
are always true?
(A) p = 0
(B) A and B are independent
(C) P ((A ∩ B)c ) = 1 − p
(D) A and B are mutually exclusive
(E) none of the above.
S3.3 If A ⊆ B , then which of the following statements are always true?
(A) A and B are independent
(B) P (A) > P (B)
(C) A and B are mutually exclusive
(D) A ∪ B = A
(E) none of the above
S3.4 If P (A) = p > 0, P (B) = p > 0 and P (A ∪ B) = 2p then P (B | A ∪ B)
equals
(A) 1
(B) p
(C) 1/2
(D) p2
(E) none of the above
S3.5 If P (A) = p, P (B) = p and P (A ∩ B) = p2 then
(A) A ∪ B = Ω
(B) A = B
(C) A and B are independent
(D) all of the above
(E) none of the above.
S3.6 Two fair coins are tossed once. One coin is painted red, and the other is
painted blue. Let R denote the outcome of the red coin (Heads or Tails). Let
B denote the outcome of the blue coin. Then

P (R = Heads | R = B)

is equal to
(A) 1/4
(B) 1/3
(C) 1/2
 (D) 1
(E) none of the above.
S3.7 Two indistinguishable six-sided fair dice are rolled. What is the probability
that they both turn up the same value?
(A) 1/72
(B) 1/36
(C) 1/12
(D) 1/6
(E) none of the above.
S3.8 Suppose P (A) = 0. Then
(A) A = Ø
(B) P (A ∩ A) = P (A)P (A)
(C) P (A ∪ B) = 0 for any B ⊆ Ω.
(D) all of the above are true
(E) none of the above are true.
S3.9 Which of the following statements are always true?
(A) P (Ω) = 1
(B) If P (A) < P (B) then A ⊆ B
(C) If P (A) = 0 then A = Ø
(D) all of the above
(E) none of the above.
S3.10 Suppose P (A) = p, P (B) = q . If A ⊆ B and A = B then
(A) p ≥ q
(B) p ≤ q
(C) p > q
(D) p < q
 (E) none of the above.
S3.11 There is one lock on a door and the key is among the six different ones you
usually carry in your pocket. Someone places a seventh, useless key, in your
pocket. What is the probability that the first key you try will open the door?
(A) 1/6
(B) 1/2
(C) 1/3
(D) 2/3
(E) none of the above.
S3.12 A multiple choice test has 12 questions, with each question having 5 possible
answers. If a student randomly picks the answer to each question, what is
the probability that the student will answer all questions correctly?
(A) 1/(125 )
(B) 1/(512 )
12
(C) 1/ 5
16
(D) 1/ 5

(E) none of the above.

S3.13 Two balls are dropped randomly into four labelled boxes, one at a time. A
box may contain more than one ball. The probability that both balls fall in
the same box is
7
(A) 4/ 2

(B) 1/4
(C) 1/16
(D) it depends on whether or not the balls are different colors
(E) none of the above.
S3.14 A student club has four officers, three of them are seniors and one of them
is a junior. Three of the students will be picked at random to go on a trip.
What is the probability that the junior stays home?
 (A) 1/4
6
(B) 1/ 3

(C) 1/4!
(D) 1/43
(E) none of the above.
S3.15 Suppose that five tiles labelled with the letters C, C, C, A, T are placed in a
container. If three tiles are selected from the container (without replacement),
what is the probability that they can be arranged to spell the word CAT?
(A) 1
(B) 1/3
5
(C) 1/ 3
5
(D) 3/ 2

(E) none of the above.

S3.16 Suppose A is any event. Then P (A) + P (Ac ) equals
(A) 0
(B) P (A)
(C) P (Ac )
(D) 1
(E) none of the above.
S3.17 If A ⊆ B with P (A) = P (B) > 0 then
(A) P (Ac ∩ B) = 0
(B) A = B
(C) Ac ∩ B = Ø
(D) all of the above are true
(E) none of the above are true.
S3.18 Which of the following statements are always true?
 (A) P (Ω) = 1
(B) If A ⊆ B then P (A) ≤ P (B)
(C) P (A ∩ B c ) ≤ P (A) for any event B
(D) all of the above are true.
(E) none of the above are true.

S3.19 Suppose P (A) = 1 and P (B) = 0.25. Which of the following statements
are always true?
(A) A and B are independent
(B) A ∩ B = B
(C) B ⊆ A
(D) all of the above are true.
(E) none of the above are true.
S3.20 A multiple choice test has 5 questions, with each question having 4 possible
answers. If a student randomly picks the answer to each question, what is
the probability that the student will answer all questions correctly?
(A) 1/(45 )
(B) 1/4
5
(C) 1/ 4
8
(D) 1/ 4

(E) none of the above.

S3.21 A fair die is tossed twice. Given that a 2 appears on the first toss, the
probability that the sum of the two tosses is 7 equals
(A) 1/36
(B) 1/18
(C) 1/9
(D) 1/6
(E) none of the above.
S3.22 If A ∩ B ∩ C = Ø then which of the following must be true
(A) A ∩ B = Ø
(B) P (A ∩ B) = 0
(C) P (A ∩ B ∩ C) = 0
(D) all of the above are true
(E) none of the above.
S3.23 Four fair dice are rolled simultaneously. The probability that exactly two of
them turn up with a value less than or equal to 4 is
(A) 24/81
(B) 1/3
(C) 4/9
(D) 1/2
(E) none of the above
S3.24 If P (A) ≤ P (Ac ) then
1
(A) 0 ≤ P (A) ≤ 2

(B) P (A) = 0
(C) A ⊆ Ac
(D) P (A) can be any value between 0 and 1
(E) none of the above.
1 1
S3.25 If P (A) = 4 and P (B) = 4 then
1
(A) P (A ∪ B) = 2
1
(B) P (A ∪ B) ≤ 2

(C) A ∩ B = Ø
(D) A = B
(E) none of the above.
S3.26 The probability that the birthdays of twelve people will fall in twelve dif-
ferent calendar months (assuming equal probabilities for the twelve months)
is
(A) 12!/(1212 )
(B) 1/(12!)
(C) 1/12
(D) 1/(1212 )
(E) none of the above.
S3.27 In a screw factor, there are three machines A, B and C that produce screws.
Machine A produces 20% of the daily output. Machine B produces 30% of
the daily output. Machine C produces the remaining 50% of the daily output.
Of their output, 5%, 4% and 2% of the screws are defective, respectively. A
screw is chosen at random from a day’s production and found to be defective.
What is the probability that it came from machine A?
1. 0.01
2. 0.032
3. 0.2
4. 0.3125
5. none of the above.
S3.28 A box contains fifty light bulbs, exactly five of which are defective. If five
bulbs are chosen at random from the box, the probability that all of them are
good is equal to
(A) (0.01)5
50
(B) 1/ 5
50
(C) 5/ 5

(D) (45!)2 /(40!50!)

(E) none of the above.
S3.29 Suppose A and B are independent events with P (A) = p and P (B) = q .
Then P (A ∩ B c ) equals
 (A) pq
(B) 1 − pq
(C) p − pq
(D) q − pq
(E) none of the above.
S3.30 Suppose A and B are independent events with P (A) = p and P (B) = q .
Then which of the following statements are true:
(A) P (A ∩ B) = pq
(B) P (A ∩ B c ) = p − pq
(C) P (Ac ∩ B) = q − pq
(D) all of the above are true
(E) none of the above are true.
S3.31 An urn contains 5 red balls and 5 black balls. One ball is drawn from the
urn at random and then thrown away. A second ball is then drawn from the
urn. The probability that the second ball is red equals
(A) 1/3
(B) 4/9
(C) 1/2
(D) 5/9
(E) none of the above.
S3.32 Suppose birthdays are assigned to people randomly with each of the 365
days of the year equally likely. A classroom has 364 students. What is the
probability that they all have different birthdays?
(A) 0
(B) 365!/(365)364
(C) 1/365
(D) (365!364!)/728!
(E) none of the above.
S3.33 A box contains 10 incandescent lamps. Five of them are good, five are bad.
The following circuit contains two such incandescent lamps in series and a
(good) battery sufficient to light both lamps:

Both lamps must be good in order for either to light. Two lamps are randomly
chosen from the box. The probability that the lamps will light is
(A) 2/9
(B) 1/4
(C) 1/2
(D) 3/4
(E) none of the above.
S3.34 Which of the following statements (A) (B) or (C) are not always true?
(A) P (E ∩ F ) = P (E)P (F ) for any events E and F
(B) P (Ω) = 1
(C) P (E) ≥ 0 for any event E
(D) all of the above are always true.
(E) none of the above are always true.
S3.35 Suppose

P (A | B) = 0.4 P (A | B c ) = 0.8 P (B) = 0.5

Then P (B | A) equals
(A) 0.2
(B) 1/3
(C) 0.5
(D) 0.6
 (E) none of the above.
S3.36 A study of automobile accidents produced the following data:

Probability of
Model Proportion of involvement
year all vehicles in an accident
2002 0.10 0.02
2003 0.20 0.01
Other 0.70 0.05

You know that an automobile has been involved in an accident. What is the
probability that the model year for this automobile is 2003?
(A) 0.0200
(B) 0.0390
(C) 0.0513
(D) 0.3333
(E) none of the above.
4
RANDOM VARIABLES

He stares in vain for what awaits him,

And sees in Love a coin to toss;
– EDWIN ARLINGTON ROBINSON, The Man Against the Sky (1921)

Basic Concepts
Definition 4.1. A random variable is a function X(·) which assigns to each
element ω of Ω a real value X(ω).
Notation:
 
(1) P {ω ∈ Ω : X(ω) = a} ≡ P {X(ω) = a} ≡ P (X = a)

Cumulative Distribution Function

Definition 4.2. For any random variable X , we define the cumulative distribu-
tion function (CDF), FX (a), as

FX (a) ≡ P (X ≤ a).

Properties of any cumulative distribution function

1. lim FX (a) = 1
a→∞

2. lim FX (a) = 0
a→−∞

3. FX (a) is a nondecreasing function of a.

4. FX (x) is a right-continuous function of a. In other words, lim FX (x) =
x↓a
FX (a)

81
 1

0.8

0.6

F(x)
0.4

0.2

0 1 2 3 4 5
x

Figure 4.1: An example of a CDF for a discrete random variable

Theorem 4.1. For any random variable X and real values a < b

P (a < X ≤ b) = FX (b) − FX (a)

Discrete Random Variables

Definition 4.3. The probability mass function (pmf) of a discrete random
variable X is given by
pX (xi ) ≡ P (X = xi ).

Properties of any probability mass function

1. pX (x) ≥ 0 for every x


2. pX (xi ) = 1
all xi

3. P (E) = pX (xi )
xi ∈E
Examples of discrete distributions

The binomial distribution

Consider a sequence of n independent, repeated trials, with each trial having two
possible outcomes, success or failure. Let p be the probability of a success for any
single trial. Let X denote the number of successes on n trials. The random variable
X is said to have a binomial distribution and has probability mass function
 
n k
(2) pX (k) = p (1 − p)(n−k) for k = 0, 1, . . . , n.
k

n
Let’s check to make sure that if X has a binomial distribution, then k=0 pX (k) =
1. We will need the binomial expansion for any polynomial:
n
 
 n k (n−k)
n
(a + b) = a b
k=0
k

So
n
 
 n k
p (1 − p)(n−k) = (p + (1 − p))n
k=0
k
= (1)n = 1

The geometric distribution

Recall the “success/failure” model that was used to derive the binomial distribution,
but, this time, we want to find the probability that the first success occurs on the
k th trial. Let p be the probability of a success for any single trial and let Y denote
the number of trials needed to obtain the first success.
Then the probability mass function for Y is given by

pY (k) = (1 − p)k−1 p for k = 1, 2, . . ..

and the random variable Y is said to have a geometric distribution with parameter
p where 0 ≤ p ≤ 1.
Some facts about geometric series When working with the geometric
distribution, the following results are often helpful:
Theorem 4.2. For any 0 < θ < 1,
∞
 1
θj =
j=0
1−θ

Proof.

∞
 ∞

θj − θj = 1
j=0 j=1
∞
 ∞

j j+1
θ0 = θ − θ = 1
j=0 j=0
∞
 ∞
θj − θ θj = 1
j=0 j=0
∞
(1 − θ) θj = 1
j=0
∞
1
θj =
j=0
1−θ

Theorem 4.3. For any 0 < θ < 1,

∞
 θ
jθj =
j=0
(1 − θ)2

Proof. Using Theorem 4.2

∞
 1
θj =
j=0
1−θ
∞
d d 1
θj =
dθ j=0
dθ 1 − θ
 ∞
 d j 1
θ =
j=0
dθ (1 − θ)2
∞
 1
jθj−1 =
j=1
(1 − θ)2
∞

−1 1
θ jθj =
j=1
(1 − θ)2
∞
θ
jθj =
j=1
(1 − θ)2
∞
θ
jθj =
j=0
(1 − θ)2

Example: Using Theorem 4.2, we can show that the probability mass function for
a geometric random variable must sum to one. Suppose

pY (k) = (1 − p)k−1 p for k = 1, 2, . . ..

then
∞
 ∞

pY (k) = (1 − p)k−1 p
k=1 k=1
∞
= (1 − p)j p
j=0
1
= p=1
1 − (1 − p)

The Poisson distribution

The random variable X is said to have a Poisson distribution with parameter

α > 0 if X has a probability mass function given by

αk
(3) pX (k) = e−α for k = 0, 1, 2, . . .
k!

Continuous Random Variables

 1

0.8

0.6

F(x)
0.4

0.2

0 1 2 3 4 5
x

Figure 4.2: An example of a CDF for a continuous random variable

Definition 4.4. The probability density function (pdf) of a continuous random

variable X is given by
d
fX (a) ≡ FX (a).
da

Properties of any probability density function

1. fX (x) ≥ 0 for every x

 ∞
2. fX (x) dx = 1
−∞

3. P (E) = fX (x) dx
E

Theorem 4.4. If X is a continuous random variable then

P (X < a) = P (X ≤ a).

Theorem 4.5. If X is a continuous random variable, then

 a
(4) FX (a) = fX (x) dx.
−∞
 Area = f(x)dx
f(x)

x x+dx

Figure 4.3: The differential mass interpretation of fX (x) dx

Differential mass interpretation of the pdf

(5) P (X ∈ [x, x + dx)) = fX (x)dx

Comparison of pmf’s and pdf’s

A comparison of the properties of the probability mass function and probability

density function.

pmf pdf

pX (x) ≥ 0 fX (x) ≥ 0

P (X = x) = pX (x) P (x ≤ X < x + dx) = fX (x) dx

  ∞
pX (x) = 1 fX (x) dx = 1
all x −∞

  a
FX (a) = pX (x) FX (a) = fX (x) dx
x≤a −∞

 
pX (x) = P (E) fX (x) dx = P (E)
x∈E E

Examples of continuous distributions

The uniform distribution

The random variable X is said to have a uniform distribution if it has the proba-
bility density function given by
⎧
1⎨
if a ≤ x ≤ b
(6) fX (x) = b−a
⎩
0 otherwise

The exponential distribution

The random variable X is said to have an exponential distribution with parameter

α > 0 if it has the probability density function given by

αe−αx if x ≥ 0
(7) fX (x) =
0 otherwise
Expectation
Definition 4.5. The expected value of a random variable X is given by

E(X) = xpX (x)
all x

+ for discrete X , and  +∞

E(X) = xfX (x) dx
−∞
for continuous X .
Note:
• E(X) is a number.
• The expected value of X is often called the “mean of X .”
• The value of E(X) can be interpreted as the center of mass or center of
gravity of the probability mass distribution for X .
Example: Roll a fair die once. Let X equal the number of pips. Then

E(X) = xpX (x)
all x
= (1)pX (1) + (2)pX (2) + (3)pX (3) + (4)pX (4) + (5)pX (5) + (6)pX (6)
= (1) 16 + (2) 16 + (3) 16 + (4) 16 + (5) 16 + (6) 16 = 3.5

Example: Let X be a continuous random variable with probability density function


2x 0 ≤ x ≤ 1
fX (x) =
0 otherwise

Then
 +∞
E(X) = xfX (x) dx
−∞
 0  1  +∞
= x0 dx + x2x dx + x0 dx
−∞ 0 1
 1 1
2 2x3  2 2
= 2x dx =  = −0=
0 3 0 3 3
Finding E(X) for some specific distributions
Binomial distribution

Let X be a binomial random variable with parameters n and 0 ≤ p ≤ 1. We have

 
n k
pX (k) = p (1 − p)n−k for k = 0, 1, . . . , n.
k

We can find E(X) as follows:

n
 
 n k
E(X) = k p (1 − p)n−k
k=0
k
n
n!
= pk (1 − p)n−k
k=1
(k − 1) !(n − k)!

Let s = k − 1
n−1
 

n − 1 s+1
E(X) = n p (1 − p)(n−s−1)
s=0
s
n−1
 
 n−1 s
= np p (1 − p)(n−s−1)
s=0
s
= np

Poisson distribution

Let X be a Poisson random variable with parameter α > 0. We have

αk
pX (k) = e−α for k = 0, 1, 2, . . .
k!
The expected value for X can be found as follows:
∞
 αk
E(X) = ke−α
k=0
k!
∞
αk
= e−α
k=1
(k − 1)!
 Let s = k − 1
∞
 αs+1
E(X) = e−α
s=0
s!
∞
 αs
= α e−α
s=0
s!
= α

Geometric distribution

Let X be a geometric random variable with parameter 0 ≤ p ≤ 1. We have

pX (k) = (1 − p)k−1 p for k = 1, 2, . . ..

To find E(X) let q = (1 − p) and then,

∞

E(X) = kq k−1 p
k=1
∞
= p kq k−1
k=1
1 1
= p =
(1 − q)2 p

Exponential distribution

Let X be an exponential random variable with parameter α > 0. We have


αe−αx if x ≥ 0
fX (x) =
0 otherwise

We can get E(X) as follows:

 ∞
E(X) = xαe−αx dx
0

Integrate this by parts, letting dv = αe−αx dx and x = u. Hence v = −e−αx and

du = dx. This produces
 ∞
∞ 1
E(X) = −xe−αx 0 + e−αx dx =
0 α
Other measures of central tendency
Definition 4.6. The median of a random variable X , denoted by m 1 (X), is given
2
by
1
P (X ≤ m 1 (X)) = .
2 2

Definition 4.7. The mode of a random variable X , denoted by Mode(X), is

given by
pX (Mode(X)) = max pX (x) if X is discrete
x

fX (Mode(X)) = sup fX (x) if X is continuous

x

Self-Test Exercises for Chapter 4

For each of the following multiple-choice questions, choose the best response
among those provided. Answers can be found in Appendix B.
S4.1 If X is a discrete random variable with

P (X = −1) = P (X = 0) = P (X = 1) = 1/3

Then the expected value of X , E(X), equals

(A) 0
(B) 1/3
(C) 1/2
(D) any of the values in the set {−1, 0, 1}
(E) none of the above are true.
S4.2 Suppose X is a continuous random variable with cumulative distribution
function ⎧
⎪
⎨ 0 if x < 0
FX (x) = x2 if 0 ≤ x ≤ 1
⎪
⎩ 1 if x > 1
Then P (X < 0.50) equals
(A) 0
 (B) 0.25
(C) 0.50
√
(D) 0.50
(E) none of the above.
S4.3 Suppose X is a random variable with cumulative distribution function
⎧
⎪
⎪ 0 if x < 0
⎪
⎨ 1/3 if 0 ≤ x < 1
FX (x) =
⎪
⎪
⎪
2/3 if 1 ≤ x < 2
⎩
1 if x ≥ 2
Then P (X = 2) equals
(A) 0
(B) 1/3
(C) 2/3
(D) 1
(E) none of the above.
S4.4 Suppose X is a continuous random variable with probability density function

1/2 if 0 ≤ x ≤ c
fX (x) =
0 otherwise
Then the value of c is
(A) 1/2
(B) 1
(C) 2
(D) +∞
(E) none of the above.
S4.5 Let X be a continuous random variable with probability density function

λe−2x if x > 0
fX (x) =
0 otherwise
The value of λ must equal
 (A) 1/2
(B) 1
(C) 2
(D) e2x
(E) none of the above.
S4.6 Suppose a fair coin is flipped 3 times. The probability that exactly two heads
turn up is
1
(A) 2
1
(B) 4
3
(C) 8
2
(D) 3

(E) none of the above.

S4.7 Let X be a random variable with cumulative distribution function FX (a).
Which of the following statements are always true:
(A) lim FX (a) = 0
a→∞

(B) P (X = a) = FX (a) − FX (a − 1)
(C) 0 ≤ FX (a) ≤ 1 for every a
(D) all of the above are true
(E) none of the above are true.
S4.8 The length X (in inches) of a steel rod is a random variable with probability
density function given by
⎧
⎨ x−9
if 9 ≤ x ≤ 11
fX (x) =
⎩ 0 2 otherwise

A rod is considered defective if its length is less than 10 inches. If 2

independent steel rods are examined, the probability that exactly one is
defective is
(A) 1/4
 (B) 3/8
(C) 1/16
(D) 3/16
(E) none of the above.
S4.9 An airplane can hold 100 passengers. Passenger behavior has been exten-
sively studied over the years. It is known that the probability is 0.01 that a
person making an airline reservation will never show up for his or her flight.
Assume that a person decides independently whether or not to show up for
a flight. If the airline permits 101 persons to make reservations for a given
flight, the probability that someone will be denied a seat is
(A) 0
(B) 0.01
(C) (0.99)101
(D) 1
(E) none of the above.
S4.10 Let X be a continuous random variable with probability density function

αe−αx if x ≥ 0
fX (x) =
0 otherwise

If P (X > 1) = e−2 , then α must equal

(A) −1
(B) 1/2
(C) 1
(D) 2
(E) none of the above.
S4.11 Let X be a continuous random variable with probability density function

x/2 if 0 ≤ x ≤ 2
fX (x) =
0 otherwise

Then P (X ≤ 1) equals
 (A) 1/8
(B) 1/4
(C) 1/3
(D) 1/2
(E) none of the above.
S4.12 Let X be a discrete random variable with probability mass function

e−1
P (X = k) = for k = 0, 1, 2, . . .
k!
Then P (X > 1) equals
(A) e−1
(B) 1 − e−1
(C) 1 − 2e−1
(D) 1
(E) none of the above.
S4.13 You have two light bulbs. The life length of each bulb (in hours) is a
continuous random variable X with probability density function

2e−2x if x > 0
fX (x) =
0 otherwise

The two bulbs are allowed to burn independently of each other. The proba-
bility that both bulbs will each last at least one hour is
(A) 0
(B) (1 − e−2 )2
(C) e−4
(D) 2e−2
(E) none of the above.
S4.14 Let X be a continuous random variable with cumulative distribution function
⎧
⎪
⎨ 0 if x < −1
1
FX (x) = 2 (x + 1) if −1 ≤ x ≤ 1
⎪
⎩ 1 if x > 1

Let fX (x) denote the probability density function for X . The value of fX (0)
is
(A) 0.00
(B) 0.50
(C) 0.75
(D) 1.00
(E) none of the above.
S4.15 Let X be a random variable with cumulative distribution function FX (x).
Suppose FX (0) = 0.3 and FX (1) = 0.5. Then P (0 < X ≤ 1) equals
(A) 0
(B) 0.2
(C) 0.3
(D) 0.5
(E) none of the above.
S4.16 Let X be a continuous random variable with probability density function

3x2 if 0 ≤ x ≤ 1
fX (x) =
0 otherwise

Let E(X) denote the expected value of X . The value of E(X) is

(A) 0
(B) 3/8
(C) 3/4
(D) 3/2
(E) none of the above.
S4.17 Suppose that an integrated circuit has a life length X (in units of 1000 hours)
that is a continuous random variable with probability density function

3e−3x if x ≥ 0
fX (x) =
0 otherwise

An integrated circuit considered defective if its life length is less than 3000
hours. The life lengths of any two integrated circuits are independent random
variables. A batch of 100 integrated circuits are tested. The expected number
of defective items is
(A) e−3
(B) (1 − e−3 )
(C) (1 − e−9 )
(D) 100(1 − e−9 )
(E) none of the above.
S4.18 Let X be a discrete random variable with cumulative distribution function,
F (a) ≡ P (X ≤ a), given by the following graph:

0.8

0.6
FX(a)

0.4

0.2

-2 -1 0 1 2 3
a

The value of P (−1 ≤ X ≤ 1) is

(A) 0.2
(B) 0.4
(C) 0.6
(D) 0.8
5
FUNCTIONS OF RANDOM VARIABLES

When you do dance, I wish you

A wave o’the sea, that you might ever do
Nothing but that, move still, still so,
And own no other function.
– WILLIAM SHAKESPEARE, The Winter’s Tale (c 1600)

Basic Concepts
Two of the primary reasons for considering random variables are:
1. They are convenient, and often obvious, representations of experimental out-
comes.
2. They allow us to restrict our attention to sample spaces that are subsets of the
real line.
There is a third, and probably more important reason for using random variables:
3. They permit algebraic manipulation of event definitions.
This chapter deals with this third “feature” of random variables. As an example,
let X be the random variable representing the Buffalo, New York temperature
at 4:00 pm next Monday in degrees Fahrenheit. This random variable might be
suitable for the use of Buffalonians, but our neighbors in Fort Erie, Ontario prefer
the random variable Y representing the same Buffalo temperature, except in degrees
Celsius. Furthermore, they are interested in the event

A = {ω ∈ Ω : Y (ω) ≤ 20}

where Ω is the sample space of the underlying experiment. Note that Ω is being
mapped by both of the random variables X and Y into the real line.
In order to compute P (A), we could evaluate the cumulative distribution function
for Y , FY (y), at the point y = 20. But suppose we only know the cumulative
distribution function for X , namely FX (x). Can we still find P (A)?

105
The answer is “yes.” First, notice that for any ω ∈ Ω,
5

(1) Y (ω) = 9 X(ω) − 32 .

We can then redefine the event A in terms of X as follows:

A = {ω ∈ Ω : Y (ω) ≤ 20}
 
5
= ω∈Ω : 9 (X(ω) − 32) ≤ 20
= {ω ∈ Ω : X(ω) − 32 ≤ 36}
= {ω ∈ Ω : X(ω) ≤ 68} ,

So, knowing a relationship like Equation 1 between X(ω) and Y (ω) for every
ω ∈ Ω, allows a representation of the event A in terms of either the random
variable X or the random variable Y .
When finding the probability of the event A, we can use the shortcut notation
presented in Chapter 5:

P (A) = P (Y ≤ 20)
= P ( 59 (X − 32) ≤ 20)
= P (X − 32 ≤ 36)
= P (X ≤ 68)
= FX (68).

So, if we know the probability distribution for X , we can compute probabilities of

events expressed in terms of the random variable Y .
The general concept surrounding the above example is the notion of a function of
a random variable. To show you what we mean by this, let the function h(x) be
defined by
h(x) = 59 (x − 32),
which is nothing more than the equation of a straight line. If X is a random variable,
we also have
Y = h(X)
as a random variable.1 The function h(x) maps merely points in the real line (the
sample space for X ) into points in the real line (the sample space for Y ) as in
Figure 5.1.
1
Note that we should have really written this as Y (ω) = h(X(ω)). In order to simplify the
notation, mathematicians usually will drop the (ω).
 Ω

X( )
Real line

Y( ) 212
Real line

100

Figure 5.1: The mapping Y (ω) = h(X(ω))

The random variable X serves as a “middleman” in the mapping shown in Fig-

ure 5.1. It may seem, at first, more obvious to define Y as mapping directly from
Ω to the real numbers. But consider the case where we have already defined X
and then are given a functional relationship between X and Y . Under those cir-
cumstances, it would be very handy to have a tool that would enable us to compute
probabilities involving Y using the facts we already know about X .
Referring to Figure 5.2, you can see that what is happening is really quite simple.
Suppose we are given a cumulative distribution function for the random variable
X , the Buffalo temperature in degrees Fahrenheit. Using this, we can compute
P (X ≤ t) for any value of t. This gives us the cumulative distribution function
FX (t) and allows us to compute the probability of many other events.
Returning to our example, the folks in Fort Erie are asking us to compute P (Y ≤
20). To oblige, we ask ourselves, “What are all of the possible outcomes of
the experiment (i.e., measuring Monday’s 4 pm temperature) that would result in
announcing that the temperature in degrees Celsius in 20 or less?” Call this event
A and compute the probability of A from the distribution of the random variable
X.
An even more general situation appears in Figure 5.3. In this case, suppose we
are given a random variable X . Also assume that we are given the cumulative
 y

)
h(x
y=
20

x
68

Figure 5.2: Representing the event {Y ≤ 20}

y
y= h(x)

B
x

= h-1(B)

Figure 5.3: A more complicated example

 distribution function for X so that we are capable of computing the probability
measure of any subset of the sample space for X . If we then define the random
variable Y = h(X) (where h is the function shown in Figure 5.3) we can compute
the probability measure of any subset B in the sample space for Y by determining
which outcomes of X will be mapped by h into B . The probability of any of these
particular outcomes of X is, therefore, also the probability of observing Y taking
on a value in B .

Some examples
Discrete case

Example: Let X be a discrete random variable with probability mass function

1
pX (−2) = 5
1
pX (−1) = 5
1
pX (0) = 5
1
pX (1) = 5
1
pX (2) = 5

Let Y = X 2 . We would like to find the probability distribution for Y .

Solution: There is a pleasant fact that helps when working with discrete random
variables:
Theorem 5.1. A function of any discrete random variable is also a discrete
random variable.
Unfortunately, we cannot make such a statement for continuous random variables,
as we will see in a later example.
To solve the problem, we should first ask ourselves, “Since X can take on the
values −2, −1, 0, 1 or 2, what values can Y assume?” We see that the function
h(x) = x2 maps the point x = −2 into the point 4, x = −1 into 1, x = 0 into
0, x = 1 into 1 and x = 2 into 4. So all of the probability mass for the random
variable Y is assigned to the set {0, 1, 4}. Such a set is called the support for the
random variable Y .
Definition 5.1. Let A be the smallest subset of R such that P (Y ∈ A) = 1. Then
A is called the support of the distribution for the random variable Y .
 Since X will take on the value −2 with probability 15 and the value 2 with probability
1
5 , and since each of these mutually exclusive events will result in Y taking on the
value 4, we can say that

pY (4) = pX (−2) + pX (2) = 25 .

Similarly, pY (1) = 25 and pY (0) = 15 . Hence the probability distribution for Y is

given by the probability mass function:
1
pY (0) = 5
2
pY (1) = 5
2
pY (4) = 5

Continuous case

Example: Let X be a uniformly distributed random variable on the interval (−1, 1),
i.e., the density for X is given by

1
if −1 ≤ x ≤ 1
fX (x) = 2
0 otherwise

and the cumulative distribution function is

⎧
⎪
⎪ 0 if x < −1
⎨
x+1
FX (x) = if −1 ≤ x ≤ 1
⎪
⎪ 2
⎩
1 if x > 1.

Find the distribution for the random variable Y = X 2 .

Solution: For continuous random variables, it is easier to deal with the cumulative
distribution functions. Let’s try to find FY (y). First, we should ask ourselves,
“Since X can take on only values from −1 to +1, what values can Y take on?”
Since Y = X 2 , the answer is “Y can assume values from 0 to 1.” With no elaborate
computation, we now know that
⎧
⎪
⎨ 0 if y < 0
FY (y) = ? if 0 ≤ y ≤ 1
⎪
⎩ 1 if y > 1.
Now suppose that y is equal to a value in the only remaining unknown case, that is
0 ≤ y ≤ 1. We then have

FY (y) = P (Y ≤ y )
 
= P X2 ≤ y
 √ √
= P − y≤X≤ y
√  √
= FX y − FX − y
√ √
y+1 − y+1
= −
√ 2 2
= y.

So, ⎧
⎪
⎨ 0 if y < 0
√
FY (y) = y if 0 ≤ y ≤ 1
⎪
⎩ 1 if y > 1

Finding the probability density function for Y is now easy. That task is left to you.
Example: Let X be a uniformly distributed random variable on the interval (0, 2),
i.e., the density for X is given by

1
if 0 ≤ x ≤ 2
fX (x) = 2
0 otherwise

and the cumulative distribution function is

⎧
⎪
⎨ 0x
⎪ if x < 0
FX (x) = if 0 ≤ x ≤ 2
⎪
⎪ 2
⎩ 1 if x > 2

Let h(x) be the greatest integer less than or equal to x, i.e., let h be given by the
step function shown in Figure 5.4. Find the distribution for Y = h(X).
Solution: Since X takes on the values ranging continuously from 0 to 2, Y can take
on only the integers 0, 1 and 2 (why 2?). We immediately know that the random
variable Y is discrete, and we have our first example of a continuous random
variable being mapped into a discrete random variable. Look at Figure 5.4 again
and try to visualize why this happened.
To find the probability mass function for Y , first let’s try to find pY (0). We know
that Y takes on the value 0 when X takes on any value from 0 to 1 (but not including
 y
3

1
x
-2 -1 1 2 3 4

Figure 5.4: The greatest integer function

1). Hence, 
 1
1 x 1 1
pY (0) = dx = = .
0 2 2 0 2
Similarly,
 2
1 1
pY (1) = dx =
1 2 2
and  2
1
pY (2) =
dx = 0,
2 2
which is good thing because we ran out of probability mass after the first two
computations.
So we have the answer,
pY (0) = pY (1) = 12 .

Invertible Functions of Continuous Random Variables

First we need a few concepts about inverse functions from calculus:
Definition 5.2. A function h : R → R is called one-to-one, if h(a) = h(b)
implies a = b. A function h : R → R is called onto, if for every y ∈ R there exists
some x ∈ R such that y = h(x).
If a function h(x) is both one-to-one and onto, then we can define the inverse
function h−1 : R → R such that h−1 (y) = x if and only if h(x) = y .
Suppose h(x) has an inverse and is increasing (or decreasing). Then, if X is a
continuous random variable, and Y = h(X), we can develop a general relationship
between the probability density function for X and the probability density function
for Y .
If h(x) a strictly increasing function and h(x) has an inverse, then h−1 (x) is
nondecreasing. Therefore, for any a ≤ b, we have h−1 (a) ≤ h−1 (b). This
produces the following relationship between the cumulative distribution function
for X and the cumulative distribution function for Y :

FY (y) = P [Y ≤ y] = P [h(X) ≤ y]
FY (y) = P [h−1 (h(X)) ≤ h−1 (y)]
FY (y) = P [X ≤ h−1 (y)]
FY (y) = FX (h−1 (y))

To get the relationship between the densities, take the first derivative of both sides
with respect to y and use the chain rule:
d d
FY (y) = FX (h−1 (y))
dy dy
d
fY (y) = fX (h−1 (y)) h−1 (y)
dy

We have just proven the following theorem:

Theorem 5.2. Suppose X is a continuous random variable with probability
density function fX (x). If h(x) is a strictly increasing function with a differen-
tiable inverse h−1 (·), then the random variable Y = h(X) has probability density
function
d
fY (y) = fX (h−1 (y)) h−1 (y)
dy

We will leave it to you to find a similar result for the case when h(x) is decreasing
rather than increasing.
Note: This transformation technique is actually a special case of a more general
technique from calculus. If one considers the function Y = h(X) as a transfor-
mation (x) → (y), the monotonicity property of h allows us to uniquely solve
y = h(x). That solution is x = h−1 (y). We can then express the probability
density function of Y as

fY (y) = fX (h−1 (y))J(y)

where J(y) is the following 1 × 1 determinant:

   
 ∂x   ∂ −1 
J(y) =   =  h (y) = ∂ h−1 (y)
∂y   ∂y  ∂y
since h−1 (y) is also an increasing function of y and, thus, has a nonnegative first
derivative with respect to y .
In calculus, when solving transformation problems with n variables, the resulting
n × n determinant is called the Jacobian of the transformation.
Example: Let X be a continuous random variable with density for X given by

|x|/4 if −2 ≤ x ≤ 2
fX (x) =
0 otherwise

Let Y = h(X) = X 3 . Find the probability density function for Y without explicitly
finding the cumulative distribution function for Y .
Solution: The support for X is [−2, 2]. So the support for Y is [−8, 8].
1
Since h−1 (y) = y 3 , we can use Theorem 5.2 to get
d −1
fY (y) = fX (h−1 (y)) h (y)
dy
1 d 1
= fX (y 3 ) y3
dy
2
= fX (y 3 )( 13 )y − 3
1

1 1 1
When y < −8 or y > 8 we have y 3 < −2 or y 3 > 2. In those cases, fX (y 3 ) = 0
hence fY (y) is zero.
When −8 ≤ y ≤ 8
1 13 1 − 23
fY (y) = 4 |y |( 3 )y
1
3 √
1 1
= 12 |y | 3 y 2

1 − 13
= 12 |y |
 Summarizing  1
|y − 3 |/12 if −8 ≤ y ≤ 8
fY (y) =
0 otherwise

Expected values for functions of random variables

Theorem 5.3. Let Y = h(X).
If X is a discrete random variable with probability mass function pX (x), then

E(Y ) = E(h(X)) = h(x)pX (x)
all x

If X is a continuous random variable with probability density function fX (x), then

 +∞
E(Y ) = E(h(X)) = h(x)fX (x) dx
−∞

Question: Toss a coin. Let X = 1 if heads, X = 0 if tails. Let h(X) denote your
winnings. 
−5 x = 0
h(x) =
10 x = 1
Find E(h(X)), your expected winnings.
Question: Let X , the life length of a transistor, be a continuous random variable
with probability density function

2e−2x x ≥ 0
fX (x) =
0 otherwise

Suppose you earn ln(x) + x2 − 3 dollars if the transistor lasts exactly x hours. What
are your expected earnings?
Theorem 5.4. E(a) = a if a is a constant.
Proof. We will provide proofs for both the discrete and continuous cases.
We are really finding E(h(X)) where h(x) = a for all x.
Suppose X is a discrete random variable with probability mass function

pX (xk ) = P (X = xk ) = pk for k = 1, 2, . . ..
Then

E(h(X)) = h(xk )pk
k

E(a) = apk
k

= a pk = a(1) = a
k
Suppose X is a continuous random variable with probability density function
fX (x). Then
 ∞
E(h(X)) = h(x)fX (x) dx
−∞
 ∞
E(a) = afX (x) dx
−∞
 ∞
= a fX (x) dx = a(1) = a
−∞

Theorem 5.5. E(aX) = aE(X) if a is a constant.

Proof. We are finding E(h(X)) where h(x) = ax.
Suppose X is a discrete random variable with probability mass function
pX (xk ) = P (X = xk ) = pk for k = 1, 2, . . ..
Then

E(h(X)) = h(xk )pk
k

E(aX) = axk pk
k

= a xk pk = a(E(X))
k
Suppose X is a continuous random variable with probability density function
fX (x). Then
 ∞
E(h(X)) = h(x)fX (x) dx
−∞
 ∞
E(aX) = axfX (x) dx
−∞
 ∞
= a xfX (x) dx = a(E(X))
−∞
Theorem 5.6. E(X + b) = E(X) + b if b is a constant.
Proof. We are finding E(h(X)) where h(x) = x + b.
Suppose X is a discrete random variable with probability mass function

pX (xk ) = P (X = xk ) = pk for k = 1, 2, . . ..

Then

E(h(X)) = h(xk )pk
k
 
E(X + b) = (xk + b)pk = (xk pk + bpk )
k k
 
= xk pk + bpk
k k
 
= xk pk + b pk = E(X) + b(1)
k k

Suppose X is a continuous random variable with probability density function

fX (x). Then
 ∞
E(h(X)) = h(x)fX (x) dx
−∞
 ∞  ∞
E(X + b) = (x + b)fX (x) dx = (xfX (x) + bfX (x)) dx
−∞ −∞
 ∞  ∞
= xfX (x) dx + bfX (x) dx = E(X) + b(1)
−∞ −∞

Theorem 5.7. E[h1 (X) + h2 (X)] = E[h1 (X)] + E[h2 (X)] for any functions
h1 (X) and h2 (X)
Proof. We are finding E(h(X)) where h(x) = h1 (x) + h2 (x).
Suppose X is a discrete random variable with probability mass function

pX (xk ) = P (X = xk ) = pk for k = 1, 2, . . ..

Then

E(h(X)) = h(xk )pk
k
 
E[h1 (X) + h2 (X)] = [h1 (xk ) + h2 (xk )]pk
k

= [h1 (xk )pk + h2 (xk )pk ]
k
 
= h1 (xk )pk + h2 (xk )pk
k k
= E[h1 (X)] + E[h2 (X)]

Suppose X is a continuous random variable with probability density function

fX (x). Then
 ∞
E(h(X)) = h(x)fX (x) dx
−∞
 ∞
E[h1 (X) + h2 (X)] = (h1 (x) + h2 (x))fX (x) dx
−∞
 ∞
= (h1 (x)fX (x) + h2 (fX (x)) dx
−∞
 ∞  ∞
= h1 (x)fX (x) dx + h2 (x)fX (x) dx
−∞ −∞
= E[h1 (X)] + E[h2 (X)]

Notice in the all of the above proofs the close relationship of

Discrete X and Continuous

 ∞ X

and
k −∞

P (X = k) = pX (xk ) = pk and P (x ≤ X ≤ x + dx) = fX (x) dx

Also notice that Theorem 5.6 is really only a corollary to Theorem 5.7. Simply use
Theorem 5.7 with h1 (x) = x and h2 (x) = b and you get Theorem 5.6.

The kth Moment of X

If we let h(x) = xk and consider E[h(X)], we get the following definition:
Definition 5.3. The quantity E(X k ) is defined as the kth moment about the
origin of the random variable X .
Variance
Recall that E(X) is simply a number. If we let h(x) = (x − E(X))2 and compute
E(h(X)) we get a very useful result:
Definition 5.4. The variance of a random variable X is

Var(X) ≡ E[(X − E(X))2 ]

.
Note that 
Var(X) = (x − E(X))2 pX (x)
all x
for a discrete random variable X , and
 +∞
Var(X) = (x − E(X))2 fX (x) dx
−∞

for a continuous random variable X .

Note: The value of Var(X) is the second moment of the probability distribution
about the expected value of X . This can be interpreted as the moment of inertia of
the probability mass distribution for X .
Definition 5.5. The standard deviation of a random variable X is given by

σX ≡ Var(X).

Notation: We often write

E(X) = μX
2
Var(X) = σX

Theorem 5.8. Var(X) = E(X 2 ) − [E(X)]2

Proof. Using the properties of expected value, we get

Var(X) = E[(X − E(X))2 ]

= E[X 2 − 2XE(X) + [E(X)]2 ] expand the square
= E[X ] + E[−2XE(X)] + E[[E(X)]2 ]
2
from Theorem 5.7
2 2
= E[X ] − 2E(X)E[X] + E[[E(X)] ] from Theorem 5.5
 = E[X 2 ] − 2E(X)E[X] + [E(X)]2 from Theorem 5.4
2 2 2
= E[X ] − 2[E[X]] + [E(X)]
= E[X 2 ] − [E[X]]2

Theorem 5.9. Var(aX) = a2 Var(X)

Proof. Using Theorem 5.8, we get

Var(aX) = E[(aX)2 ] − [E(aX)]2

= E[a2 X 2 ] − [E(aX)]2
= a2 E[X 2 ] − [aE(X)]2 Using Theorem 5.5
2 2 2 2
= a E[X ] − a [E(X)]
= a2 (E[X 2 ] − [E(X)]2 )
= a2 Var(X) Using Theorem 5.8

Theorem 5.10. Var(X + b) = Var(X)

Proof. Using Definition 5.4, we get

Var(X + b) = E[((X + b) − E[X + b])2 ]

= E[(X + b − E[X] − b)2 ]
= E[(X − E[X])2 ]
= Var(X)

The proof of the following result is left to the reader:

Theorem 5.11. Let Y = h(X) then
 
Var(Y ) = E [h(X) − E(h(X))]2

Example: Suppose X is a continuous random variable with probability density

function 
3x2 if 0 ≤ x ≤ 1
fX (x) =
0 otherwise
 Let Y = h(X) = 1/X . Then
 1
E(Y ) = h(x)(3x2 ) dx
0
 1
1
= (3x2 ) dx
0 x

3 2 1 3
= x =
2 0 2
3
Then we can use E(Y ) = E(h(X)) = 2 to compute
 1
Var(Y ) = (h(x) − E(h(X)))2 (3x2 ) dx
0
 1 
1 3 2
= − (3x2 ) dx
0 x 2
 1 
1 3 9
= − + (3x2 ) dx
0 x2 x 4
 1  1  1
27 2 3
= 3 dx − 9x dx + x dx =
0 0 0 4 4

Approximating E(h(X)) and Var(h(X))

This section is based on material in a text by P. L. Meyer.2
We have already shown that, given a random variable X and function Y = h(X),
we can compute E(Y ) = E(h(X)) directly using only the probability distribution
for X . This avoids the need for explicitly finding the probability distribution for
the random variable Y = h(X) if we are only computing expected values.
However, if the function h(·) is very complicated, then even using these short-cut
methods for computing E(h(X)) and Var(h(X)) may result in difficult integrals.
Often, we can get good results using the following approximations:
Theorem 5.12. Let X be a random variable with E(X) = μ and Var(X) = σ 2 .
Suppose Y = h(X). Then
h (μ) 2
(2) E(Y ) ≈ h(μ) + σ
2
(3) Var(Y ) ≈ [h (μ)]2 σ 2
2
Meyer, P., Introductory probability theory and statistical applications, Addison-Wesley, Reading
MA, 1965.
Proof. (This is only a sketch of the proof.)
Expand the function h in a Taylor series about x = μ to two terms. We obtain,

(X − μ)2 h (μ)

Y = h(μ) + (X − μ)h (μ) + + R1
2
where R1 is the remainder of the expansion. Discard the remainder term R1 and
take the expected value of both sides to get Equation (2).
For the second approximation, expand h in a Taylor series about x = μ to only one
term to get
Y = h(μ) + (X − μ)h (μ) + R2
where R2 is the remainder. Discard the term R2 and take the variance of both sides
to get Equation (3).
A common mistake when using expected values is to assume that E(h(X)) =
h(E(X)). This is not generally true.
For example, suppose we are selling a product, and the random variable X repre-
sents the monthly demand (in terms of the number of items sold). Then E(X) = μ
represents the expected number of items sold in a month.
Now suppose, Y = h(X) is the actual profit from selling X items in a month.
Often, the number h(μ) is incorrectly interpreted as the expected profit. That is,
the expected sales figure is “plugged into” the profit function, believing the result
is the expected profit.
In fact, the expected profit is actually E(Y ) = E(h(X)). Unless h is linear, this
will not be the same as h(μ) = h(E(X)).
Equation (2) in Theorem 5.12 provides an approximation for the error if you
incorrectly use h(μ) rather than E(h(X)).

h (μ) 2
E(Y ) ≈ h(μ) + σ
2
h (μ) 2
h(μ) − E(Y ) ≈ − σ
2
Therefore, if you incorrectly state that h(μ) is your expected monthly profit, your
answer will be wrong by approximately
h (μ) 2
Δ=− σ
2
Note that if h is linear, h (x) = 0, and Δ = 0. This result for the linear case is also
provided by Theorems 5.5, 5.6 and 5.7.
Example: (Meyer) Under certain conditions, the surface tension of a liquid
(dyn/cm) is given by the formula S = 2(1 − 0.005T )1.2 , where T is the tem-
perature of the liquid (degrees centigrade).
Suppose T is a continuous random variable with probability density function

3000t−4 if t ≥ 10
fX (x) =
0 otherwise

Therefore,
 ∞
E(T ) = 3000t−3 dt = 15 (degrees centigrade)
10
Var(T ) = E(T 2 ) − (15)2
 ∞
= 3000t−2 dt − 225 = 75 (degrees centigrade)2
10

In order to use Equations (2) and (3), we will use

h(t) = 2(1 − 0.005t)1.2

h (t) = −0.012(1 − 0.005t)0.2
h (t) = 0.000012(1 − 0.005t)−0.8

Since μ = 15, we have

h(15) = 1.82
h (15) = 0.01
h (15) = 0.000012(1 − 0.005(15))−0.8 ≈ 0+

and, using Theorem 5.12, results in

E(S) ≈ h(15) + 75h (15) = 1.82 (dyne/cm)

 2
Var(S) ≈ 75[h (15)] = 0.87 (dyne/cm)2
Chebychev’s Inequality
Chebychev’s inequality, given in the following Theorem 5.13, is applicable for
any random variable X with finite variance. For a given Var(X), the inequality
establishes a limit on the amount of probability mass that can be found at distances
far away from E(X) (the boondocks3 of the probability distribution).
Theorem 5.13. Let X be a discrete or continuous random variable with E(X) =
μ and Var(X) = σ 2 . Then for any a > 0

σ2
P (|X − μ| ≥ a) ≤
a2

Proof. We will prove this result when X is a continuous random variable. (The
proof for X discrete is similar using summations rather than integrals, and fX (x) dx
replaced by the probability mass function for X .)
We have for any a > 0
 +∞
σ2 = (x − μ)2 fX (x) dx
−∞
 μ−a  μ+a  +∞
2 2
= (x − μ) fX (x) dx + (x − μ) fX (x) dx + (x − μ)2 fX (x) dx
−∞ μ−a μ+a
 μ−a  +∞
= (x − μ)2 fX (x) dx + something ≥ 0 + (x − μ)2 fX (x) dx
−∞ μ+a
 μ−a  +∞
2
≥ (x − μ) fX (x) dx + (x − μ)2 fX (x) dx
−∞ μ+a

 μ+a
since (x − μ)2 fX (x) dx ≥ 0.
μ−a

Note that (x − μ)2 ≥ a2 for every x ∈ (−∞, μ − a] ∪ [μ + a, +∞). Hence, for all
x over this region of integration, we have (x − μ)2 fX (x) ≥ a2 fX (x). Therefore,
 μ−a  μ−a
2
(x − μ) fX (x) dx ≥ a2 fX (x) dx and
−∞ −∞
 +∞  +∞
(x − μ)2 fX (x) dx ≥ a2 fX (x) dx.
μ+a μ+a
3
boon·docks (bōōn’dŏks’) plural noun. Rural country; the backwoods. Example usage: The
1965 Billy Joe Royal Song: “Down in the Boondocks;” Source: The American HeritageR Dictionary
of the English Language, Fourth Edition, Houghton Mifflin Company.
So, we get
 μ−a  +∞
σ2 ≥ (x − μ)2 fX (x) dx + (x − μ)2 fX (x) dx
−∞ μ+a
 μ−a  +∞
≥ a2 fX (x) dx + a2 fX (x) dx
−∞ μ+a
2 2
= a P (X ≤ μ − a) + a (P (X ≥ μ + a)
= a2 P (|X − μ| ≥ a)

Note: X can be discrete. An equivalent way to state Chebychev’s inequality is

Let X be a discrete or continuous random variable with E(X) = μ
and Var(X) = σ 2 . Then for any a > 0

σ2
P (|X − μ| < a) ≥ 1 −
a2

Example:4
The number of customers who visit a car dealer’s showroom on a Saturday morning
is a random variable, X . Using extensive prior experience, the manager, a former
EAS305 student, has determined that the mean and variance of X are 18 and 6.25
respectively.
What is the probability that there will be between 8 and 28 customers on any given
Saturday?
Solution We are given E(X) = μ = 18 and Var(X) = σ 2 = 6.25, so

P (8 < X < 28) = P (8 − 18 < X − 18 < 28 − 18)

= P (−10 < X − 18 < 10)
= P (|X − 18| < 10)
6.25
= P (|X − μ| < 10) ≥ 1 − = 0.9375
102
This is the best statement we can make without knowing more about the distribution
for X .
4
Thanks to Dr. Linda Chattin at Arizona State University for this example.
The Law of Large Numbers
This is an important application of Chebychev’s inequality.
Theorem 5.14. Let an experiment be conducted n times, with each outcome of
the experiment independent of all others. Let

Xn ≡ number of times the event A occurs

p ≡ P (A)

Then for any > 0   

 Xn 
lim P  − p < = 1.
n→∞ n

The above result is the weak law of large numbers. The theorem guarantees that,
as n gets large, the probability that the ratio Xn /n is observed to be “close” to p
approaches one.
There is a related result called the strong law of large numbers that makes a similar
(but stronger statement) about the entire sequence of random variables

{X1 , 12 X2 , 13 X3 , 14 X4 , . . .}

The strong law states that, with probability one, this sequence converges to p. In
this case, we often say that the sequence { n1 Xn } converges to p almost surely.

The Normal Distribution

Definition 5.6. The random variable X is said to have a normal distribution if
it has probability density function
 2
1 − 12 x−μ
fX (x) = √ e σ for −∞ < x < ∞
σ 2π

Note that X has two parameters, μ and σ .

Theorem 5.15. If X ∼ N (μ, σ 2 ) then E(X) = μ. and Var(X) = σ 2 .
We often say that “X is distributed normally with mean μ and variance σ 2 ”
We write
X ∼ N (μ, σ 2 )
One of the amazing facts about the normal distribution is that the cumulative
distribution function
 a  2
1 − 12 x−μ
(4) FX (a) = √ e σ dx
−∞ σ 2π
does not exist in closed form. In other words, there exists no function in closed
form whose derivative is

1 1 x−μ 2
fX (x) = √ e− 2 σ
σ 2π
The only way to compute probabilities for normally distributed random variables
is to evaluate the cumulative distribution function in Equation 4 using numerical
integration.
If X ∼ N (μ, σ 2 ), there is an important property of the normal distribution that
allows us to convert any probability statement, such as P (X ≤ a), to an equivalent
probability statement involving only a normal random variable with mean zero and
variance one. We can then refer to a table of N (0, 1) probabilities, and thus avoid
numerically integrating Equation 4 for every conceivable pair of values μ and σ .
Such a table is provided in Table I of Appendix A.
Definition 5.7. The standard normal random variable, Z , has probability
density function
1 1 2
φ(z) = √ e− 2 z
2π
i.e., Z ∼ N (0, 1).
We denote the cumulative distribution function for Z by Φ(a) = P (Z ≤ a).
Theorem 5.16. If X ∼ N (μ, σ 2 ), then the random variable

X −μ
Z= ∼ N (0, 1).
σ

Proof. Suppose X ∼ N (μ, σ 2 ). Therefore, the probability density function for X

is 
1 1 x−μ 2
fX (x) = √ e− 2 σ for −∞ < x < ∞
σ 2π
Define the function
x−μ
h(x) =
σ
Note that z = h(x) is an invertible, increasing function of x with

h−1 (z) = σz + μ
d −1
h (z) = σ
dz
Using Theorem 5.2, the probability density function for the random variable Z =
h(X) is

d −1
fZ (z) = fX (h−1 (z)) h (z)

dz

1 (σz+μ)−μ 2
1
= √ e− 2 σ σ for −∞ < σz + μ < ∞
σ 2π
1 1 2
= √ e− 2 z for −∞ < z < ∞
2π

Note that we couldn’t use the cumulative distribution function of X to find the
cumulative distribution function of Z since neither cumulative distribution function
exists in closed form.
X −μ
If X ∼ N (μ, σ 2 ), Theorem 5.16 tells us three things about Z =
σ
1. E(Z) = 0
2. Var(Z) = 1, and
3. Z has a normal distribution
Actually, the first two results would be true even if X did not have a normal
distribution:
Lemma 5.17. Suppose X is any random variable with finite mean E(X) = μ
and finite, nonzero variance Var(X) = σ 2 . Define the random variable

X −μ
Z= .
σ
Then E(Z) = 0 and Var(Z) = 1.
Proof. We have

X −μ
E(Z) = E
σ
 
1
= E (X − μ)
σ
1
= E[X − μ]
σ
1
= (E[X] − μ)
σ
1
= (0) = 0
σ
and

X −μ
Var(Z) = Var
σ

1
= Var (X − μ)
σ
1
= Var[X − μ]
σ2
1
= Var[X] = 1
σ2

So, it is no surprise that, in Theorem 5.16, E(Z) = 0 and Var(Z) = 1. What

is important in Theorem 5.16 is that if X is normally distributed, then the linear
transformation of X , namely,
X −μ
Z=
σ
is also normally distributed.
A distribution that has zero mean and unit variance is said to be in standard form.
When we perform the above transformation (i.e., Z = (X − μ)/σ ), we say we are
standardizing the distribution of X .)
With a little additional work we can show that id X is normally distributed, then
any linear function h(X) = aX + b is also normally distributed as long as a = 0.
This fact is provided by the following theorem:
Theorem 5.18. If X ∼ N (μ, σ 2 ), then for any constants a = 0 and b, the random
variable
Y = aX + b ∼ N (aμ + b, a2 σ 2 ).

Proof. This is just a special case of Theorem 5.2. If a > 0, then the function
Y = h(X) is strictly increasing and if a < 0, the function Y = h(X) is strictly
decreasing.
We know  2
1 − 12 x−μ
fX (x) = √ e σ
σ 2π
Suppose a > 0, then
FY (y) = P (Y ≤ y) = P (aX + b ≤ y)
 
y−b
= P X≤
a
 
y−b
= FX
a
Take derivatives on both sides and use the chain rule to obtain
 
d d y−b
fY (y) = FY (y) = FX
dy dy a
 
y−b 1
= fX
a a
 2
y−b
− 12 a −μ
1 1 σ
= √ e
σ 2π a

1 y−(aμ+b) 2
1
= √ e− 2 aσ
aσ 2π
Hence, Y ∼ N (aμ + b, a2 σ 2 ).
Now suppose a < 0, then
FY (y) = P (Y ≤ y) = P (aX + b ≤ y)
 
y−b
= P X≥
a
   
y−b y−b
= P X> +P X =
a a
   
y−b y−b
= 1 − FX +P X =
a a
= and since X is continuous.
 . .
y−b
= 1 − FX +0
a

Take derivatives on both sides and use the chain rule to obtain
  
d d y−b
fY (y) = FY (y) = 1 − FX
dy dy a
  
y−b 1
= −fX
a a
and remembering that a < 0 . . .
 2
y−b
− 12 a −μ
1 1 σ
= √ e
σ 2π −a
 y−(aμ+b) 2
1 1
= √ e− 2 aσ
−aσ 2π
and since a2 = (−a)2 . . .

1 y−(aμ+b) 2

1
= √ e− 2 −aσ
−aσ 2π
So, again, we have Y ∼ N (aμ + b, a2 σ 2 ).
Question: What happens to the probability distribution of Y = aX + b when
a = 0?
Linear transformations rarely preserve the “shape” of the probability distribution.
For example, if X is exponentially distributed, Z would not be exponentially
distributed. If X was a Poisson random variable, Z would not be a Poisson random
variable. But, as provided by Theorem 5.16, if X is normally distributed, then Z
is also normally distributed. (See Question 5.6 to see that the uniform distribution
is preserved under linear transformations.)

Tools for using the standard normal tables

P (Z ≤ a) = Φ(a) is given in the tables for various values of a. It can be shown

that since Z is a continuous random variable and the probability density function
for Z is symmetric about zero, we have

Φ(a) = 1 − Φ(−a).

Example: Suppose X ∼ N (0, 4). Let’s find P (X ≤ 1).

P (X ≤ 1) = P (X − 0 ≤ 1 − 0)
 
X −0 1−0
= P ≤
2 2
Since X ∼ N (0, 4) we know from Theorem 5.16 that
 
X −0
Z= ∼ N (0, 1)
2
so, using Table I in Appendix A, we have
 
1−0
P (X ≤ 1) = P Z ≤
2
 
1
= P Z≤ 2 = 0.6915

This is the area shown in the following figure:

0.5

0.4

0.3

φ(z )
φ(
0.2

0.1

-2.0 0.0

-1.8
-1.6
-1.4
-1.2
-1.1
-0.9
-0.7
-0.5
-0.3
-0.1
0.1
0.3
0.5
0.7
0.9
1.0
1.2
1.4
1.6
1.8
2.0
z

Example: Suppose X ∼ N (2, 9). Let’s find P (0 ≤ X ≤ 3).

P (0 ≤ X ≤ 3) = P (0 − 2 ≤ X − 2 ≤ 3 − 2)
 
0−2 X −2 3−2
= P ≤ ≤
3 3 3
Since X ∼ N (2, 9) we know that
 
X −2
Z= ∼ N (0, 1)
3
Hence,
 
P (0 ≤ X ≤ 3) = P − 23 ≤ Z ≤ 1
3
   
1
= P Z≤ 3 − P Z < − 23

Since the N (0, 1) distribution is symmetric about zero,

     
P Z < − 23 = P Z > 2
3 =1−P Z ≤ 2
3

Finally, from Table I, we get

    
1 2
P (0 ≤ X ≤ 3) = P Z ≤ 3 − 1−P Z ≤ 3
= 0.63 − (1 − 0.75) = 0.38
 which is the area shown in the following figure:

0.5

0.4

0.3

φ(z )
φ(
0.2

0.1

0.0

-2.0
-1.8
-1.6
-1.4
-1.2
-1.1
-0.9
-0.7
-0.5
-0.3
-0.1
0.1
0.3
0.5
0.7
0.9
1.0
1.2
1.4
1.6
1.8
2.0
z

Self-Test Exercises for Chapter 5

For each of the following multiple-choice questions, choose the best response
among those provided. Answers can be found in Appendix B.
S5.1 Let X be a continuous random variable with probability density function

1
if −1 ≤ x ≤ 1
fX (x) = 2
0 otherwise.

The value of P (X 2 ≥ 14 ) is
(A) 0
1
(B) 4
1
(C) 2

(D) 1
(E) none of the above.
S5.2 Let X be a random variable with E(X) = 0 and E(X 2 ) = 2. Then the value
of E[X(1 − X)] is
(A) −2
(B) 0
(C) 1
(D) 2
(E) Var(X)
 (F) none of the above.
S5.3 Let X be a random variable with E(X) = 1 and E(X 2 ) = 2. Then Var(X)
equals
(A) −1
(B) 0
(C) 1
(D) 3
(E) none of the above.
1
S5.4 Suppose that X is a continuous random variable with P (X < −2) = 4 and
P (X < 2) = 12 . Then P (X 2 ≤ 4) equals.
1
(A) 4
1
(B) 2
3
(C) 4

(D) 1
(E) none of the above.
S5.5 Let X be a random variable with E(X) = 2. Then E[(X − 2)2 ] equals
(A) 0
(B) 2
(C) Var(X)
(D) E(X 2 )
(E) none of the above.
S5.6 Let X be a random variable with cumulative distribution function
⎧
⎪
⎨ 0 if x < 0
FX (x) = x if 0 ≤ x ≤ 1
⎪
⎩ 1 if x > 1

Then P (3X ≤ 1) equals

(A) 1/6
 (B) 1/4
(C) 1/3
(D) 1/2
(E) none of the above.
S5.7 Let X be a discrete random variable with probability mass function

pX (−1) = p pX (0) = 1 − 2p pX (1) = p

where 0 < p < 12 . The value of P (X 2 = 1) is

(A) p2
(B) 2p
(C) (1 − 2p)2
(D) 1
(E) none of the above.
S5.8 Let X be a discrete random variable with probability mass function

1/4 if x = −2, −1, 1 or 2
pX (x) =
0 otherwise

Let Y = 1/X . Then P (Y ≤ 21 ) equals

(A) 0
(B) 1/4
(C) 3/4
(D) 1
(E) none of the above.
S5.9 Let X be a continuous random variable with probability density function

1/4 if −2 ≤ x ≤ 2
fX (x) =
0 otherwise

Let Y = 1/X . Then P (Y ≤ 21 ) equals

 (A) 0
(B) 1/2
(C) 3/4
(D) 1
(E) none of the above.
S5.10 Let X be a random variable with

P (X < −1) = 1/4

P (X ≤ +1) = 3/4

Then P (|X| > 1) equals

(A) 1/4
(B) 1/2
(C) 3/4
(D) 1
(E) none of the above.
S5.11 Let X be a continuous random variable with probability density function

1
if −1 ≤ x ≤ 1
fX (x) = 2
0 otherwise

Then P (|X| ≤ 14 ) equals

(A) 0
(B) 1/4
(C) 1/2
(D) 1
(E) none of the above.
S5.12 Suppose that X is a random variable with P (X > 0) = r where 0 < r < 1.
Then P (X 2 ≥ 0) equals
(A) 0
 √
(B) r
(C) r
(D) 1
(E) none of the above.
S5.13 Suppose X is a random variable with E(X) = α and Var(X) = α, where
α > 0. Then E(X 2 ) equals
(A) 0
(B) α
(C) α2
(D) α + α2
(E) none of the above.
S5.14 A fair coin is tossed 10 times, with each outcome independent of the others.
Suppose you win $2 for every head and lose $1 for every tail, Let Y denote
your actual net amount won (which may be negative). Then P (Y ≥ 0)
equals
(A) 2/3
(B) 848/1024
(C) 904/1024
(D) 1
(E) none of the above.
S5.15 The daily demand for chocolate cake in a neighborhood bakery is a random
variable X with probability mass function given by

pX (0) = 0.2 pX (1) = 0.5

pX (2) = 0.2 pX (3) = 0.1

where X represents the number of cakes requested by customers.

At the beginning of the day, the bakery owner bakes two chocolate cakes at
a cost of $2.00 per cake. The cake is sold for $5.00 a cake and all unsold
chocolate cakes are thrown away at the end of the day. The expected daily
profit from chocolate cake is equal to
 (A) 0
(B) $1.50
(C) $2.00
(D) $3.00
(E) none of the above.
S5.16 The daily demand for rye bread in a corner grocery store is a random variable
X with probability mass function given by

pX (0) = 0.2 pX (1) = 0.2 pX (2) = 0.6

At the beginning of the day, the store owner bakes two loaves of rye bread at
a cost of 20 cents per loaf. The bread is sold for 50 cents a loaf and all unsold
loaves are thrown away at the end of the day. The expected daily profit from
rye bread is equal to
(A) 0
(B) 20 cents
(C) 30 cents
(D) 40 cents
(E) none of the above.
S5.17 Suppose that an electronic device has a life length X (in units of 1000
hours) that is considered a continuous random variable with probability
density function

e−x if x > 0
fX (x) =
0 otherwise

Suppose that the cost of manufacturing one such item is $2.00. The man-
ufacturer sells the item for $5.00, but guarantees a total refund if X ≤ 1.
What is the manufacturer’s expected profit per item?
(A) 5e−1
(B) 2e−1
(C) −2 + 5e−1
 (D) 5 − 2e−1
(E) none of the above.
S5.18 Let X be a random variable with E(X) = 2 and E(X 2 ) = 5. Then Var(X)
equals
(A) 1
(B) 3
(C) 7
(D) 9
(E) none of the above.
S5.19 Let X be a random variable with E(X) = 1 and Var(X) = 4 then

E((X − 1)2 )

equals
(A) 0
(B) 1
(C) 4
(D) 16
(E) none of the above.
S5.20 A fair coin is tossed 3 times. You win $2 for each head that appears, and
you lose $1 for each tail that appears. Let Y denote your earnings after 3
tosses. The value of E(Y ) is
(A) 0
(B) 3/2
(C) 1
(D) 3
(E) none of the above.
S5.21 Let X be any random variable with E(X) = μ and Var(X) = σ 2 . Consider
the random variable
X −μ
Y = .
σ
Which of the following are always true:
(A) E(Y ) = 0 and Var(Y ) = 1
(B) Y has a normal distribution
(C) P (Y > 0) = 1/2
(D) all of the above are true.
(E) none of the above are true.
S5.22 Let X be a normally distributed random variable with E(X) = 3 and
Var(X) = 9. Then P (X < 0) equals
(A) 0
(B) 0.1587
(C) 0.5
(D) 0.8413
(E) none of the above
S5.23 If Y is a random variable with P (Y ≥ 0) = p, then P (−2Y < 0) equals
(A) p
(B) 1 − p
(C) 0.5
(D) 0
(E) none of the above.
S5.24 Let X be a random variable with

P (X = −1) = 0.5 and P (X = +1) = 0.5.

Then the value of E(|X|) is

(A) 0
 (B) 0.5
(C) 1
(D) 2
(E) none of the above are true.
S5.25 The time to wait for service in the El Gasso Mexican Restaurant is a con-
tinuous random variable X (in minutes) with probability density function

5e−5x x≥0
fX (x) =
0 otherwise

The restaurant will pay a customer a $10 reward if they wait more than 5
minutes for service. Let Y denote the reward for any one customer. The
value of E(Y ), the expected reward, is
(A) 0
(B) 5
(C) 10
(D) 10e−25
(E) none of the above.
S5.26 Suppose that the random variable X has a normal distribution with mean 1
and variance 4. Then, P (|X| ≤ 1) equals
(A) 0.1528
(B) 0.1826
(C) 0.3174
(D) 0.3413
(E) none of the above.
S5.27 Suppose X is a continuous random variable with cumulative distribution
function ⎧
⎪
⎨ 0 if x < 0
FX (x) = x2 if 0 ≤ x ≤ 1
⎪
⎩ 1 if x > 1
 Let ln(x) denote the natural logarithm of x. Define the random variable
Y = ln(X). Then P (Y < ln(0.50)) equals
(A) 0
(B) 0.25
(C) 0.50
√
(D) 0.50
(E) none of the above.
S5.28 The life length X (in years) of a printer costing $200 is a continuous random
variable with probability density function

e−x if x ≥ 0
fX (x) =
0 otherwise

The manufacturer agrees to pay a full refund to the buyer if the printer fails
within the first year. If the manufacturer sells 100 printers, how much should
he expect to pay in refunds?
(A) $2706.71
(B) $7357.59
(C) $12,642.41
(D) $17,293.29
(E) none of the above.
6
RANDOM VECTORS

Basic Concepts
Often, a single random variable cannot adequately provide all of the information
needed about the outcome of an experiment. For example, tomorrow’s weather
is really best described by an array of random variables that includes wind speed,
wind direction, atmospheric pressure, relative humidity and temperature. It would
not be either easy or desirable to attempt to combine all of this information into a
single measurement.
We would like to extend the notion of a random variable to deal with an experiment
that results in several observations each time the experiment is run. For example,
let T be a random variable representing tomorrow’s maximum temperature and
let R be a random variable representing tomorrow’s total rainfall. It would be
reasonable to ask for the probability that tomorrow’s temperature is greater than
70◦ and tomorrow’s total rainfall is less than 0.1 inch. In other words, we wish to
determine the probability of the event

A = {T > 70, R < 0.1}.

Another question that we might like to have answered is, “What is the probability
that the temperature will be greater than 70◦ regardless of the rainfall?” To answer
this question, we would need to compute the probability of the event

B = {T > 70}.

In this chapter, we will build on our probability model and extend our definition of
a random variable to permit such calculations.

Definition
The first thing we must do is to precisely define what we mean by “an array of
random variables.”

155
 y

ω 3

1 (4,1)

1 2 3 4 5
x
Ω

Figure 6.1: A mapping using (X1 , X2 )

Definition 6.1. Let Ω be a sample space. An n-dimensional random variable or

random vector, X1 (·), X 2
(·), . . . , Xn (·) , is a vector

of functions that assigns to
each point ω ∈ Ω a point X1 (ω), X2 (ω), . . . , Xn (ω) in n-dimensional Euclidean
space.
Example: Consider an experiment where a die is rolled twice. Let X1 denote the
number of the first roll, and X2 the number of the second roll. Then (X1 , X2 ) is a
two-dimensional random vector. A possible sample point in Ω is

that is mapped into the point (4, 1) as shown in Figure 6.1.

 (x,y)

Figure 6.2: The region representing the event P (X ≤ x, Y ≤ y).

Joint Distributions
Now that we know the definition of a random vector, we can begin to use it to assign
probabilities to events. For any random vector, we can define a joint cumulative
distribution function for all of the components as follows:
Definition 6.2. Let (X1 , X2 , . . . , Xn ) be a random vector. The joint cumulative
distribution function for this random vector is given by

FX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) ≡ P (X1 ≤ x1 , X2 ≤ x2 , . . . , Xn ≤ xn ).

In the two-dimensional case, the joint cumulative distribution function for the
random vector (X, Y ) evaluated at the point (x, y), namely

FX,Y (x, y),

is the probability that the experiment results in a two-dimensional value within the
shaded region shown in Figure 6.2.
Every joint cumulative distribution function must posses the following properties:
1. lim FX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = 0
all xi →−∞
 x2

6
5
4
3
2
1
x1
1 2 3 4 5 6

Figure 6.3: The possible outcomes from rolling a die twice.

2. lim FX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = 1

all xi →+∞

3. As xi varies, with all other xj ’s (j = i) fixed,

FX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn )

is a nondecreasing function of xi .
As in the case of one-dimensional random variables, we shall identify two ma-
jor classifications of vector-valued random variables: discrete and continuous.
Although there are many common properties between these two types, we shall
discuss each separately.

Discrete Distributions
A random vector that can only assume at most a countable collection of discrete
values is said to be discrete. As an example, consider once again the example on
page 156 where a die is rolled twice. The possible values for either X1 or X2 are in
the set {1, 2, 3, 4, 5, 6}. Hence, the random vector (X1 , X2 ) can only take on one
of the 36 values shown in Figure 6.3.
If the die is fair, then each of the points can be considered to have a probability
 1
mass of 36 . This prompts us to define a joint probability mass function for this type
of random vector, as follows:
Definition 6.3. Let (X1 , X2 , . . . Xn ) be a discrete random vector. Then

pX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) ≡ P (X1 = x1 , X2 = x2 , . . . , Xn = xn ).

is the joint probability mass function for the random vector (X1 , X2 , . . . , Xn ).
Referring again to the example on page 156, we find that the joint probability mass
function for (X1 , X2 ) is given by
1
pX1 ,X2 (x1 , x2 ) = 36 for x1 = 1, 2, . . . , 6 and x2 = 1, 2, . . . , 6

Note that for any probability mass function,

  
FX1 ,X2 ,...,Xn (b1 , b2 , . . . , bn ) = ··· pX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ).
x1 ≤b1 x2 ≤b2 xn ≤bn

Therefore, if we wished to evaluate FX1 ,X2 (3.0, 4.5) we would sum all of the
probability mass in the shaded region shown in Figure 6.3, and obtain
1
FX1 ,X2 (3.0, 4.5) = 12 36 = 13 .

This is the probability that the first roll is less than or equal to 3 and the second roll
is less than or equal to 4.5.
Every joint probability mass function must have the following properties:
1. pX1 ,X2 ,...,Xn (x1 , x2 , . . . xn ) ≥ 0 for any (x1 , x2 , . . . , xn ).
 
2. ... pX1 ,X2 ,...,Xn (x1 , x2 , . . . xn ) = 1
all x1 all xn

3. P (E) = pX1 ,X2 ,...,Xn (x1 , x2 , . . . xn ) for any event E .
(x1 ,...,xn )∈E

You should compare these properties with those of probability mass functions for
single-valued discrete random variables given in Chapter 5.

Continuous Distributions
Extending our notion of probability density functions to continuous random vectors
is a bit tricky and the mathematical details of the problem is beyond the scope of
an introductory course. In essence, it is not possible to define the joint density as a
derivative of the cumulative distribution function as we did in the one-dimensional
case.
Let Rn denote n-dimensional Euclidean space. We sidestep the problem by defining
the density of an n-dimensional random vector to be a function that when integrated
over the set

{(x1 , . . . , xn ) ∈ Rn : x1 ≤ b1 , x2 ≤ b2 , . . . , xn ≤ bn }

will yield the value for the cumulative distribution function evaluated at

(b1 , b2 , . . . , bn ).

More formally, we have the following:

Definition 6.4. Let (X1 , . . . , Xn ) be a continuous random vector with joint
cumulative distribution function

FX1 ,...,Xn (x1 , . . . , xn ).

The function fX1 ,...,Xn (x1 , . . . , xn ) that satisfies the equation

 b1  bn
FX1 ,...,Xn (b1 , . . . , bn ) = ··· fX1 ,...,Xn (t1 , . . . , tn ) dt1 · · · dtn
−∞ −∞

for all (b1 , . . . , bn ) is called the joint probability density function for the random
vector (X1 , . . . , Xn ).
Now, instead of obtaining a derived relationship between the density and the
cumulative distribution function by using integrals as anti-derivatives, we have
enforced such a relationship by the above definition.
Every joint probability density function must have the following properties:
1. fX1 ,X2 ,...,Xn (x1 , x2 , . . . xn ) ≥ 0 for any (x1 , x2 , . . . , xn ).
 +∞  +∞
2. ··· fX1 ,...,Xn (x1 , . . . , xn ) dx1 · · · dxn = 1
−∞ −∞
 
3. P (E) = ··· fX1 ,...,Xn (x1 , . . . , xn ) dx1 · · · dxn for any event E .
E
You should compare these properties with those of probability density functions
for single-valued continuous random variables given in Chapter 5.
 x2

b2

a2

x1
a1 b1

Figure 6.4: Computing P (a1 < X1 ≤ b1 , a2 < X2 ≤ b2 ).

In the one-dimensional case, we had the handy formula

P (a < X ≤ b) = FX (b) − FX (a).

This worked for any type of probability distribution. The situation in the multi-
dimensional case is a little more complicated, with a comparable formula given
by

P (a1 < X1 ≤ b1 , a2 < X2 ≤ b2 ) =

FX1 ,X2 (b1 , b2 ) − FX1 ,X2 (a1 , b2 ) − FX1 ,X2 (b1 , a2 ) + FX1 ,X2 (a1 , a2 ).

You should be able to verify this formula for yourself by accounting for all of the
probability masses in the regions shown in Figure 6.4.
Example: Let (X, Y ) be a two-dimensional random variable with the following
joint probability density function (see Figure 6.5):

2−y if 0 ≤ x ≤ 2 and 1 ≤ y ≤ 2
fX,Y (x, y) =
0 otherwise

Note that  2 2
(2 − y) dx dy = 1.
1 0
 fXY(x,y)

1 2
y

Figure 6.5: A two-dimensional probability density function

Suppose we would like to compute P (X ≤ 1.0, Y ≤ 1.5). To do this, we calculate

the volume under the surface fX,Y (x, y) over the region {(x, y) : x ≤ 1, y ≤ 1.5}.
This region of integration is shown shaded (in green) in Figure 6.5. Performing the
integration, we get,
 1.5  1.0
P (X ≤ 1.0, Y ≤ 1.5) = fX,Y (x, y) dx dy
−∞ −∞
 1.5  1.0
= (2 − y) dx dy = 38 .
1.0 0

Marginal Distributions
Given the probability distribution for a vector-valued random variable (X1 , . . . Xn ),
we might ask the question, “Can we determine the distribution of X1 , disregarding
the other components?” The answer is yes, and the solution requires the careful
use of English rather than mathematics.
For example, in the two-dimensional case, we may be given a random vector (X, Y )
with joint cumulative distribution function FX,Y (x, y). Suppose we would like to
find the cumulative distribution function for X alone, i.e., FX (x)? We know that

FX,Y (x, y) = P (X ≤ x, Y ≤ y)
and we are asking for
(1) FX (x) = P (X ≤ x).
But in terms of both X and Y , expression 1 can be read: “the probability that X
takes on a value less than or equal to x and Y takes on any value.” Therefore, it
would make sense to say

FX (x) = P (X ≤ x)
= P (X ≤ x, Y ≤ ∞)
= lim FX,Y (x, y).
y→∞

Using this idea, we shall define what we will call the marginal cumulative distri-
bution function:
Definition 6.5. Let (X1 , . . . , Xn ) be a random vector with joint cumulative dis-
tribution function FX1 ,...,Xn (x1 , . . . , xn ). The marginal cumulative distribution
function for X1 is given by

FX1 (x1 ) = lim lim · · · lim FX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ).

x2 →∞ x3 →∞ xn →∞

Notice that we can renumber the components of the random vector and call any one
of them X1 . So we can use the above definition to find the marginal cumulative
distribution function for any of the Xi ’s.
Although Definition 6.5 is a nice definition, it is more useful to examine marginal
probability mass functions and marginal probability density functions. For exam-
ple, suppose we have a discrete random vector (X, Y ) with joint probability mass
function pX,Y (x, y). To find pX (x), we ask “What is the probability that X = x
regardless of the value that Y takes on? This can be written as

pX (x) = P (X = x) = P (X = x, Y = any value)


= pX,Y (x, y).
all y

Example: In the die example (on page 156)

1
pX1 ,X2 (x1 , x2 ) = 36 for x1 = 1, 2, . . . , 6 and x2 = 1, 2, . . . , 6

To find pX1 (2), for example, we compute


6
pX1 (2) = P (X1 = 2) = pX1 ,X2 (2, k) = 16 .
k=1
 Table 6.1: Joint pmf for daily production

Y
pX,Y (x, y) 1 2 3 4 5
1 .05 0 0 0 0
2 .15 .10 0 0 0
X 3 .05 .05 .10 0 0
4 .05 .025 .025 0 0
5 .10 .10 .10 .10 0

This is hardly a surprising result, but it brings some comfort to know we can get it
from all of the mathematical machinery we’ve developed thus far.
Example: Let X be the total number of items produced in a day’s work at a factory,
and let Y be the number of defective items produced. Suppose that the probability
mass function for (X, Y ) is given by Table 6.1. Using this joint distribution, we
can see that the probability of producing 2 items with exactly 1 of those items being
defective is
pX,Y (2, 1) = 0.15.

To find the marginal probability mass function for the total daily production, X ,
we sum the probabilities over all possible values of Y for each fixed x:

pX (1) = pX,Y (1, 1) = 0.05

pX (2) = pX,Y (2, 1) + pX,Y (2, 2) = 0.15 + 0.10 = 0.25
pX (3) = pX,Y (3, 1) + pX,Y (3, 2) + pX,Y (3, 3) = 0.05 + 0.05 + 0.10 = 0.20
etc.

But notice that in these computations, we are simply adding the entries all columns
for each row of Table 6.1. Doing this for Y as well as X we can obtain Table 6.2
So, for example, P (Y = 2) = pY (2) = 0.275. We simply look for the result in
the margin1 for the entry Y = 2.
The procedure is similar for obtaining marginal probability density functions. Re-
call that a density, fX (x), itself is not a probability measure, but fX (x)dx, is. So
1
Would you believe that this is why they are called marginal distributions?
 Table 6.2: Marginal pmf’s for daily production

Y
pX,Y (x, y) 1 2 3 4 5 pX (x)
1 .05 0 0 0 0 .05
2 .15 .10 0 0 0 .25
X 3 .05 .05 .10 0 0 .20
4 .05 .025 .025 0 0 .10
5 .10 .10 .10 .10 0 .40
pY (y) 0.4 .275 .225 .10 0

with a little loose-speaking integration notation we should be able to compute

fX (x) dx = P (x ≤ X < x + dx)

= P (x ≤ X < x + dx, Y = any value)
 +∞
= fX,Y (x, y) dy dx
−∞

where y is the variable of integration in the above integral. Looking at this

relationship as
 +∞
fX (x) dx = fX,Y (x, y) dy dx
−∞

it would seem reasonable to define

 +∞
fX (x) ≡ fX,Y (x, y) dy
−∞

and we therefore offer the following:

Definition 6.6. Let (X1 , . . . , Xn ) be a continuous random variable with joint
probability density function fX1 ,...,Xn . The marginal probability density function
for the random variable X1 is given by
 +∞  +∞
fX1 (x1 ) = ··· fX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) dx2 · · · dxn .
−∞ −∞

Notice that in both the discrete and continuous cases, we sum (or integrate) over
all possible values of the unwanted random variable components.
 y

x
0 a 1

Figure 6.6: The support for the random vector (X, Y ) in the example

The trick in such problems is to insure that your limits of integration are correct.
Drawing a picture of the region where there is positive probability mass (the support
of the distribution) often helps.
For the above example, the picture of the support would be as shown in Figure 6.6.
If the dotted line in Figure 6.6 indicates a particular value for x (call it a), by
integrating over all values of y , we are actually determining how much probability
mass has been placed along the line x = a. The integration process assigns all of
that mass to x = a in one-dimension. Repeating this process for each x yields the
desired probability density function.
Example: Let (X, Y ) be a two-dimensional continuous random variable with joint
probability density function

x+y 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1
fX,Y (x, y) =
0 otherwise

Find the marginal probability density function for X .

Solution: Let’s consider the case where we fix x so that 0 ≤ x ≤ 1. We compute
 +∞
fX (x) = fX,Y (x, y) dy
−∞
  1
= (x + y) dy
0
 y=1
y2 

= xy + 
2 
y=0
1
= x+
2
If x is outside the interval [0, 1], we have fX (x) = 0.
So, summarizing these computations we find that

1
x+ 0≤x≤1
fX (x) = 2
0 otherwise

We will leave it to you to check that fX (x) is, in fact, a probability density
! +∞
functionby making sure that fX (x) ≥ 0 for all x and that −∞ fX (x) dx = 1.
Example: Let (X, Y ) be the two-dimensional random variable with the following
joint probability density function (see Figure 6.5 on page 162)

2−y if 0 ≤ x ≤ 2 and 1 ≤ y ≤ 2
fX,Y (x, y) =
0 otherwise

Find the marginal probability density function for X and the marginal probability
density function for Y .
Solution: Let’s first find the marginal probability density function for X . Consider
the case where we fix x so that 0 ≤ x ≤ 2. We compute
 +∞
fX (x) = fX,Y (x, y) dy
−∞
 2
= (2 − y) dy
1
 y=2
y2 

= 2y − 
2 
y=1
1
=
2
If x is outside the interval [0, 2], we have fX (x) = 0. Therefore,

1
0≤x≤2
fX (x) = 2
0 otherwise
 To find the marginal probability density function for Y , consider the case where
we fix y so that 1 ≤ y ≤ 2. We compute
 +∞
fY (y) = fX,Y (x, y) dx
−∞
 2
= (2 − y) dx
0
= (2 − y)x|x=2
x=0
= 4 − 2y
If y is outside the interval [1, 2], we have fX,Y (x, y) = 0 giving us fY (y) = 0.
Therefore, 
4 − 2y 1≤y≤2
fY (x) =
0 otherwise
You should also double check that fX and fY are both probability density functions.

Functions of Random Vectors

The technique for computing functions of one-dimensional random variables carries
over to the multi-dimensional case. Most of these problems just require a little
careful reasoning.
Example: Let (X1 , X2 ) be the die-tossing random vector of the example on
page 156. Find the probability mass function for the random variable Z = X1 +X2 ,
the sum of the two rolls.
1
Solution: We already know that pX1 ,X2 (x1 , x2 ) = 36 for x1 = 1, . . . , 6 and x2 =
1, . . . , 6. We ask the question, “What are the possible values that Z can take on?”
The answer: “The integers 2, 3, 4, . . . , 12.” For example, Z equals 4 precisely
when any one of the mutually exclusive events

{X1 = 1, X2 = 3},
{X1 = 2, X2 = 2},
or
{X1 = 3, X2 = 1}
occurs. So,
3
pZ (4) = pX1 ,X2 (1, 3) + pX1 ,X2 (2, 2) + pX1 ,X2 (3, 1) = 36 .
Continuing in this manner, you should be able to verify that
1 2 3
pZ (2) = 36 ; pZ (3) = 36 ; pZ (4) = 36 ;
4 5 6
pZ (5) = 36 ; pZ (6) = 36 ; pZ (7) = 36 ;
5 4 3
pZ (8) = 36 ; pZ (9) = 36 ; pZ (10) = 36 ;
2 1
pZ (11) = 36 ; pZ (12) = 36 .

Example: Let (X, Y ) be the two-dimensional random variable given in the exam-
ple on page 166. Find the cumulative distribution function for the random variable
Z =X +Y.
Solution: The support for the random variable Z is the interval [0, 2], so
⎧
⎪
⎨ 0 if z < 0
FZ (z) = ? if 0 ≤ z ≤ 2
⎪
⎩ 1 if z > 2

For the case 0 ≤ z ≤ 2, we wish to evaluate

FZ (z) = P (Z ≤ z) = P (X + Y ≤ z).

In other words, we are computing the probability mass assigned to the shaded set
in Figure 6.7 as z varies from 0 to 2.
In establishing limits of integration, we notice that there are two cases to worry
about as shown in Figure 6.8:
Case I (z ≤ 1):
 z  z−y
FZ (z) = (x + y) dx dy
0 0
1 3
= 3z .

Case II (z > 1):

 z−1  1  1  z−y
FZ (z) = (x + y) dx dy + (x + y) dx dy
0 0 z−1 0

= z 2 − 13 z 3 − 13 .
 y

1
z x+
y=
z

x
0 z 1

Figure 6.7: The event {Z ≤ z}.

y y
z
1 1
z

x x
0 z 1 0 1 z

Case I Case II

Figure 6.8: Two cases to worry about for the example

 These two cases can be summarized as
⎧
⎪
⎪ 0 if z < 0
⎪
⎨ 1 z3 if 0 ≤ z ≤ 1
FZ (z) = 3
⎪
⎪
⎪
z 2 − 13 z 3 − 13 if 1 < z ≤ 2
⎩
1 if z > 2
Notice what we thought at first to be one case (0 ≤ z ≤ 2) had to be divided into
two cases (0 ≤ z ≤ 1 and 1 < z ≤ 2).

Independence of Random Variables

Definition 6.7. A sequence of n random variables X1 , X2 , . . . , Xn is independent
if and only if, and
FX1 ,X2 ,...,Xn (b1 , b2 , . . . , bn ) = FX1 (b1 )FX2 (b2 ) · · · FXn (bn )
for all values b1 , b2 , . . . , bn .
Definition 6.8. A sequence of n random variables X1 , X2 , . . . , Xn is a random
sample if and only if
1. X1 , X2 , . . . , Xn are independent, and
2. FXi (x) = F (x) for all x and for all i (i.e., each Xi has the same marginal
distribution, F (x)).
We say that a random sample is a vector of independent and identically dis-
tributed (i.i.d.) random variables.
Recall: An event A is independent of an event B if and only if
P (A ∩ B) = P (A)P (B).

Theorem 6.1. If X and Y are independent random variables then any event A
involving X alone is independent of any event B involving Y alone.

Testing for independence

Case I: Discrete

A discrete random variable X is independent of a discrete random variable Y if

and only if
pX,Y (x, y) = [pX (x)][pY (y)]
for all x and y .
Case II: Continuous

A continuous random variable X is independent of a continuous random variable

Y if and only if
fX,Y (x, y) = [fX (x)][fY (y)]
for all x and y .
Example: A company produces two types of compressors, grade A and grade B.
Let X denote the number of grade A compressors produced on a given day. Let
Y denote the number of grade B compressors produced on the same day. Suppose
that the joint probability mass function pX,Y (x, y) = P (X = x, Y = y) is given
by the following table:

y
pX,Y (x, y) 0 1
0 0.1 0.3
x 1 0.2 0.1
2 0.2 0.1

The random variables X and Y are not independent. Note that

pX,Y (0, 0) = 0.1 = pX (0)pY (0) = (0.4)(0.5) = 0.2

Example: Suppose an electronic circuit contains two transistors. Let X be the

time to failure of transistor 1 and let Y be the time to failure of transistor 2.

4e−2(x+y) x ≥ 0, y ≥ 0
fX,Y (x, y) =
0 otherwise

The marginal densities are


2e−2x x ≥ 0
fX (x) =
0 otherwise


2e−2y y ≥ 0
fY (y) =
0 otherwise
 We must check the probability density functions for (X, Y ), X and Y for all values
of (x, y).
For x ≥ 0 and y ≥ 0:

fX,Y (x, y) = 4e−2(x+y) = fX (x)fY (y) = 2e−2x 2e−2y

For x ≥ 0 and y < 0:

fX,Y (x, y) = 0 = fX (x)fY (y) = 2e−2x (0)

For x < 0 and y ≥ 0:

fX,Y (x, y) = 0 = fX (x)fY (y) = (0)2e−2y

For x < 0 and y < 0:

fX,Y (x, y) = 0 = fX (x)fY (y) = (0)(0)

So the random variables X and Y are independent.

Expectation and random vectors

Suppose we are given a random vector (X, Y ) and a function g(x, y). Can we find
E(g(X, Y ))?
Theorem 6.2.

E(g(X, Y )) = g(x, y)pX,Y (x, y) if (X, Y ) is discrete
all x all y
 ∞  ∞
E(g(X, Y )) = g(x, y)fX,Y (x, y) dy dx if (X, Y ) is continuous
−∞ −∞

Example: Suppose X and Y have joint probability density function


x + y 0 ≤ x ≤ 1, 0 ≤ y ≤ 1
fX,Y (x, y) =
0 otherwise
Let Z = XY . To find E(Z) = E(XY ) use Theorem 6.2 to get
 ∞  ∞
E(XY ) = xyfX,Y (x, y) dx dy
−∞ −∞
 1 1
= xy(x + y) dx dy
0 0
 1 1
= x2 y + xy 2 dx dy
0 0
 1 1
1 3 
= 3x y + 12 x2 y 2  dy
0 0
 1
1
= 3y + 12 y 2 dy
0
1
1 2 
= 6y + 16 y 3  = 1
3
0

We will prove the following results for the case when (X, Y ) is a continuous random
vector. The proofs for the discrete case are similar using summations rather than
integrals, probability mass functions rather than probability density functions.
Theorem 6.3. E(X + Y ) = E(X) + E(Y )
Proof. Using Theorem 6.2:
 ∞  ∞
E(X + Y ) = (x + y)fX,Y (x, y) dx dy
−∞ −∞
 ∞  ∞  ∞  ∞
= xfX,Y (x, y) dx dy + yfX,Y (x, y) dx dy
−∞ −∞ −∞ −∞
 ∞  ∞  ∞  ∞
= xfX,Y (x, y) dy dx + yfX,Y (x, y) dx dy
−∞ −∞ −∞ −∞
 ∞  ∞  ∞  ∞
= x fX,Y (x, y) dy dx + y fX,Y (x, y) dx dy
−∞ −∞ −∞ −∞
 ∞  ∞
   
= x fX (x) dx + y fY (y) dy
−∞ −∞
= E(X) + E(Y )

Theorem 6.4. If X and Y are independent, then

E[h(X)g(Y )] = E[h(X)]E[g(Y )]
Proof. Using Theorem 6.2:
 ∞  ∞
E[h(X)g(Y )] = h(x)g(y)fX,Y (x, y) dx dy
−∞ −∞

since X and Y are independent. . .

 ∞  ∞
= h(x)g(y)fX (x)fY (y) dx dy
−∞ −∞
 ∞  ∞
= g(y)fY (y) h(x)fX (x) dx dy
−∞ −∞
 ∞
 
= g(y)fY (y) E(h(X)) dy
−∞

since E(h(X)) is a constant. . .

 ∞
= E(h(X)) g(y)fY (y) dy
−∞
= E(h(X))E(g(Y ))

Corollary 6.5. If X and Y are independent, then E(XY ) = E(X)E(Y )

Proof. Using Theorem 6.4, set h(x) = x and g(y) = y to get

E[h(X)g(Y )] = E(h(X))E(g(Y ))
E[XY ] = E(X)E(Y )

Definition 6.9. The covariance of the random variables X and Y is

Cov(X, Y ) ≡ E[(X − E(X))(Y − E(Y ))].

Note that Cov(X, X) = Var(X).

Theorem 6.6. For any random variables X and Y

Var(X + Y ) = Var(X) + Var(Y ) + 2Cov(X, Y )

Proof. Remember that the variance of a random variable W is defined as

Var(W ) = E[(W − E(W ))2 ]

Now let W = X + Y . . .

Var(X + Y ) = E[(X + Y − E(X + Y ))2 ]

= E[(X + Y − E(X) − E(Y ))2 ]
= E[({X − E(X)} + {Y − E(Y )})2 ]

Now let a = {X − E(X)}, let b = {Y − E(Y )}, and

expand (a + b)2 to get. . .

= E[{X − E(X)}2 + {Y − E(Y )}2 + 2{X − E(X)}{Y − E(Y )}]

= E[{X − E(X)}2 ] + E[{Y − E(Y )}2 ]
+2E[{X − E(X)}{Y − E(Y )}]
= Var(X) + Var(Y ) + 2Cov(X, Y )

Theorem 6.7. Cov(X, Y ) = E(XY ) − E(X)E(Y )

Proof. Using Definition 6.9, we get

Cov(X, Y ) = E[(X − E(X))(Y − E(Y ))]

= E[XY − XE(Y ) − E(X)Y + E(X)E(Y )]
= E[XY ] + E[−XE(Y )] + E[−E(X)Y ] + E[E(X)E(Y )]]

Since E[X], E[Y ], and E[XY ] are all constants. . .

= E[XY ] − E(Y )E[X] − E(X)E[Y ] + E(X)E(Y )

= E[XY ] − E(X)E(Y )

Corollary 6.8. If X and Y are independent then Cov(X, Y ) = 0.

Proof. If X and Y are independent then, from Corollary 6.5, E(XY ) =
E(X)E(Y ). We can then use Theorem 6.7:

Cov(X, Y ) = E(XY ) − E(X)E(Y ) = 0

Corollary 6.9. If X and Y are independent then Var(X+Y ) = Var(X)+Var(Y ).
Proof. If X and Y are independent then, from Theorem 6.6 and Corollary 6.8:

Var(X + Y ) = Var(X) + Var(Y ) + 2Cov(X, Y )

Var(X + Y ) = Var(X) + Var(Y ) + 2(0)
Var(X + Y ) = Var(X) + Var(Y )

Definition 6.10. The covariance of two random variables X and Y is given by

 
Cov(X, Y ) ≡ E (X − E(X))(Y − E(Y )) .

Definition 6.11. The correlation coefficient for two random variables X and Y
is given by
Cov(X, Y )
ρ(X, Y ) ≡ " .
Var(X)Var(Y )

Theorems 6.10, 6.11 and 6.12 are stated without proof. Proofs of these results may
be found in the book by Meyer.2
Theorem 6.10. For any random variables X and Y ,

|ρ(X, Y )| ≤ 1.

Theorem 6.11. Suppose that |ρ(X, Y )| = 1. Then (with probability one),

Y = aX + b for some constants a and b. In other words: If the correlation
coefficient ρ is ±1, the Y is a linear function of X (with probability one).
The converse of this theorem is also true:
Theorem 6.12. Suppose that X and Y are two random variables, such that
Y = aX + b where a and b are constants. Then |ρ(X, Y )| = 1. If a > 0, then
ρ(X, Y ) = +1. If a < 0, then ρ(X, Y ) = −1.
2
Meyer, P., Introductory probability theory and statistical applications, Addison-Wesley, Reading
MA, 1965.
Random vectors and conditional probability

Example: Consider the compressor problem again.

y
pX,Y (x, y) 0 1
0 0.1 0.3
x 1 0.2 0.1
2 0.2 0.1

Given that no grade B compressors were produced on a given day, what is the
probability that 2 grade A compressors were produced?
Solution:
P (X = 2, Y = 0)
P (X = 2 | Y = 0) =
P (Y = 0)
0.2 2
= =
0.5 5

Given that 2 compressors were produced on a given day, what is the probability
that one of them is a grade B compressor?
Solution:
P (Y = 1, X + Y = 2)
P (Y = 1 | X + Y = 2) =
P (X + Y = 2)
P (X = 1, Y = 1)
=
P (X + Y = 2)
0.1 1
= =
0.3 3

Example: And, again, consider the two transistors. . .


4e−2(x+y) x ≥ 0, y ≥ 0
fX,Y (x, y) =
0 otherwise

Given that the total life time for the two transistors is less than two hours, what is
the probability that the first transistor lasted more than one hour?
 Solution:
P (X > 1, X + Y ≤ 2)
P (X > 1 | X + Y ≤ 2) =
P (X + Y ≤ 2)
We then compute
 2  2−x
P (X > 1, X + Y ≤ 2) = 4e−2(x+y) dy dx
1 0
−2
= e − 3e−4

and
 2  2−x
P (X + Y ≤ 2) = 4e−2(x+y) dy dx
0 0
= 1 − 5e−4

to get
e−2 − 3e−4
P (X > 1 | X + Y ≤ 2) = = 0.0885
1 − 5e−4

Conditional distributions

Case I: Discrete

Let X and Y be random variables with joint probability mass function pX,Y (x, y)
and let pY (y) be the marginal probability mass function for Y .
We define the conditional probability mass function of X given Y as
pX,Y (x, y)
pX|Y (x | y) =
pY (y)
whenever pY (y) > 0.

Case II: Continuous

Let X and Y be random variables with joint probability density function fX,Y (x, y)
and let fY (y) be the marginal probability density function for Y .
We define the conditional probability density function of X given Y as
fX,Y (x, y)
fX|Y (x | y) =
fY (y)
whenever fY (y) > 0.
Law of total probability

Case I: Discrete

pX (x) = pX|Y (x | y)pY (y)
y

Case II: Continuous

 +∞
fX (x) = fX|Y (x | y)fY (y) dy
−∞

Self-Test Exercises for Chapter 6

For each of the following multiple-choice questions, choose the best response
among those provided. Answers can be found in Appendix B.
S6.1 Let X1 , X2 , X3 , X4 be independent and identically distributed random vari-
ables, each with P (Xi = 1) = 12 and P (Xi = 0) = 12 . Let P (X1 + X2 +
X3 + X4 = 3) = r, then the value of r is
1
(A) 16
1
(B) 4
1
(C) 2

(D) 1
(E) none of the above.
S6.2 Let (X, Y ) be a discrete random vector with joint probability mass function
given by

pX,Y (0, 0) = 1/4

pX,Y (0, 1) = 1/4
pX,Y (1, 1) = 1/2

Then P (Y = 1) equals
(A) 1/3
(B) 1/4
(C) 1/2
 (D) 3/4
(E) none of the above.
S6.3 Let (X, Y ) be a random vector with joint probability mass function given by
1 1 1
pX,Y (1, −1) = 4 pX,Y (1, 1) = 2 pX,Y (0, 0) = 4

Let pX (x) denote the marginal probability mass function for X . The value
of pX (1) is
(A) 0
1
(B) 4
1
(C) 2

(D) 1
(E) none of the above.
S6.4 Let (X, Y ) be a random vector with joint probability mass function given by
1 1 1
pX,Y (1, −1) = 4 pX,Y (1, 1) = 2 pX,Y (0, 0) = 4

The value of P (XY > 0) is

(A) 0
1
(B) 4
1
(C) 2

(D) 1
(E) none of the above.
S6.5 Let (X, Y ) be a continuous random vector with joint probability density
function

1/2 if −1 ≤ x ≤ 1 and 0 ≤ y ≤ 1
fX,Y (x, y) =
0 otherwise

Then P (X > 0) equals

(A) 0
(B) 1/4
 (C) 1/2
(D) 1
(E) none of the above.
S6.6 Suppose (X, Y ) is a continuous random vector with joint probability density
function 
e−y if 0 ≤ x ≤ 1 and y ≥ 0
fX,Y (x, y) =
0 otherwise
Then P (X > 12 ) equals
(A) 0
(B) 1/2
(C) e−1/2
1 −y
(D) 2e

(E) none of the above.

S6.7 Let (X, Y ) be a discrete random vector with joint probability mass function
given by
pX,Y (0, 0) = 1/3
pX,Y (0, 1) = 1/3
pX,Y (1, 0) = 1/3
Then P (X + Y = 1) equals
(A) 0
(B) 1/3
(C) 2/3
(D) 1
(E) none of the above.
S6.8 Let (X, Y ) be a discrete random vector with joint probability mass function
given by
pX,Y (0, 0) = 1/3
pX,Y (0, 1) = 1/3
pX,Y (1, 0) = 1/3
 Let W = max{X, Y }. Then P (W = 1) equals
(A) 0
(B) 1/3
(C) 2/3
(D) 1
(E) none of the above.
S6.9 Suppose (X, Y ) is a continuous random vector with joint probability density
function ⎧
⎪
⎨ y if 0 ≤ x ≤ 2 and
fX,Y (x, y) = 0≤y≤1
⎪
⎩ 0 otherwise
Then E(X 2 ) equals
(A) 0
(B) 1
(C) 4/3
(D) 8/3
(E) none of the above.
S6.10 Suppose (X, Y ) is a continuous random vector with joint probability density
function ⎧
⎪
⎨ y if 0 ≤ x ≤ 2 and
fX,Y (x, y) = 0≤y≤1
⎪
⎩ 0 otherwise
Then P (X < 1, Y < 0.5) equals
(A) 0
(B) 1/8
(C) 1/4
(D) 1/2
(E) none of the above.
S6.11 The weights of individual oranges are independent random variables, each
having an expected value of 6 ounces and standard deviation of 2 ounces.
Let Y denote the total net weight (in ounces) of a basket of n oranges. The
variance of Y is equal to
√
(A) 4 n
(B) 4n
(C) 4n2
(D) 4
(E) none of the above.
S6.12 Let X1 , X2 , X3 be independent and identically distributed random variables,
each with P (Xi = 1) = p and P (Xi = 0) = 1 − p. If P (X1 + X2 + X3 =
3) = r, then the value of p is
1
(A) 3
1
(B) 2
√
(C) 3
r
(D) r3
(E) none of the above.
S6.13 If X and Y are random variables with Var(X) = Var(Y ), then Var(X + Y )
must equal
(A) 2Var(X)
√
(B) 2(Var(X))
(C) Var(2X)
(D) 4Var(X)
(E) none of the above.
S6.14 Let (X, Y ) be a continuous random vector with joint probability density
function 
1/π if x2 + y 2 ≤ 1
fX,Y (x, y) =
0 otherwise
Then P (X > 0) equals
 (A) 1/(4π)
(B) 1/(2π)
(C) 1/2
(D) 1
(E) none of the above.
S6.15 Let X be a random variable. Then E[(X + 1)2 ] − (E[X + 1])2 equals
(A) Var(X)
(B) 2E[(X + 1)2 ]
(C) 0
(D) 1
(E) none of the above.
S6.16 Let X , Y and Z be independent random variables with

E(X) = 1 E(Y ) = 0 E(Z) = 1

Let
W = X(Y + Z).
Then E(W ) equals
(A) 0
(B) E(X 2 )
(C) 1
(D) 2
(E) none of the above
S6.17 Toss a fair die twice. Let the random variable X represent the outcome of
the first toss and the random variable Y represent the outcome of the second
toss. What is the probability that X is odd and Y is even.
(A) 1/4
(B) 1/3
(C) 1/2
 (D) 1
(E) none of the above
S6.18 Assume that (X, Y ) is random vector with joint probability density function

k for 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1 and y ≤ x
fX,Y (x, y) =
0 otherwise

where k is a some positive constant. Let W = X −Y . To find the cumulative

distribution function for W we can compute
 B D
FW (w) = k dx dy
A C

for 0 ≤ w ≤ 1. The limit of integration that should appear in position D is

(A) min{1, x − w}
(B) min{1, y − w}
(C) min{1, x − y}
(D) min{1, y + w}
(E) none of the above.
S6.19 Assume that (X, Y ) is a random vector with joint probability mass function
given by

pX,Y (−1, 0) = 1/4

pX,Y (0, 0) = 1/2
pX,Y (0, 1) = 1/4

Define the random variable W = XY . The value of P (W = 0) is

(A) 0
(B) 1/4
(C) 1/2
(D) 3/4
(E) none of the above.
S6.20 Let X be a random variable with E(X) = 0 and Var(X) = 1. Let Y be a
random variable with E(Y ) = 0 and Var(Y ) = 4. Then E(X 2 + Y 2 ) equals
(A) 0
(B) 1
(C) 3
(D) 5
(E) none of the above.
S6.21 Suppose (X, Y ) is a continuous random vector with joint probability density
function

4xy if 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1
fX,Y (x, y) =
0 otherwise

1
Then E equals
XY
(A) +∞
(B) 0
(C) 1/2
(D) 4
(E) none of the above.
S6.22 The life lengths of two transistors in an electronic circuit is a random vector
(X, Y ) where X is the life length of transistor 1 and Y is the life length of
transistor 2. The joint probability density function of (X, Y ) is given by

2e−(x+2y) if x ≥ 0 and y ≥ 0
fX,Y (x, y) =
0 otherwise

Then P (X + Y ≤ 1) equals
!1!1
(A) 0 0 2e−(x+2y) dx dy
! 1 ! 1−y
(B) 0 0 2e−(x+2y) dx dy
!1!1
(C) 0 y 2e−(x+2y) dx dy
7
TRANSFORMS AND SUMS

Transforms
Notation

Throughout this section we will use the following notation:

Definition 7.1. For any differentiable function h(u), and constant c

dk 
(k) 
h (c) ≡ h(u) .
duk 
u=c

In other words, h(k) (c) is the k th derivative of h evaluated at u = c.

Example: Let h(x) = x3 then

d2 
(2) 
h (4) = h(x)
dx2 
x=4
d2 3 
= x 
dx2 x=4


= 3x2 
x=4
= 3(4)2 = 48

Probability generating functions

Definition 7.2. Let X be a discrete random variable with support on the nonneg-
ative integers. Then the probability generating function for X is given by

gX (z) ≡ E(z X ) for |z| < 1.

199
Example: Let X have a Poisson distribution with parameter α > 0 Then

e−α αk
pX (k) = for k = 0, 1, 2, . . .
k!
∞
Recalling that ex = k
k=0 (x /k !), we get
∞

X e−α αk
gX (z) = E(z ) = zk
k=0
k!
∞
 (αz)k
= e−α
k=0
k!
−α αz
= e e
α(z−1)
= e

Theorem 7.1. If gX (z) is the probability generating function of X , then

(k)
gX (0)
P (X = k) =
k!

Proof. (Informal proof) Let g (k) (z) denote the k th derivative of g with respect to
z . We have

gX (z) = E(z X )
∞

= z k P (X = k)
k=0

Writing out the sum explicitly, we get. . .

gX (z) = z 0 P (X = 0) + z 1 P (X = 1) + z 2 P (X = 2) + z 3 P (X = 3) + . . .
gX (0) = (1)P (X = 0)

Now take the first derivative of gX (z). . .

(1)
gX (z) = P (X = 1) + 2zP (X = 2) + 3z 2 P (X = 3) + 4x3 P (X = 4) + . . .
(1)
gX (0) = P (X = 1)

Another derivative produces. . .

 (2)
gX (z) = 2P (X = 2) + 3(2)zP (X = 3) + 4(3)z 2 P (X = 4) + 5(4)z 3 P (X = 5) + . . .
(2)
gX (0) = 2P (X = 2)

And once more to see the pattern. . .

(3)
gX (z) = 3(2)P (X = 3) + 4(3)(2)zP (X = 4) + 5(4)(3)z 2 P (X = 5) + . . .
(3)
gX (0) = 3(2)P (X = 3)

And, in general

(n)
gX (0) = n!P (X = n)

Theorem 7.2. If gX (z) is the probability generating function of X , then

(1)
E(X) = gX (1)
(1)
where gX (1) is the first derivative of gX (z) with respect to z , evaluated at z = 1.
Proof. As above,
(1)
gX (z) = P (X = 1) + 2zP (X = 2) + 3z 2 P (X = 3) + 4x3 P (X = 4) + . . .
(1)
gX (1) = P (X = 1) + 2P (X = 2) + 3P (X = 3) + 4P (X = 4) + . . .
∞

= kP (X = k) = E(X)
k=0

Theorem 7.3. If X1 , X2 , . . . , Xn are independent random variables with proba-

bility generating functions gX1 (z), gX2 (z), . . . , gXn (z), respectively, then the prob-
ability generating function of the random variable Y = X1 + X2 + · · · + Xn is
n
#
gY (z) = gXi (z).
i=1

Proof. Recall that if X and Y are independent random variables, then

E(h(X)g(Y )) = E(h(X))E(g(Y ))
Therefore,

gY (z) = E(z Y ) = E(z X1 +X2 +...+Xn )

= E(z X1 z X2 · · · z Xn )
= E(z X1 )E(z X2 ) · · · E(z Xn )
= gX1 (z)gX2 (z) · · · gXn (z)

Theorem 7.4. If gX (z) is the probability generating function of X , and a is

a nonnegative, integer constant, then the probability generating function of the
random variable Y = a + X is

gY (z) = z a gX (z).

Proof. We have,

gY (z) = E(z Y ) = E(z a+X )

= E(z a z X )
= z a E(z X )
= z a gX (z)

We offer Theorem 7.5 without proof. Except for some technical details, it is a
direct result of Theorem 7.1.
Theorem 7.5. The probability generating function of a random variable uniquely
determines its probability distribution.
Example: Let T have a Bernoulli distribution with parameter p.

P (T = 1) = p
P (T = 0) = 1 − p = q

Then the probability generating function of T is given by

gT (z) = z 0 q + z 1 p = q + zp
 Example: Now suppose, T1 , T2 , . . . , Tn are n independent Bernoulli random vari-
ables each with

P (Ti = 1) = p
P (Ti = 0) = 1 − p = q

We know (from Chapter 4) that X = T1 + T2 + · · · + Tn has a Binomial distribution

with parameters n and p. That is
 
n k (n−k)
P (X = k) = p q for k = 0, 1, . . . , n.
k

One could obtain the probability generating function for X is given by computing
n
 
 n k (n−k)
X k
gX (z) = E(z ) = z p q
k=0
k

But we can use Theorem 7.3 with the Ti ’s to easily get

gX (z) = gT1 (z)gT2 (z) · · · gTn (z) = (q + zp)n

As an exercise, use Theorem 7.2 to get E(X) from gX (z).

Moment generating functions

Definition 7.3. The kth moment about the origin of a random variable X is
defined as E(X k ).
Definition 7.4. Let X be (almost) any random variable. Then the moment
generating function for X is given by

MX (t) ≡ E(etX ).

Example: Let X have a uniform distribution on the interval (a, b). Then
⎧
⎨ 1
if a ≤ x ≤ b
fX (x) = b − a
⎩
0 otherwise
Therefore,
 ∞
tX
MX (t) = E(e ) = etx fX (x) dx
−∞
 b
1
= etx dx
a b−a
 b
1
= tetx dx
t(b − a) a

1 b
= etx 
t(b − a) a
etb − eta
= for t = 0
t(b − a)

Example: Let X have a Poisson distribution with parameter α > 0 Then

e−α αk
pX (k) = for k = 0, 1, 2, . . .
k!
∞
Recalling that ex = k
k=0 (x /k !), we get
∞
 e−α αk
MX (t) = E(etX ) = etk
k=0
k!
∞
 (αet )k
= e−α
k=0
k!
−α αet
= e e
α(et −1)
= e
Note that since a Poisson random variable X has support {0, 1, 2, . . .}, it can
have a probability generating function and a moment generating function. When a
random variable has both a probability generating function and a moment generating
function, we can derive the following relationship:
gX (z) = E[z X ]
substituting z = et , we obtain
gX (et ) = E[(et )X ]
= E[etX ] = MX (t)
Since the probability generating function for a Poisson random variable is
gX (z) = eα(z−1)
the moment generating function for a Poisson random variable must be
t −1)
MX (t) = gX (et ) = eα(e

It’s important to remember that this relationship will only hold for random vari-
ables with support {0, 1, 2, . . .} (that is, those random variables with probability
generating functions).
Example: Let X have an exponential distribution with parameter α > 0. Then

αe−αx if x ≥ 0
fX (x) =
0 otherwise

Therefore,
 ∞
tX
MX (t) = E(e ) = etx αe−αx dx
0
 ∞
= αex(t−α) dx
0

Note that this integral only converges if t < α. Continuing,


α x(t−α) ∞
MX (t) = e 
t−α 0
α
= for t < α
α−t

Example: (From Meyer.1 ) Let X ∼ N (μ, σ 2 ). Then

  
1 1 x−μ 2
fX (x) = √ exp −
2πσ 2 σ

Therefore, for every t,

 +∞   
1 1 x−μ 2
MX (t) = √ etx exp − dx
2πσ −∞ 2 σ

Let (x − μ)/σ = s; thus x = σs + μ and dx = σds. Therefore,

 +∞
1
exp[t(σs + μ)]e−s
2 /2
MX (t) = √ ds
2π −∞
1
Meyer, P., Introductory probability theory and statistical applications, Addison-Wesley, Reading
MA, 1965.
 +∞ 
1
tμ
= e √ exp(− 12 [s2 − 2σts]) ds
2π −∞
 +∞
1
= etμ √ exp{− 12 [(s − σt)2 − σ 2 t2 ]} ds
2π −∞
 +∞
tμ+σ 2 t2 /2 1
= e √ exp(− 12 [s − σt]2 ) ds
2π −∞
Now let s − σt = v ; then ds = dv and we obtain
 +∞
1
MX (t) = etμ+σ
2 t2 /2
e−v
2 /2
√ dv
2π −∞

= e(tμ+σ
2 t2 /2)
.

Theorem 7.6. If MX (t) is the moment generating function of X , then

(k)
E(X k ) = MX (0) for k = 0, 1, 2, . . .

(k)
Proof. (Informal proof) Let MX (t) denote the k th derivative of MX with respect
to t. We have

MX (t) = E(etX )
∞ 
 (tX)k
= E
k=0
k!

Writing out the sum explicitly, we get. . .

 
(tX)2 (tX)3 (tX)4
MX (t) = E 1 + tX + + + + ···
2! 3! 4!

If the sum in the right hand side converges, we get . . .

t2 E(X 2 ) t3 E(X 3 ) t4 E(X 4 )

MX (t) = 1 + tE(X) + + + + ···
2! 3! 4!
Take the first derivative of both sides to get . . .

(1) t2 E(X 3 ) t3 E(X 4 )

MX (t) = E(X) + tE(X 2 ) + + + ···
2! 3!
 and when evaluated at t = 0. . .
(1)
MX (0) = E(X)

Take the second derivative of both sides to get . . .

(2) t2 E(X 4 ) t3 E(X 5 )

MX (t) = E(X 2 ) + tE(X 3 ) + + + ···
2! 3!
and when evaluated at t = 0. . .
(2)
MX (0) = E(X 2 )

Continuing, we see that. . .

(k) t2 E(X k+2 ) t3 E(X k+3 )

MX (t) = E(X k ) + tE(X k+1 ) + + + ···
2! 3!
and when evaluated at t = 0. . .
(k)
MX (0) = E(X k )

Theorem 7.7. If X1 , X2 , . . . , Xn are independent random variables with moment

generating functions MX1 (t), MX2 (t), . . . , MXn (t), respectively, then the moment
generating function of the random variable Y = X1 + X2 + · · · + Xn is
n
#
MY (t) = MXi (t).
i=1

Theorem 7.8. If MX (t) is the moment generating function of X , and a and b are
constants, then the moment generating function of the random variable Y = aX +b
is
MY (t) = ebt MX (at).

Theorem 7.9. The moment generating function of a random variable uniquely

determines its probability distribution.
Example: If X ∼ N (μ, σ 2 ) then the moment generating function for X is
1 2 2
MX (t) = etμ+ 2 t σ
So, if someone tells you that the random variable W has moment generating
function
MW (t) = e−2t+4t
2

then the distribution of W must be N (−2, 8).

And, if you encounter a random Y with moment generating function

MY (t) = et
2 /2

you can immediately say that Y ∼ N (0, 1).

Now, suppose you are given that the random variable X has a normal distribution
with mean μ and variance σ 2 . So the moment generating function for X is
1 2 2
MX (t) = etμ+ 2 t σ

Consider the random variable

X −μ 1 μ
Z= = X−
σ σ σ
Using Theorem 7.8 with
1 μ
a= and b=−
σ σ
we find that the moment generating function for Z is

MZ (t) = ebt MX (at)

= e−tμ/σ MX (t/σ)
t2
tμ [ σt μ+ σ2 ]
= e[− σ ] e 2σ 2
t2
[− tμ + tμ + σ2 ]
= e σ σ 2σ 2

= et
2 /2

which is the moment generating function of a normal random variable with expected
value equal to zero and variance equal to one. Therefore, we have shown that if
X ∼ N (μ, σ 2 ) then Z = (X − μ)/σ ∼ N (0, 1).
Other transforms of random variables

Laplace transform

Definition 7.5. Let X be (almost) any random variable. Then the Laplace
transform for X is given by

LX (t) ≡ E(e−tX ).

Note that the moment generating function for a continuous random variable X
with probability density function fX (x) is
 ∞
MX (t) = etx fX (x) dx
−∞

and the Laplace transform for the same density function is

 ∞
LX (t) = e−tx fX (x) dx
−∞

So the moment generating function of a continuous random variable is simply

the Laplace transform of the probability density function with t replaced by −t
everywhere in the formula for the transform.

Characteristic function

Definition 7.6. Let X be any random variable. Then the characteristic function
for X is given by
ψX (t) ≡ E(eitX )
√
where i ≡ −1.

Linear combinations of normal random variables

Theorem 7.10. if X1 , X2 , . . . , Xn are independent normal random variables with
Xi ∼ N (μi , σi2 ) and a1 , a2 , . . . , an and b are constants, then
n

Y =b+ ai Xi ∼ N (μY , σY2 )
i=1

where
n

μY = b + ai μi
i=1
and
n

σY2 = a2i σi2 .
i=1

Example: The weights of snails are normally distributed with mean 6 gm. and
variance 4 gm2 . Therefore, it is reasonable to assume that the weights of n snails
chosen at random from the population of all snails are n independent identically
distributed random variables, X1 , X2 , . . . , Xn , with each Xi ∼ N (6, 4).

3 Suppose that 4 snails are selected at random. and we would like to find the
probability that the total weight, T = X1 + X2 + X3 + X4 , will exceed 28
gm.?
Since X1 , X2 , . . . , Xn , are iid with each Xi ∼ N (6, 4), we have T =
4
i=1 Xi ∼ N (24, 16). Therefore,
 
T − 24 28 − 24
P (T > 28) = P >
4 4
= P (Z > 1) where Z ∼ N (0, 1)
= 1 − P (Z ≤ 1) = 1 − 0.8413 = 0.1587

3 Once again, suppose that 4 snails are selected at random. Let X = 14 (X1 +
X2 + X3 + X4 ) denote the average weight of the 4 snails. Let’s find the
probability that the observed average weight deviates from the expected
weight by more than 2 gm. That is, we want to find P (|X − 6| > 2).
Since X1 , X2 , . . . , Xn , are iid with each Xi ∼ N (6, 4), we have X =
1 4
4 i=1 Xi ∼ N (6, 1). Therefore,

P (|X − 6| > 2) = 1 − P (|X − 6| ≤ 2)

= 1 − P (−2 ≤ X − 6 ≤ 2)
 
−2 X −6 2
= 1−P ≤ ≤
1 1 1
= 1 − P (−2 ≤ Z ≤ 2) where Z ∼ N (0, 1)
= 1 − [P (Z ≤ 2) − P (Z ≤ −2)]
= 1 − [P (Z ≤ 2) − (1 − P (Z ≤ 2))]
= 2 − 2P (Z ≤ 2) = 2 − 2(0.9772) = 0.0456
The distribution of the sample mean

Let X1 , X2 , . . . , Xn be a random sample from a N (μ, σ 2 ) distribution. Let X =

1 n
n i=1 Xi . Remember that X , the sample mean, is a random variable. Then

X ∼ N (μ, σ 2 /n).

The Central Limit Theorem

Theorem 7.11. (Central Limit Theorem) If
1. X1 , X2 , . . . , Xn is a random sample with E(Xi ) = μ and Var(Xi ) = σ 2 for
all i, and
2. Sn = X1 + X2 + · · · + Xn , and
Sn − nμ
3. Zn = √ ,
σ n
then
lim P (Zn ≤ z) = Φ(z).
n→∞

Proof. First note that E(Sn ) = nμ and Var(Sn ) = nσ 2 . We will get the result by
showing that the moment generating function for Zn must approach the moment
generating function of a N (0, 1) random variable. Since each Xi has the same
moment generating function for all i, we can let M (t) denote the common moment
generating function of each X1 , X2 , . . . , Xn . Then

MZn (t) = M Sn√

−nμ (t) = M S√
n −
√
nμ (t)
σ n σ n σ

Since MaY +b (t) = etb MY (at) for any random variable Y and any constants a and
b, we get √ √
(1) MZn (t) = e−μ nt/σ MSn (t/(σ n))
Since each Xi has the same moment generating function for all i, we can let
M (t) denote the common moment generating function of each X1 , X2 , . . . , Xn .
Therefore, MSn (t) = [M (t)]n , and
√  √ n
MZn (t) = e−μ nt/σ
M (t/(σ n))

We are going to need the following two facts:

 ∞
 E(X k ) k
(A) M (t) = 1 + t
k=1
k!
∞
 (−1)j−1 j
(B) ln(1 + ξ) = ξ for |ξ| < 1.
j=1
j

Now, take the natural log of both sides of equation (1) and use fact (A):
√  
−μ nt t
ln MZn (t) = + n ln M √
σ σ n
√  ∞  k 
−μ nt  E(X k ) t
= + n ln 1 + √ .
σ k=1
k! σ n

Note that the natural log term looks like fact (B) with
∞
  k
E(X k ) t
ξ= √ .
k=1
k! σ n

As n → ∞, we can make this summation as small as we want. Hence, for suitably

large n, the summation (i.e., ξ ) will eventually be less than one, permitting us to
use fact (B), so
ln MZn (t)
⎡ ⎧ ∞  ⎫⎤
√ ∞ ⎨ k )  t k j ⎬
−μ nt  (−1 )j−1  E(X
= + n⎣ √ ⎦
σ ⎩ j k ! σ n ⎭
j=1 k=1
⎡  ∞  ⎤
√ k )  t k
−μ nt ∞
E(X 1  E(X k )  t k 2
= + n⎣ √ − √ + · · ·⎦ .
σ k=1
k ! σ n 2 k=1
k ! σ n

Collect similar powers of t to obtain

√     
−μ nt E(X) E(X) 1 E(X)2 2
ln MZn (t) = +n √ t+ − t + ··· .
σ σ n 2!σ 2 n 2 σ 2 n
Hence,
 √   
(−μ + E(X)) n E(X 2 ) − E(X)2 2
ln MZn (t) = t+ t + [other terms]
σ 2σ 2

But E(X) = μ and E(X 2 ) − E(X)2 = σ 2 . Therefore,

1
ln MZn (t) = 0t + t2 + [other terms].
2
 But the “[other terms]” for tn with n ≥ 3 go to zero as n → ∞ because there is
a power of n in the denominator of each of the “[other terms]” (expand the series
further if you need convincing). Hence,

 t2
lim ln MZn (t) = .
n→∞ 2
Therefore,
lim MZn (t) = et
2 /2

n→∞

which is the moment generating function for N (0, 1). So limn→∞ P (Zn ≤ a) =
Φ(a).

The DeMoivre-Laplace Theorem

A proof of the Central Limit Theorem was provided by Liapounoff in 1902. But
DeMoivre (1733) and Laplace (1812) proved a similar result for the following
special case:
Let T1 , T2 , . . . , Tn be a random sample from a Bernoulli distribution, that is let

P (Ti = 1) = p
P (Ti = 0) = 1 − p.
n
Then E(Ti ) = p and Var(Ti ) = p(1 − p). We already know that Sn = i=1 Ti
has a Binomial distribution with parameters n and p, that is
 
n k
P (Sn = k) = p (1 − p)n−k for k = 0, 1, . . . , n.
k

However, because of the Central Limit Theorem,2 we get the following

Theorem 7.12. (DeMoivre-Laplace Theorem) As n gets large (i.e., as the
number of Bernoulli trials increases) the distribution of
Sn − np
Zn = "
np(1 − p)

approaches the distribution of a N (0, 1) random variable.

2
Of course, DeMoivre and Laplace didn’t know about the Central Limit Theorem and had to prove
this in a different way.
 Another (more casual) way to state the DeMoivre-Laplace Theorem is to say that
as n → ∞, the distribution of the random variable Sn approaches the distribution
of a N (np, np(1 − p)) random variable. In the limit, this is not very useful (and
not precise), because as n → ∞ both the mean np and variance np(1 − p) go to
∞. But for finite n, we get what is often called the “normal approximation to the
binomial,” as follows:
Corollary 7.13. Suppose n is an integer and 0 < p < 1, then for “large” values
of n,  
n k (k−np)2
−1
p (1 − p)n−k ≈ √ 1 e 2 np(1−p)
k 2πnp(1−p)

This approximation usually works well when np(1 − p) > 9.

The distribution of the sample mean for large n

Let X1 , X2 , . . . , Xn be a random sample from (just about any) distribution with

E(Xi ) = μ and Var(Xi ) = σ 2 . And recall that the sample mean is defined as
n
1
X= Xi
n i=1

Simply using the rules for expected value and variance, we get

E(X) = μ
Var(X) = σ 2 /n

But, furthermore, since X is composed of a sum of independent random variables,

we can use the Central Limit Theorem to say that the probability distribution of X
approaches a N (μ, σ 2 /n) distribution.
Combining this fact with the case discussed on page 211, we can summarize both
results as follows:
• If X1 , X2 , . . . , Xn is a random sample from a normal distribution with
E(Xi ) = μ and Var(Xi ) = σ 2 . Then the probability distribution of the

sample mean X = n1 ni=1 Xi is N (μ, σ 2 /n).
• If X1 , X2 , . . . , Xn is a random sample with E(Xi ) = μ and Var(Xi ) = σ 2
but the distribution of the Xi is not normal. Then, because of the Central
Limit Theorem, we can still say that the probability distribution of the sample

mean X = n1 ni=1 Xi is approximately N (μ, σ 2 /n).
Self-Test Exercises for Chapter 7
For each of the following multiple-choice questions, choose the best response
among those provided. Answers can be found in Appendix B.
S7.1 Let X be a random variable with probability generating function

z + z5 + z6
g(z) =
3
The value of E(X) equals
(A) 0
1
(B) 3

(C) 1
(D) 4
(E) none of the above.
S7.2 Suppose X ∼ N (2, 9) and Y ∼ N (2, 16) are independent random variables.
Then P (X + Y < 9) equals (approximately)
(A) 0.1587
(B) 0.5199
(C) 0.8413
(D) 0.9772
(E) none of the above.
S7.3 Suppose X is a random variable with moment generating function

MX (t) = E(etX ) = e0.5t

2

Let X1 and X2 be two independent random variables, each with the same
probability distribution as X . Then the moment generating function for
Y = X1 + X2 is
2
(A) e0.5t
2
(B) 2e0.5t
(C) et
2
 4
(D) e0.5t
(E) none of the above.
S7.4 An automated assembly line produces incandescent light bulbs. Suppose
that a batch of n bulbs are selected at random. Let T1 , T2 , . . . , Tn be n
independent identically distributed random variables with {Ti = 1} if light

bulb i is good, and {Ti = 0} if light bulb i is bad. Then S = ni=1 Ti is the
total number of good light bulbs in the batch. If p is the probability that any
one light bulb is good, then each Ti has probability mass function

1−p if k = 0
pTi (k) =
p if k = 1

where 0 < p < 1. Then P (S ≥ 1) equals

(A) 1 − pn
(B) 1 − (1 − p)n
! ∞ 1 −x2 /2
(C) √ e dx
1 2π

(D) 1
(E) none of the above.
S7.5 If X and Y are independent, normally distributed random variables each
with mean 0 and variance 1, then 3X − Y
(A) has a normal distribution with mean 2 and variance 1
(B) has a normal distribution with mean 0 and variance 10
(C) has a normal distribution with mean 0 and variance 2
(D) has a normal distribution with mean 2 and variance 4
(E) none of the above.
S7.6 Let X and Y be independent random variables with X ∼ N (0, 1) and
Y ∼ N (0, 2). The value of P (X > Y ) is
(A) 0
(B) 0.05
(C) 0.50
 (D) 0.95
(E) none of the above.
S7.7 Let X be a random variable with probability generating function

z(1 + z)
g(z) =
2
The value of P (X = 0) is
(A) 0
(B) 1/4
(C) 1/2
(D) 1
(E) none of the above.
S7.8 Let X1 , X2 , X3 , X4 be independent random variables, each with E(Xi ) = 0
and Var(Xi ) = 1 for i = 1, 2, 3, 4. Let Y = X1 + X2 + X3 + X4 . Then
Var(Y ) equals
(A) 1/16
(B) 1/4
(C) 4
(D) 16
(E) none of the above.
S7.9 Let X and Y be independent, normally distributed random variables each
with mean 0 and variance 1. Then the random variables

W1 = X + Y
W2 = X − Y

(A) have the same mean

(B) have the same variance
(C) are both normally distributed
 (D) all of the above
(E) none of the above.
S7.10 Let X be a random variable with moment generating function

et (1 + e2t )
MX (t) =
2
The value of E(X) is
(A) 0
(B) 1
(C) 2
(D) 3
(E) none of the above.
S7.11 Let X be a random variable with moment generating function MX (t) =
(1 − t)−1 . Let Y be a random variable with moment generating function
MY (s) = (1 − s)−1 . If X and Y are independent, then X + Y + 1 has a
moment generating function equal to
(A) et (1 − t)−2
(B) (1 − t)−2
(C) 2et (1 − t)−1
(D) es (1 − t)(1 − s)
(E) none of the above.
S7.12 Jack is attempting to fill a 100 gallon water tank using a one gallon bucket.
Jack repeatedly goes to a well at the top of a hill, fills his one gallon bucket
with water and brings that water down to the 100 gallon tank. The amount
of water he carries on any one trip from the well to the tank is normally
distributed with mean 1 gallon and standard deviation 0.1 gallon. The amount
of water carried on any trip is independent of all other trips. The probability
that Jack will need to make more than 100 trips to the well is
(A) 0.05
(B) 0.0636
 (C) 0.2580
(D) 0.5
(E) none of the above.
S7.13 Let X and Y be independent random variables with X ∼ N (0, 1) and
Y ∼ N (0, 2). The value of P (X − Y < 1) equals
(A) 0
(B) 0.5
(C) 1
(D) P (X + Y < 1)
(E) none of the above.
S7.14 Let X and Y be independent random variables with X ∼ N (1, 2) and
Y ∼ N (1, 2). Then P (|X − Y | ≤ 1) equals
(A) 0
(B) 0.3830
(C) 0.6915
(D) 1.0
(E) none of the above.
S7.15 Let X be a discrete random variable with probability mass function given
by
P (X = 0) = 13 P (X = 1) = 23
Then the moment generating function for X is
(A) e2t/3
(B) 1
3 (1 + 2et )
(C) 1
2 (1 + et )
1 t
(D) 2 e (1 + e2t )
(E) none of the above.
S7.16 Suppose X1 , X2 , . . . , Xn are independent random variables with each Xi ∼

N (2, 4) for i = 1, 2, . . . , n. Let W = ni=1 Xi and define the random
variable
W − 2n
Z=
c
Then Z ∼ N (0, 1) if and only if
(A) c = 1
(B) c = n
√
(C) c = 2 n
√
(D) c = 2/ n
(E) none of the above.
S7.17 Let X be a random variable with moment generating function

et (1 + et )
MX (t) =
2
The value of E(X) is
(A) 0
2
(B) 3

(C) 1
3
(D) 2

(E) none of the above.

S7.18 Let X1 , . . . , Xn be independent, identically distributed random variables
with each Xi having a probability mass function given by

P (Xi = 0) = 1 − p P (Xi = 1) = p

where 0 ≤ p ≤ 1. Define the random variable

n
1
Y = Xi
n i=1

Then E(Y ) equals

(A) 0
(B) p
(C) p2
(D) 1
(E) none of the above.
8
ESTIMATION

Good and bad luck are often mistaken for good and bad jusment
– ANONYMOUS, Salada TM tea bag tag

Basic Concepts
Random samples

Definition 8.1. A random sample is any sequence, X1 , X2 , . . . , Xn of indepen-

dent, identically distributed random variables.
If a continuous random variable X has an unknown parameter θ, we often write
it’s probability density function as

fX (x; θ).

Example: Let X have an exponential distribution with probability density function


1 − θ1 x
θe x>0
fX (x; θ) =
0 otherwise

where θ is unknown. Then a random sample, X1 , X2 , . . . , Xn representing n

independent observations from this random variable has a joint probability density
function given by

1 − θ1 (x1 +x2 +···+xn )
θe xi > 0 for all i
fX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ; θ) =
0 otherwise

Statistics

Definition 8.2. A statistic is any function, h(X1 , X2 , . . . , Xn ), of independent,

identically distributed random variables.

227
If a statistic is being used to establish statistical inferences about a particular
unknown parameter θ, then it is often written as θ̂(X1 , X2 , . . . , Xn ).
Note: Any function of a random sample can be called a statistic.
Example: Let X1 , X2 , . . . , Xn be a random sample. You can think of the random
variables X1 , X2 , . . . , Xn as representing the future data values that will be obtained
by making n independent observations of our experiment.
Then the following are statistics:
n
1
1. θ̂(X1 , X2 , . . . , Xn ) = Xi
n i=1

The random variable θ̂ is the average of the observations.

n
#
2. θ̂(X1 , X2 , . . . , Xn ) = Xi
i=1

The random variable θ̂ is the product of the observations.

3. θ̂(X1 , X2 , . . . , Xn ) = max(X1 , X2 , . . . , Xn )
The random variable θ̂ is the largest of the observations.
4. θ̂(X1 , X2 , . . . , Xn ) = X1
The random variable θ̂ is the first observation, with the others discarded.
There are two important statistics that arise so frequently that they are given special
symbols:
Definition 8.3. Let X1 , X2 , . . . , Xn be a random sample. The sample mean is
the statistic
n
1
X≡ Xi
n i=1
and the sample variance is the statistic
n
1 
s2X ≡ (Xi − X)2 .
n − 1 i=1
Point Estimation
The problem

Let X be a random variable with cumulative distribution function FX (x; θ) that

is completely determined except for the unknown constant θ. A random sample
X1 , X2 , . . . , Xn is to be drawn from this distribution. Suppose that a statistic
θ̂(X1 , X2 , . . . , Xn ) has been already been selected for use.
Let x1 , x2 , . . . , xn denote the actual observed values taken on by X1 , X2 , . . . , Xn .
Applying the function θ̂ to these observations yields the number θ̂(x1 , x2 , . . . , xn ).
This number can be thought of as an actual observation of the random variable
θ̂(X1 , X2 , . . . , Xn ). We will call the number θ̂(x1 , x2 , . . . , xn ) our estimate for the
value of θ. We call the random variable θ̂(X1 , X2 , . . . , Xn ) an estimator for θ.
Since any statistic can be called an estimator of any parameter, we will have to find
ways to distinguish good estimators from poor estimators.

Unbiased estimators

Definition 8.4. The estimator θ̂ is an unbiased estimator of θ if its expected

value equals θ, i.e.,
E(θ̂) = θ
for all θ.
Example: Let X be a discrete random variable with probability mass function
given by pX (0) = 1 − θ and pX (1) = θ, where θ is unknown. If X1 , X2 , . . . , Xn
is a random sample from X then
n
1
θ̂(X1 , X2 , . . . , Xn ) = Xi
n i=1

is an unbiased estimator for θ.

Finding Estimators
Method of maximum likelihood

Definition 8.5. Let X1 , X2 , . . . , Xn be a random sample from a discrete random

variable X with probability mass function pX (x; θ). The likelihood function for
 X1 , X2 , . . . , Xn is given by
n
#
L(x1 , x2 , . . . , xn ; θ) = pX (xi ; θ).
i=1

If, instead, X is a continuous random variable with probability density function

fX (x; θ), then the likelihood function is given by
n
#
L(x1 , x2 , . . . , xn ; θ) = fX (xi ; θ).
i=1

Maximum likelihood estimators are obtained by finding that value of θ that max-
imizes L for a given set of observations x1 , x2 , . . . , xn . Since the value of θ that
does this will usually vary with x1 , x2 , . . . , xn , θ can be thought of as a function of
x1 , x2 , . . . , xn , namely θ̂(x1 , x2 , . . . , xn ).
To evaluate the properties of θ̂, we can look at its performance prior to actually
making the observations x1 , x2 , . . . , xn . That is we can substitute Xi for xi in the
specification for θ̂ and look at its properties as the statistic

θ̂(X1 , X2 , . . . , Xn ).

For example, one of the properties that we might like to check for is whether θ̂ is
an unbiased estimator for θ (i.e., check to see if E(θ̂) = θ).

Self-Test Exercises for Chapter 8

For the following multiple-choice question, choose the best response among those
provided. The answer can be found in Appendix B.
S8.1 Suppose that X1 , X2 , . . . , Xn are independent identically distributed random
variables each with marginal probability density function

1 1 x−μ 2
fXi (x) = √ e− 2 σ
σ 2π
for −∞ < x < +∞, where σ > 0. Then an unbiased estimator for μ is
(A) (X1 )(X2 ) · · · (Xn )
(B) (X1 + X2 )2 /2
1 n
(C) n i=1 Xi
(D) σ
(E) none of the above.
9
PROBABILITY MODELS

Coin Tossing (Binary) Models

Definition 9.1. A Bernoulli trial is an experiment with only two outcomes. These
outcomes are conventionally called success and failure.
Definition 9.2. A Bernoulli random variable, T is a random variable with a
probability mass function given by

pT (0) = 1 − p
pT (1) = p

Definition 9.3. Let T1 , T2 , . . . , Tn be a sequence of n independent and identically

distributed Bernoulli random variables, with P (Ti = 1) = p for all i. Let X =
n
i=1 Ti . Then X is said to have a binomial distribution.

Note: X denotes the total number of successes from n independent and identically
distributed Bernoulli trials.
Theorem 9.1. If X is a binomial random variable with parameters n and p, then
 
n k
pX (k) = p (1 − p)n−k k = 0, 1, 2, . . . , n
k

The binomial distribution

Support: 0, 1, 2, . . . , n
 
n k
Probability mass function: pX (k) = p (1 − p)n−k
k
k = 0, 1, 2, . . . , n

233
 Probability generating function: g(z) = (zp + (1 − p))n
Moment generating function: MX (t) = (et p + (1 − p))n
Moments: E(X) = np Var(X) = np(1 − p)

Notation

If X has a binomial distribution with parameters n and p, we often write X ∼

B(n, p).

The geometric distribution

Support: 1, 2, . . .
Probability mass function: pX (k) = p(1 − p)k−1 k = 1, 2, . . .
zp 1
Probability generating function: g(z) = for |z| <
1 − z(1 − p) 1−p
et p
Moment generating function: MX (t) =
1 − et (1 − p)
Moments: E(X) = 1/p Var(X) = (1 − p)/p2

Note

X denotes the number of independent and identically distributed Bernoulli trials

required until the first success (counting the successful trial).
Example: A computer breaks down at least once a day with probability 0.8. It
therefore remains operational for an entire day with probability 0.2.
1. What is the probability that it breaks down exactly 3 days out of a week?
Solution: Let X denote the number of breakdowns, X ∼ B(7, 0.8). Hence,
 
7
P (X = 3) = (0.8)3 (0.2)4 = 0.287
3

2. What is the probability that it is operational for 3 days in a row before a

breakdown?
Solution: Let Y denote the number of days until a breakdown.

P (Y = 4) = (0.2)3 (0.8) = 0.0064

The Poisson distribution

Support: 0, 1, 2, . . .
e−α αk
Probability mass function: pX (k) = k = 0, 1, 2, . . .
k!
Probability generating function: g(z) = eα(z−1) for |z| < 1
t −1)
Moment generating function: MX (t) = eα(e
Moments: E(X) = α Var(X) = α

Theorem 9.2. If X is a binomial random variable with parameters n and p, then

for “large” n and “small” p
 
n k e−np (np)k
P (X = k) = p (1 − p)n−k ≈
k k!

Random Particle Models

n


1

The expected number of particles per interval of length one is equal to α. Suppose
the interval is divided into n equal segments with at most one particle placed in
any segment. What is the probability that there are exactly k particles in the unit
interval?
Let X denote the number of particles in the unit interval. X is a binomial random
variable with parameters n and p = α/n. Therefore,
    n−k
k
n α α
P (X = k) = 1−
k n n
  −k        n
α n n−1 n−k+1 αk α
= 1− ··· 1−
n n n n k! n
for k = 0, 1, 2, . . . , n.
Taking the limit as n gets large (the segment widths decrease) we obtain

αk −α
lim P (X = k) = 1(1)(1) · · · (1) e
n→∞ k!
e−α αk
=
k!
for k = 0, 1, 2, . . ..
This limiting particle process is called the Poisson Process with

α = expected number of occurrences per unit interval

X = number of occurrences in a unit interval

and the probability mass function for X is given by

αk e−α
pX (k) = k = 0, 1, 2, . . .
k!

Let Yi denote the interarrival time between occurrences i − 1 and i (see Figure 9.1).
Each of the Yi are independent and identically distributed random variables with
probability density function given by

αe−αy if y ≥ 0
fYi (y) =
0 otherwise

Question: Suppose a person begins to observe the process at time t0 chosen

independently of the future of the process? What is the distribution of the waiting
time to the next occurrence, W (see Figure 9.1)?
Theorem 9.3. 
αe−αw if w ≥ 0
fW (w) =
0 otherwise
 Yi

t
i-1 i
W
t
t0

Figure 9.1: Interarrival and waiting times

Self-Test Exercises for Chapter 9

For each of the following multiple-choice questions, choose the best response
among those provided. Answers can be found in Appendix B.
S9.1 It costs the Cheapy Flashbulb Company c cents to manufacture a single
flashbulb. The company sells each bulb for r cents. The probability that any
one bulb is good is p > 0 and it is independent of all other bulbs. Flashbulbs
are packed n to a box. If any of the bulbs in a box is bad, Cheapy will refund
the purchase price of the entire box to the customer. What is the probability
that every bulb in a box is good?
(A) 0
(B) p
(C) 1 − p
(D) pn
(E) none of the above.
S9.2 Using the information in the previous question, let Y denote the profit from
selling a box of Cheapy flashbulbs. Note that Y may be negative. The value
of E(Y ) is
(A) 0
 (B) n(rp − c)
(C) nrpn − nc
(D) nrpn + nc − 2ncpn
(E) none of the above.
S9.3 The number of knots in a piece of lumber is a random variable, X . Suppose
that X has a Poisson distribution with E(X) = α > 0. If four independent
pieces of lumber are examined, the probability that they all have no knots is
(A) 1/4
(B) 4e−α
(C) e−4α
(D) α4 e−4α
(E) none of the above.
TABLE I: STANDARD NORMAL CUMULATIVE DISTRIBUTION FUNCTION

0.40

0.35
Φ(a) 0.30

 
0.25

ϕ(z) a a
z

0.20

1 2
Φ(a) = ϕ(z) dz = √1 e− 2 z
0.15

0.10 a 2π
dz
0.05 −∞ −∞
0.00 0.1
0.3
0.5

0.8
1.0
1.2
1.5
1.7

1.9
-2.0
-1.8
-1.5

-1.3
-1.1
-0.9
-0.6

-0.4
-0.2

zz

a 0 1 2 3 4 5 6 7 8 9
0.0 0.5000 0.5040 0.5080 0.5120 0.5160 0.5199 0.5239 0.5279 0.5319 0.5359
0.1 0.5398 0.5438 0.5478 0.5517 0.5557 0.5596 0.5636 0.5675 0.5714 0.5753
0.2 0.5793 0.5832 0.5871 0.5910 0.5948 0.5987 0.6026 0.6064 0.6103 0.6141
0.3 0.6179 0.6217 0.6255 0.6293 0.6331 0.6368 0.6406 0.6443 0.6480 0.6517
0.4 0.6554 0.6591 0.6628 0.6664 0.6700 0.6736 0.6772 0.6808 0.6844 0.6879
0.5 0.6915 0.6950 0.6985 0.7019 0.7054 0.7088 0.7123 0.7157 0.7190 0.7224
0.6 0.7257 0.7291 0.7324 0.7357 0.7389 0.7422 0.7454 0.7486 0.7517 0.7549
0.7 0.7580 0.7611 0.7642 0.7673 0.7704 0.7734 0.7764 0.7794 0.7823 0.7852
0.8 0.7881 0.7910 0.7939 0.7967 0.7995 0.8023 0.8051 0.8078 0.8106 0.8133
0.9 0.8159 0.8186 0.8212 0.8238 0.8264 0.8289 0.8315 0.8340 0.8365 0.8389
1.0 0.8413 0.8438 0.8461 0.8485 0.8508 0.8531 0.8554 0.8577 0.8599 0.8621
1.1 0.8643 0.8665 0.8686 0.8708 0.8729 0.8749 0.8770 0.8790 0.8810 0.8830
1.2 0.8849 0.8869 0.8888 0.8907 0.8925 0.8944 0.8962 0.8980 0.8997 0.9015
1.3 0.9032 0.9049 0.9066 0.9082 0.9099 0.9115 0.9131 0.9147 0.9162 0.9177
1.4 0.9192 0.9207 0.9222 0.9236 0.9251 0.9265 0.9279 0.9292 0.9306 0.9319
1.5 0.9332 0.9345 0.9357 0.9370 0.9382 0.9394 0.9406 0.9418 0.9429 0.9441
1.6 0.9452 0.9463 0.9474 0.9484 0.9495 0.9505 0.9515 0.9525 0.9535 0.9545
1.7 0.9554 0.9564 0.9573 0.9582 0.9591 0.9599 0.9608 0.9616 0.9625 0.9633
1.8 0.9641 0.9649 0.9656 0.9664 0.9671 0.9678 0.9686 0.9693 0.9699 0.9706
1.9 0.9713 0.9719 0.9726 0.9732 0.9738 0.9744 0.9750 0.9756 0.9761 0.9767
2.0 0.9772 0.9778 0.9783 0.9788 0.9793 0.9798 0.9803 0.9808 0.9812 0.9817
2.1 0.9821 0.9826 0.9830 0.9834 0.9838 0.9842 0.9846 0.9850 0.9854 0.9857
2.2 0.9861 0.9864 0.9868 0.9871 0.9875 0.9878 0.9881 0.9884 0.9887 0.9890
2.3 0.9893 0.9896 0.9898 0.9901 0.9904 0.9906 0.9909 0.9911 0.9913 0.9916
2.4 0.9918 0.9920 0.9922 0.9925 0.9927 0.9929 0.9931 0.9932 0.9934 0.9936
2.5 0.9938 0.9940 0.9941 0.9943 0.9945 0.9946 0.9948 0.9949 0.9951 0.9952
2.6 0.9953 0.9955 0.9956 0.9957 0.9959 0.9960 0.9961 0.9962 0.9963 0.9964
2.7 0.9965 0.9966 0.9967 0.9968 0.9969 0.9970 0.9971 0.9972 0.9973 0.9974
2.8 0.9974 0.9975 0.9976 0.9977 0.9977 0.9978 0.9979 0.9979 0.9980 0.9981
2.9 0.9981 0.9982 0.9982 0.9983 0.9984 0.9984 0.9985 0.9985 0.9986 0.9986
3.0 0.9987 0.9987 0.9987 0.9988 0.9988 0.9989 0.9989 0.9989 0.9990 0.9990