MTH4107/MTH4207:

Introduction to Probability

Rosemary J. Harris
School of Mathematical Sciences

Notes corresponding to undergraduate lecture course

· Autumn 2020 ·
Contents

0 Prologue 1
0.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
0.2 This Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
0.3 Further Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
0.4 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1 Sample Spaces and Events 6

1.1 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 Basic Set Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 More on Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2 Properties of Probabilities 13
2.1 Kolmogorov’s Axioms . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Deductions from the Axioms . . . . . . . . . . . . . . . . . . . . . 15
2.3 Inclusion-Exclusion Formulae . . . . . . . . . . . . . . . . . . . . . 18
2.4 Equally-Likely Outcomes . . . . . . . . . . . . . . . . . . . . . . . 21
2.5 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3 Sampling 24
3.1 Basics for Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2 Ordered Sampling with Replacement . . . . . . . . . . . . . . . . . 25
3.3 Ordered Sampling without Replacement . . . . . . . . . . . . . . . 26
3.4 Unordered Sampling without Replacement . . . . . . . . . . . . . . 28
3.5 Sampling in Practice . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.6 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4 Conditional Probability 34
4.1 Introducing Extra Information . . . . . . . . . . . . . . . . . . . . 34
4.2 Implications of Extra Information . . . . . . . . . . . . . . . . . . . 36
4.3 The Multiplication Rule . . . . . . . . . . . . . . . . . . . . . . . . 38
4.4 Ordered Sampling Revisited . . . . . . . . . . . . . . . . . . . . . . 39
4.5 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

i
5 Independence 42
5.1 Independence for Two Events – Basic Definition . . . . . . . . . . 42
5.2 Independence for Two Events – More Details . . . . . . . . . . . . 43
5.3 Independence for Three or More Events . . . . . . . . . . . . . . . 45
5.4 Conditional Independence . . . . . . . . . . . . . . . . . . . . . . . 47
5.5 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

6 Total Probability and Bayes’ Theorem 51

6.1 Law of Total Probability . . . . . . . . . . . . . . . . . . . . . . . . 51
6.2 Total Probability for Conditional Probabilities . . . . . . . . . . . 53
6.3 Bayes’ Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.4 From Axioms to Applications . . . . . . . . . . . . . . . . . . . . . 57
6.5 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

7 Interlude (and Self-Study Ideas) 60

7.1 Looking Back and Looking Forward . . . . . . . . . . . . . . . . . 60
7.2 Tips for Reading the Lecture Notes . . . . . . . . . . . . . . . . . . 60
7.3 Tips for Doing Examples/Exercises . . . . . . . . . . . . . . . . . . 61

8 Introduction to Random Variables 63

8.1 Concept of a Random Variable . . . . . . . . . . . . . . . . . . . . 63
8.2 Distributions of Discrete Random Variables . . . . . . . . . . . . . 65
8.3 Properties of the Probability Mass Function . . . . . . . . . . . . . 68
8.4 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

9 Expectation and Variance 73

9.1 Expected Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
9.2 Expectation of a Function of a Random Variable . . . . . . . . . . 75
9.3 Moments and Variance . . . . . . . . . . . . . . . . . . . . . . . . . 77
9.4 Useful Properties of Expectation and Variance . . . . . . . . . . . 80
9.5 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

10 Special Discrete Random Variables 83

10.1 Bernoulli Distribution . . . . . . . . . . . . . . . . . . . . . . . . . 83
10.2 Binomial Distribution . . . . . . . . . . . . . . . . . . . . . . . . . 84
10.3 Geometric Distribution . . . . . . . . . . . . . . . . . . . . . . . . . 86
10.4 Poisson Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 87
10.5 Distributions in Practice . . . . . . . . . . . . . . . . . . . . . . . . 90
10.6 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

11 Several Random Variables 94

11.1 Joint and Marginal Distributions . . . . . . . . . . . . . . . . . . . 94
11.2 Expectations in the Multivariate Context . . . . . . . . . . . . . . 96
11.3 Independence for Random Variables . . . . . . . . . . . . . . . . . 98
11.4 Binomial Distribution Revisited . . . . . . . . . . . . . . . . . . . . 100
11.5 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

ii
12 Covariance and Conditional Expectation 103
12.1 Covariance and Correlation . . . . . . . . . . . . . . . . . . . . . . 103
12.2 Conditional Expectation . . . . . . . . . . . . . . . . . . . . . . . . 105
12.3 Law of Total Probability for Expectations . . . . . . . . . . . . . . 107
12.4 Further Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

13 Epilogue (and Exams etc.) 110

13.1 Tips for Revision and Exams . . . . . . . . . . . . . . . . . . . . . 110
13.2 Probability in Perspective . . . . . . . . . . . . . . . . . . . . . . . 111

A Errata 113

iii
Chapter 0

Prologue

0.1 Motivation
Before starting the module properly, we will spend a bit of time thinking about
what probability is and why it is important to study. As a warm-up, I invite you
to think about the following question.

Exercise 0.1: Birthday matching

What is the chance that two people in your tutorial group share a birthday?

[You may well have seen a calculation similar to this before; if not, then try to
guess what the answer might be and perhaps discuss it with your class in the first
tutorial.]
In general terms, probability theory is about “chance”; it helps to describe
situations where there is some randomness, i.e., events we cannot predict with
certainty. Such situations could be truly random (arguably like tossing a coin)
or they may simply be beyond our knowledge (like the birthdays). Of course,
probability is about more than just description – we want to be able to quantify
the randomness (loosely speaking, “give it a number”). Indeed, for Exercise 0.1
the best answer would be not just something vague like “very low” but a fraction
or decimal with value depending on the size of your group. [You should certainly
be able to do this calculation by the end of Chapter 3. In fact, most people have
rather poor intuition for the birthday problem and similar probability questions
so you might find that your initial guess was a long way from the true answer.]
Introductory probability courses are typically full of examples involving birth-
days, dice, coins, playing cards, etc. and I’m afraid you will see plenty of them
here too. They enable us to clearly demonstrate the mathematical framework of
sample spaces and events (as introduced in Chapter 1) but you should not assume
that the applications of probability are limited to such artificial scenarios. To

1
0.1 Motivation 2

name just a few more “real world” examples: forecasting the weather, predicting
financial markets, and modelling the spread of diseases all rely crucially on prob-
ability theory. At QMUL you will encounter probability in many later courses,
e.g., if you take “Actuarial Mathematics” modules you will be concerned with the
probabilities of life and death, together with their implications for life insurance.
Notice that in the above paragraphs we have not actually defined probability.
In fact, the question of what probability is does not have an entirely satisfactory
answer. We need to associate a number to an event which will measure the like-
lihood of it occurring but what does this really mean? Well, you can think of
the required number as some kind of limiting frequency – an informal definition is
given by the following procedure:

• Repeat an experiment (say, roll a die1 ) N times;

• Let A denote an event (say, the die shows an even number);

• Suppose the event comes up m times (among the N repetitions of the ex-
periment);

• Then, in the limit of very large values of N , the ratio m/N gives the proba-
bility of the event A.

In Chapter 2 we will give a more precise mathematical definition which, roughly

speaking, defines probability in terms of the properties it should have. This is
called the “axiomatic approach to probability” and we will see how many important
results can be derived from the basic axioms. Probability theory is thus a beautiful
demonstration of pure maths (prepare for some proofs!) as well as an important
tool in applied maths.
In concluding this introductory section, let me raise a few more thought-
provoking questions to illustrate some important concepts we will meet during
the course.

Exercise 0.2: Are you smarter than a pigeon?

On the mean streets of London a street performer engages you in a socially-distanced card
game. He shows you three playing cards of which one is the ace of hearts. You will win a
mystery prize if you pick this card. No prize is associated with the other cards.
The performer shuffles the cards and holds them up so he can see what they are but
you can’t. You are asked to pick a card by pointing at it. As part of the game, he now
shows you one of the two unpicked cards which he knows is not the ace. Now only the
card you picked and one remaining card are left unrevealed. You are asked if you would
like to switch your choice. Should you switch?
1
For the avoidance of doubt, “die” is the singular of “dice”.
0.2 This Course 3

[The above exercise shows the danger of assuming that all outcomes are equally
likely – we will discuss this more carefully later.]

Exercise 0.3: Innocent or guilty?

Professor Damson has been discovered dead on the top floor of the Maths Building (making
rather a mess of the nice carpet) with the murder weapon left by her body. A suspect’s
fingerprints match those found on the murder weapon. The chance of a match occurring
by chance is around 1 in 50,000. Is the suspect more likely to be innocent or more likely
to be guilty?

[This “prosecutor’s fallacy” scenario emphasizes how important it is to work out

exactly what information we are given and what we want to know. I encourage you
to think carefully about it now and return to it in due course; to answer properly
requires conditional probability (which we will introduce in Chapter 4) and Bayes’
Theorem (Chapter 6).]

Exercise 0.4: Choices

Would you rather...

• ...be given £5, or toss a coin winning £10 if it comes up heads?

• ...be given £5000, or toss a coin winning £10000 if it comes up heads?

• ...be given £1, or toss a coin 10 times winning £1000 if it comes up heads every
time?

[This last exercise is not entirely a mathematical one and there is no right or wrong
answer for each part. However some tools from probability can help to explain the
various choices. In each case the average amount we expect to win and the degree
of variation in our gain are relevant. Properties of so-called random variables
(the focus of the second half of this course) can be used to describe the choices
quantitatively. Of course there are lots of extra ingredients which will influence
an individual’s decision (for instance how useful a particular sum of money is to
them, and how much they enjoy the excitement of taking a risk). One attempt to
model some of these extra factors mathematically is the idea of utility functions
from game theory but this is beyond the scope of the present module.]

0.2 This Course

These notes define the examinable content for the course – everything here is
examinable unless explicitly stated otherwise. Each chapter corresponds to one
week’s material and the notes should always be available by the start of the relevant
week so you can read them in advance of the teaching sessions. To help you with
learning, important new probability words are generally printed in bold. You
0.3 Further Resources 4

must be sure to understand what these mean, both in everyday language (could you
explain them to your grandmother or your next-door neighbour?) and in terms
of the associated mathematical formalism. The notes also contain a number of
“examples” and “exercises”. For the former, you will find full details of the working
which you should study carefully to see not just how to get correct answers but
how to present your solutions (an important skill for university-level mathematics).
For the latter, at most the answers are in the notes although some solutions will
be presented in interactive sessions (see below). A good way to check your own
understanding would be to read the text and try the associated exercises – please
ask if you have any difficulty getting answers. I intend the unstarred exercises to
correspond roughly to the Key Objectives for the course (i.e., everyone should be
able to do them); single-starred exercises should be attempted by those aiming
for a high grade and double-starred exercises are for those seeking a real challenge
(beyond the syllabus).
The content of the notes will also be presented in a series of recorded video
lectures (typically one per section) in which the main points will be emphasized
and further explanation and exam tips given. These lectures will feature some
“real-time” written demonstrations and you are strongly recommended to follow
along and fill in the gaps in the slides or annotate these notes – for most people,
the practice of writing out maths (rather than just looking at a screen) helps the
material to be absorbed and the necessary notation to become automatic. Corre-
sponding to each video lecture there will also be an interactive session in which
the solutions to some of the associated exercises will be presented and there will
be an opportunity for queries/misunderstandings to be addressed. The QMplus
page contains full information about course practicalities (timetable, coursework,
assessment details, etc.,) and important announcements/corrections will also be
posted there.

0.3 Further Resources

The course is designed to be fairly self-contained and does not follow any one
textbook. However, if you would like further background reading (and many more
practice exercises!) then the following are recommended.

• Sheldon Ross, A First Course in Probability (Pearson, 2020):

– Cited in these notes as [Ros20];
– Probably closest to the general treatment of this course.
0.4 Acknowledgements 5

• David F. Anderson, Timo Seppäläinen, and Benedek Valkó, Introduction to

Probability (Cambridge, 2018):
– Cited in these notes as [ASV18];
– Slightly more rigorous but very readable.
• Henk Tijms, Understanding Probability (Cambridge, 2012):
– Cited in these notes as [Tij12];
– Content covered in this module is contained in Chapters 7–9, while parts
of Chapters 1–6 nicely motivate the theoretical ideas with examples.
[All of these are available in physical and electronic form from the QMUL library.]

0.4 Acknowledgements
Large sections of these notes are based on those of previous lecturers, Dr Robert
Johnson and Dr Wolfram Just, and I am greatly indebted to both of them. How-
ever, the responsibility for errors and typos remains mine; if you find any, please
contact me at [email protected].
Chapter 1

Sample Spaces and Events

1.1 Framework
The general setting is that we perform an experiment and record an outcome.
Outcomes must be precisely specified, mutually exclusive and cover all possibilities.
Definition 1.1. The sample space is the set of all possible outcomes for the
experiment. It is denoted by S (or Ω).
Definition 1.2. An event is a subset of the sample space. The event occurs if
the actual outcome is an element of this subset.
To see how the framework is applied in practice, let’s consider some examples.
We start with a very simple one.
Example 1.1: Die roll
Suppose your experiment is to roll an ordinary six-sided die and record the number showing
as the outcome.
(a) Use set notation to write down the sample space.
(b) Denote by A the event “You roll an even number” and write it as a subset of the
sample space.
Solution:
(a) The sample space is the set containing the integers 1 to 6 (inclusive) so we write

S = {1, 2, 3, 4, 5, 6}.

(b) The event corresponding to the rolling of an even number is A = {2, 4, 6}.

The situation is more complicated when the outcome is not just an observation
of a single thing. In this case, in choosing your notation you need to think about
whether order matters.

6
1.1 Framework 7

Example 1.2: Tossing thrice

A coin is tossed three times and the sequence of Heads/Tails is recorded.

(a) Use set notation to write down the sample space.

(b) State the following events as subsets of the sample space:

“Exactly one Head is seen”;
“The second toss is a Head”.

Solution:

(a) Denoting a Head with h and a Tail with t, we can write the sample space as

S = {hhh, hht, hth, htt, thh, tht, tth, ttt}

where, for example, htt means the first toss is a Head, the second is a Tail, and the
third is also a Tail. Note that here order matters: htt is not the same as tth so we have
to include both. In general, if we need to include information about order, we can
either write an ordered list as above or use round brackets and commas, e.g., (h, t, t).
However, this is not the only way to write the sample space for this example. If you are
a (lazy) experimenter recording outcomes you might realise that you only need to note
which tosses are, say, Heads and you have all the information about what happened.
So, adding a tilde (“squiggle”) to the S to indicate we’ve changed notation, we could
write
S̃ = {{1, 2, 3}, {1, 2}, {1, 3}, {2, 3}, {1}, {2}, {3}, {}}

where we record the set of tosses which are Heads so the outcome {1, 3} corresponds
to hth. Note that the curly braces indicate that order doesn’t matter; {3, 1} would
mean exactly the same thing as {1, 3}. [Here we are representing the outcomes within
the sample space as sets and in set notation the order of elements is unimportant –
for a brief review of set theory see the next section.]

(b) Using the first notation for the sample space and denoting the events “Exactly one
Head is seen” and “The second toss is a Head” by B and C respectively, we have

B = {htt, tht, tth} and C = {hhh, hht, thh, tht},

whereas with the second notation, we have,

B̃ = {{1}, {2}, {3}} and C̃ = {{2}, {1, 2}, {2, 3}, {1, 2, 3}}.

[Normally you should avoid using two different notations within the same solution but, if
for some reason you need to, you should distinguish them clearly, e.g., by S and S̃, S and
S1 , or S and S 0 ; in the last case, the prime is not to be confused with the set complement
for which we will use a c superscript. Notice too that the number of elements is always
invariant under a change of notation and this can sometimes provide a way to check your
answers are sensible.]
1.2 Basic Set Theory 8

Exercise 1.3: Two die rolls

You throw an ordinary six-sided die and write down the number showing. Then you throw
it again and write down the number showing.
(a) Write down the sample space for this experiment.
(b) How many elements does the sample space contain?
(c) Write in words some possible events corresponding to this experiment and then state
them as subsets of the sample space.

In this course we will mainly be interested in experiments with discrete out-

comes and finite sample spaces (especially when proving rigorous results). How-
ever, we will also encounter some cases where the sample space contains a countably
infinite number of elements. [The extension to continuous outcomes will be left
for future courses.]
Exercise 1.4: Exam strategy
Suppose your experiment is to take an exam repeatedly until you pass.1 Ignoring compli-
cations such as finite lifetimes, which of the following could represent the sample space?
(a) S1 = {P, F P, F F P, F F F P, . . .},
(b) S2 = {1, 2, 3, 4 . . .},
(c) S3 = {0, 1, 2, 3, . . .},
(d) S4 = {F F F P, F F P, F P, P }.

Definition 1.3. An event E is a simple event (or elementary event) if it

consists of a single element of the sample space S.
Exercise 1.5: Simple events
Identify some simple events from the examples in this section.

1.2 Basic Set Theory

We use extensively the terminology and notation of basic set theory.2 Informally
a set is an unordered collection of well-defined distinct objects. For instance,
{b, −2.4, 2} is a set. However, strictly speaking, {2, 3, 3} is not a set because it
contains repeated elements (not distinct). {2, 3, 5, 1} and {3, 5, 2, 1} are the same
set as order does not matter. Indeed, two sets A and B are equal, A = B, if they
contain precisely the same elements.
A set can be specified in various ways:
1
This is not a good strategy at QMUL – you only get two attempts at each exam!
2
If you are taking the module “Numbers, Sets and Functions”, you should find some helpful
overlap in material.
1.2 Basic Set Theory 9

• By listing all the objects in it between braces ({, }) separated by commas,

e.g., {1, 2, 3, 4}.

• By listing enough elements to determine a pattern (usually for infinite sets),

e.g., {2, 4, 6, 8, . . . } which is the set of positive even integers. [A set which
can be written as a comma-separated list is said to be countable.]

• By giving a rule, e.g., {x : x is an even integer} which we read as “the set of

all x such that x is an even integer”.

If A is a set we write x ∈ A to mean that the object x is in the set A and say that x
is an element of A. If x is not an element of A then we write x 6∈ A. For a finite set,
the size (or cardinality) is just the number of elements; if A = {a1 , a2 , . . . , an },
we write |A| = n. [Do not confuse the size of a set with the absolute value of a
number.]
We now summarize some facts which will be useful in the rest of this course.
Let A and B both be sets.

• A ∪ B (“A union B”) is the set of elements in A or B (or both):

A ∪ B = {x : x ∈ A or x ∈ B}. (1.1)

• A ∩ B (“A intersection B”) is the set of elements in both A and B:3

A ∩ B = {x : x ∈ A and x ∈ B}. (1.2)

• A \ B (“A take away B”) is the set of elements in A but not in B:

A \ B = {x : x ∈ A and x 6∈ B}. (1.3)

• A4B (“symmetric difference of A and B”) is the set of elements in either A

or B but not both:
A4B = (A \ B) ∪ (B \ A). (1.4)

• If all the elements of A are contained in the set B, we say that A is a subset
of B and write A ⊆ B.

• If all sets are subsets of some fixed set S, then Ac (“the complement of A”)
3
Some books, including [Ros20, ASV18, Tij12], use AB without a “cap” in the middle to
denote the intersection of two events.
1.2 Basic Set Theory 10

is the set of all elements of S which are not elements of A:

Ac = S \ A. (1.5)

• We say two sets A and B are disjoint (or mutually exclusive) if they have
no element in common, i.e., A ∩ B = {}. The empty set {} is often denoted
by ∅.
Exercise 1.6: Venn diagrams
Draw Venn diagrams to illustrate the bullet points above.

Exercise 1.7: Symmetric difference

Express A4B in terms of intersections, unions, and complements.

Exercise 1.8: Disjoint decomposition

Suppose that A and B are subsets of S.
(a) Show that A ∩ B and A ∩ B c are disjoint.
(b) Express A as a union of two disjoint sets.
(c) Express A ∪ B as a union of three mutually exclusive sets.
[These kind of tricks will be important later on.]

Exercise 1.9: Set examples (based on [ASV18])

Consider the following three sets:

A = {2, 3, 4, 5, 6, 7, 8}, B = {1, 3, 5, 7}, C = {2, 4, 5, 8}.

(a) Can the set D = {2, 4, 8} be expressed in terms of A, B, and C using intersections,
unions, and complements?
(b) Can the set E = {4, 5} be expressed in terms of A, B, and C using intersections,
unions, and complements?

It is very useful to remember the following two identities which are known as
De Morgan’s laws:

(A ∪ B)c = Ac ∩ B c , (1.6)
(A ∩ B)c = Ac ∪ B c . (1.7)

Exercise 1.10: *De Morgan’s laws

(a) You may have seen a proof of De Morgan’s laws elsewhere. Show that, in fact, you
can derive (1.7) from (1.6) so you only really need to remember one of them.
(b) Write down (or look up) the generalization of De Morgan’s laws to n sets.
1.3 More on Events 11

1.3 More on Events

The previous section was about set theory in general – we now return to the
implications for events. Let A and B denote events (i.e. subsets of the sample
space S).

• If A is an event then Ac contains the elements of the sample space which are
not contained in A, i.e., Ac is the event that “A does not occur”.

• If A and B are events then the event E1 “A and B both occur” consists of
all elements of both A and B, i.e., E1 = A ∩ B.

• The event E2 “at least one of A or B occurs” consists of all elements in A

or B, i.e., E2 = A ∪ B.

• The event E3 “A occurs but B does not” consists of all elements in A but
not in B, i.e., E3 = A \ B.

• The event E4 “exactly one of A or B occurs” consists of all elements in A or

B but not in both, i.e., E4 = A4B.

Example 1.11: Die roll (revisited)

You roll a die as in Example 1.1. Let A be the event that an even number occurs and B
be the event that the outcome is a prime number. Express the following events in terms
of A and B and write them explicitly as subsets of the sample space:
F1 : “The outcome is an even number or a prime”;
F2 : “The outcome is either an even number or a prime (i.e., not both)”.

Solution:
We have A = {2, 4, 6} (see Example 1.1) and B = {2, 3, 5}. Using these, the required
events are
F1 = A ∪ B = {2, 3, 4, 5, 6},

and
F2 = A4B = {3, 4, 5, 6}.

Exercise 1.12: Lecture attendance

Four students, Amina, Brandon, Chloe and Daniel, are supposed to be attending a lecture
course. In the last lecture of the semester their attendance is recorded.

(a) Write down the sample space, explaining your notation carefully.

(b) Write down the event “Exactly three of them attend the last lecture” as a set.

(c) Write down the event “Amina attends the last lecture but Daniel does not” as a set.
1.4 Further Exercises 12

Exercise 1.13: Rugby squad

One player is chosen at random from a squad of world cup rugby players. Let C be the
event “the player chosen is the captain”, F be the event “the player chosen is a forward”,
and I be the event “the player chosen is injured”.
(a) Express the following in symbols:

(i) The event “an injured forward is chosen”;

(ii) The event “the chosen player is a forward but is not the captain”;
(iii) The statement “none of the forwards are injured”.

(b) Express the following in words:

(i) The event F c ∪ I c ;

(ii) The statement |I c | < 15.

1.4 Further Exercises

Exercise 1.14: Real life probability
Find an article from the news illustrating a “real world” scenario where probability might
be important. Try to identify the experiment, the possible outcomes, and some events
which might be of interest.

Exercise 1.15: Horse racing

A race takes place between three horses Adobe, Brandy, and Chopin. It is possible that
one or more of them may fall and so fail to complete the race. The finishing horses are
recorded in the order in which they finish.
(a) Write down the sample space, explaining your notation.
(b) Write down the event “The race is won by Adobe” as a set.
(c) Write down the event “Brandy falls” as a set.
(d) Write down the event “All horses complete the race” as a set.

Exercise 1.16: Rules for stopping

You toss an ordinary coin repeatedly, recording the outcome of each toss. You do this
until you have seen either two Heads or three Tails in total and then you stop.
(a) Write down the sample space.
(b) Write down the event “you toss the coin exactly four times” as a subset of the sample
space.
I perform the same experiment but I do not stop until I have seen either seven Heads or
eight Tails in total. Let Ei be the event “I toss the coin exactly i times”.
(c) For which i is it the case that Ei = ∅?
Chapter 2

Properties of Probabilities

2.1 Kolmogorov’s Axioms

We want to assign a numerical value to an event which reflects the chance that it
occurs. To be more precise, probability is a concept/recipe (or, in formal terms, a
function) P which assigns a (real) number P(A) to each event A.
Sometimes we have some intuition about how probabilities should be assigned.
For example, if we toss a fair coin we should have probability 1/2 of seeing Heads
and probability 1/2 of seeing Tails. This is an example of the special case where all
outcomes are equally likely and the probability of an event A is just the ratio of the
number of outcomes in A to the total number of outcomes in S. We will explore
this situation more in Section 2.4 but note that it does not constitute a general
recipe for probability – one cannot just assume equally-likely outcomes without
good reason! For example, what if we toss a biased coin? What probabilities would
be reasonable in that case? Can they be any real numbers?
The formal approach is to regard probability as a mathematical construction
satisfying certain axioms.1

Definition 2.1 (Kolmogorov’s axioms for probability). Probability is a function

P which assigns to each event A a real number P(A) such that:

(a) For every event A we have P(A) ≥ 0,

(b) P(S) = 1,

(c) If A1 , A2 , . . . , An are n pairwise disjoint events (Ak ∩ A` = ∅ for all k 6= `)

1
Roughly speaking, an axiom is a statement which is assumed to be true and then can be used
to make other logical deductions.

13
2.1 Kolmogorov’s Axioms 14

then
n
X
P(A1 ∪ A2 ∪ · · · ∪ An ) = P(A1 ) + P(A2 ) + · · · + P(An ) = P(Ak ).
k=1

Remarks:

• The function P is sometimes called a probability measure.

• Pairwise disjoint in Definition 2.1(c) simply means that every pair of events
is disjoint. [Equivalently, we can use the term mutually exclusive.]

• Definition 2.1(c) is here stated for a finite number of events. In fact, there
is a version for a countably infinite number of events as well but we do not
state that formally in this module as we would need to clarify the notion of
an infinite sum first.2 Further subtleties occur if S is not countable.

Exercise 2.1: Fair coin

Show that the probabilities associated with a single toss of a fair coin satisfy Kolmogorov’s
Axioms. [Hint: We will check this for the whole class of experiments with equally-likely
outcomes in Section 2.4.]

Example 2.2: Biased coin

Consider a single toss of a biased coin where the outcome is still Heads (h) or Tails (t)
but the probabilities of the possible events are given by:

P(∅) = 0 [Seeing nothing is not possible.],

1
P({h}) = ,
3
2
P({t}) = ,
3
P({h, t}) = 1 [We must see either a Head or a Tail.].

Show that this defines a probability measure, i.e., that the given probabilities satisfy
Kolmogorov’s Axioms.

Solution:
All four events (i.e., all four subsets of the sample space {h, t}) have probabilities greater
than or equal to zero so Definition 2.1(a) is obviously satisfied. We also straightforwardly
have P(S) = P({h, t}) = 1 so Definition 2.1(b) is also satisfied. For Definition 2.1(c) we
2
A concept many of you will meet in the module “Calculus II”.
2.2 Deductions from the Axioms 15

need to check all possible unions of pairwise disjoint events:

1 2
P({h} ∪ {t}) = P({h, t}) = 1 = + = P({h}) + P({t}),
3 3
1 1
P(∅ ∪ {h}) = P({h}) = = 0 + = P(∅) + P({h}),
3 3
2 2
P(∅ ∪ {t}) = P({t}) = = 0 + = P(∅) + P({t}),
3 3
P(∅ ∪ {h, t}) = P({h, t}) = 1 = 0 + 1 = P(∅) + P({h, t}),
1 2
P(∅ ∪ {h} ∪ {t}) = P({h, t}) = 1 = 0 + + = P(∅) + P({h}) + P({t}).
3 3
Hence Definition 2.1(c) is satisfied and we have a probability measure as required.

Note that for simple events you may sometimes see notation like P(h) as a
shorthand for P({h}) but it is better to include the curly braces as a reminder
that events are always sets.
Exercise 2.3: A more complicated construction
Let S = {1, 2, 3, 4, 5, 6, 7, 8} and for A ⊆ S define a probability measure by

1
P(A) = (|A ∩ {1, 2, 3, 4}| + 2|A ∩ {5, 6, 7, 8}|) .
12
(a) Verify that this satisfies the axioms for probability.
(b) Give an example of a physical situation which this probability measure could describe.

2.2 Deductions from the Axioms

Starting from the axioms we can deduce various properties. Hopefully, these will
agree with our intuition about probability – if they did not then this would suggest
that we had not made a good choice of axioms! The proofs of all of these are simple
deductions from the axioms.

Proposition 2.2. If A is an event then

P(Ac ) = 1 − P(A).

This statement makes perfect sense: if P(A) is the probability of the event A
then the probability of the complementary event Ac should be 1 − P(A). [This
can be very useful in calculations; sometimes it is much easier to calculate P(Ac )
than P(A).] Although the proposition may be obvious, we want to provide a
formal proof starting from Definition 2.1 to provide evidence that our axioms are
consistent with the real world and to demonstrate the structure of probability
theory.
2.2 Deductions from the Axioms 16

Proof:
Let A be any event. Then we can set A1 = A and A2 = Ac . By definition of the
complement, A1 ∩ A2 = ∅ and so we can apply Definition 2.1(c), with n = 2, to
get
P(A1 ∪ A2 ) = P(A1 ) + P(A2 ) = P(A) + P(Ac ). (2.1)

Now, again by definition of the complement, A1 ∪ A2 = S so, using Defini-

tion 2.1(b),
1 = P(S) = P(A1 ∪ A2 ). (2.2)

Hence, combining (2.1) and (2.2), we have

1 = P(A) + P(Ac )

and rearranging this gives the desired result.

We can use what we have just proved to deduce further results which are
called here “corollaries” as they are straightforward consequences of the preceding
proposition.

Corollary 2.3.
P(∅) = 0.

This statement makes perfect sense as well. The probability of “no outcome” is
zero. [We already had this in Example 2.2 but we now see that it has to be true.]

Proof:
By definition of the complement, S c = S \ S = ∅. Hence, by Proposition 2.2,

P(∅) = P(S c ) = 1 − P(S).

Then, using Definition 2.1(b), we have

P(∅) = 1 − 1 = 0,

as required.

Corollary 2.4. If A is an event then P(A) ≤ 1.

Again the statement agrees with our intuition. Probabilities are always smaller
than or equal to one.

Proof:
By Proposition 2.2,
P(Ac ) = 1 − P(A).
2.2 Deductions from the Axioms 17

But Ac is also an event so, by Definition 2.1(a),

0 ≤ P(Ac ) = 1 − P(A)

and hence
P(A) ≤ 1,

as required.

The following statements are less obvious consequences of Definition 2.1 and
the statements we have shown so far. Thus we call them again propositions.

Proposition 2.5. If A and B are events and A ⊆ B then

P(A) ≤ P(B).

This statement looks sensible as well. If an event B contains all the outcomes of
an event A then the probability of the former must be at least as big as that of
the latter.

Proof:
Consider the events A1 = A and A2 = B \ A, with A ⊆ B. Then A1 ∩ A2 = ∅ (i.e.,
the two events are pairwise disjoint) and A1 ∪ A2 = B. [A Venn diagram may help
you to see what is going on here.] So, by Definition 2.1(c), with n = 2,

P(B) = P(A1 ∪ A2 ) = P(A1 ) + P(A2 ) = P(A) + P(B \ A).

Since B \ A is also an event, Definition 2.1(a) tells us that

P(B) − P(A) = P(B \ A) ≥ 0,

and the required statement follows by rearrangement.

Proposition 2.6. If A = {a1 , a2 , . . . , an } is a finite event then

n
X
P(A) = P({a1 }) + P({a2 }) + · · · + P({an }) = P({ai }).
i=1

This statement is quite remarkable. The probability of a (finite) event is the sum
of the probabilities of the corresponding simple events. [Again, you may see P(ai )
for P({ai }) although writing the former should really be avoided.]

Proof:
Denote the simple events by Ai = {ai }, i = 1, .., n. These events are pairwise
2.3 Inclusion-Exclusion Formulae 18

disjoint and A1 ∪ A2 ∪ · · · ∪ An = A. Hence by Definition 2.1(c),

P(A) = P(A1 ∪ A2 ∪ · · · ∪ An )
= P(A1 ) + P(A2 ) + · · · + P(An )
= P({a1 }) + P({a2 }) + · · · + P({an }),

as required.

Remarks:

• Note that combining Proposition 2.6 with Definition 2.1(b) leads to the ob-
vious fact that the sum of the individual probabilities for all outcomes in the
sample space is unity. In other words, this agrees with our intuition that
“probabilities sum to one”.

• Notice that most of the proofs in this section involve expressing events of in-
terest as the union of disjoint events so that Definition 2.1(c) can be applied.
This is a powerful general strategy.
Exercise 2.4: Rugby squad (revisited)
Consider the set-up of Exercise 1.13. Suppose that 50% of the squad are forwards, 25%
are injured and 10% are injured forwards and that the player chosen is equally likely to be
any member of the squad. Calculate the probability that the chosen player is a forward
who is not injured.

Exercise 2.5: Friendly discussion

Let A and B be events with P(A) = 1/2, P(B) = 1/4, P(A ∩ B) = 1/10. Your friend says
that P(A ∪ B) = 3/4. Explain carefully whether or not they are correct.

2.3 Inclusion-Exclusion Formulae

We now move on to what was once described by the Italian-American mathe-
matician Gian-Carlo Rota as “One of the most useful principles of enumeration in
discrete probability and combinatorial theory” [Rot64].

Proposition 2.7 (Inclusion-exclusion for two events). For any two events A and
B we have
P(A ∪ B) = P(A) + P(B) − P(A ∩ B).

This statement is not entirely obvious. For general events the probability of the
event “A or B” is not the sum of the probabilities of events A and B; in fact,
because of some “double counting” one needs to correct by the probability of the
event “A and B”.
2.3 Inclusion-Exclusion Formulae 19

Proof:
Consider the three events E1 = A \ B, E2 = A ∩ B and E3 = B \ A. The events
are pairwise disjoint and E1 ∪ E2 ∪ E3 = A ∪ B. [This should remind you of
Exercise 1.8.] Hence, by Definition 2.1(c) with n = 3,

P(A ∪ B) = P(E1 ) + P(E2 ) + P(E3 ).

Furthermore E1 ∪ E2 = A and E2 ∪ E3 = B. Thus Definition 2.1(c) with n = 2

yields

P(A) = P(E1 ) + P(E2 ),

P(B) = P(E2 ) + P(E3 ).

Since P (A ∩ B) = P(E2 ), we finally have

P(A) + P(B) − P(A ∩ B) = P(E1 ) + P(E2 ) + P(E2 ) + P(E3 ) − P(E2 )

= P(E1 ) + P(E2 ) + P(E3 )
= P(A ∪ B),

as required.

Although normally written in the form above, the inclusion-exclusion formula

can obviously be rearranged to express any one of P(A), P(B), P(A ∪ B), P(A ∩ B)
in terms of the other three.
Example 2.6: A tale of two cities
A bus operator operates two lines each connecting the city Xinji with Yiwu. Line 1 runs
a bus A from Xinji to Ulanqab and a bus B from Ulanqab to Yiwu, so that passengers
have to change at Ulanqab. Line 2 runs a bus C from Xinji to Weihui and a bus D from
Weihui to Yiwu, so that passengers have to change at Weihui. Buses are running with
probabilities P(A) = 0.9, P(B) = 0.8, P(C) = 0.7, P(D) = 0.8. Furthermore the company
ensures that at least three of the four buses are running.
Using the inclusion-exclusion principle, or otherwise, compute the probability that I
can travel via line 1 from Xinji to Yiwu. [2018 exam question (part)]

Solution:
The probability of being able to travel via line 1 is P (A∩B). Using the inclusion-exclusion
formula we can write this as

P(A ∩ B) = P(A) + P(B) − P(A ∪ B)

but this only helps if we know P(A ∪ B). The trick here is to remember that the union of
A and B means at least one of A and B occurs. Since at least three of the four buses are
2.3 Inclusion-Exclusion Formulae 20

running, at least one of the two on line 1 must must be running and hence P (A ∪ B) = 1.
The probability of being able to travel via line 1 is thus

P(A ∩ B) = 0.9 + 0.8 − 1 = 0.7.

Proposition 2.8 (Inclusion-exclusion for three events). For any three events A,
B, and C we have

P(A∪B ∪C) = P(A)+P(B)+P(C)−P(A∩B)−P(A∩C)−P(B ∩C)+P(A∩B ∩C).

As for two events, there exists an “intuitive” argument but that is not a proof.

Proof:
Essentially we will apply Proposition 2.7 three times. Let D = A ∪ B so that
A ∪ B ∪ C = C ∪ D. Then

P(A ∪ B ∪ C) = P(C ∪ D)
= P(C) + P(D) − P(C ∩ D) [by Proposition 2.7]
= P(C) + P(A ∪ B) − P(C ∩ D)
= P(C) + P(A) + P(B) − P(A ∩ B) − P(C ∩ D) [by Proposition 2.7].
(2.3)

Now C ∩ D = C ∩ (A ∪ B) = (C ∩ A) ∪ (C ∩ B) so that

P(C ∩ D) = P((C ∩ A) ∪ (C ∩ B))

= P(C ∩ A) + P(C ∩ B) − P((C ∩ A) ∩ (C ∩ B)) [by Proposition 2.7]
= P(C ∩ A) + P(C ∩ B) − P(A ∩ B ∩ C). (2.4)

Finally, substituting (2.4) in (2.3) yields

P(A∪B ∪C) = P(C)+P(A)+P(B)−P(A∩B)−P(C ∩A)−P(C ∩B)+P(A∩B ∩C),

as required.

Exercise 2.7: Three events

Suppose that the probabilities for each of the three events A, B, and C are 1/3, i.e.

1
P(A) = P(B) = P(C) = .
3
Furthermore, assume that the probabilities for each of the events “A and B”, “A and C”,
2.4 Equally-Likely Outcomes 21

and “B and C” are 1/10, i.e.,

1
P(A ∩ B) = P(A ∩ C) = P(B ∩ C) = .
10
What can be said about the probability of the event that none of A, B, or C occur?

Exercise 2.8: **Inclusion-exclusion for more events

(a) Derive the inclusion-exclusion formula for four events.

(b) Now try to write down a general formula for n events. How could you prove it?

Remark: Notice how the few simple axioms of Section 2.1 led to a plethora of
results in Sections 2.2 and 2.3 – such structure is essentially what mathematics is
about. In the next section, we will see how the general framework applies to the
special case of equally-likely outcomes.

2.4 Equally-Likely Outcomes

As mentioned at the beginning of Section 2.1, in some situations it is reasonable
to say that the probability of an event A is the ratio of the number of outcomes
in A to the total number of outcomes in S, i.e., to define probability by

|A|
P(A) = . (2.5)
|S|

For example, if we roll a fair die then the sample space is S = {1, 2, 3, 4, 5, 6} and
the probability of the event A “the number shown is smaller than 3” is given by

|{1, 2}| 2 1
P(A) = = = .
|{1, 2, 3, 4, 5, 6}| 6 3

We emphasize again that we are using here that every outcome in the sample space
is equally likely (we say “we pick an outcome at random”) and this special case
should not be assumed without justification; for example, it wouldn’t apply to a
biased die. We also run into difficulties if S is infinite. If S = N (the set of positive
integers) as in Exercise 1.4, then there is no reasonable way to choose an element
of S with all outcomes equally likely; there are, however, ways to choose so that
every positive integer has some chance of occurring.

Example 2.9: Checking the axioms

Suppose that the sample space S is finite. Show that defining probabilities according
to (2.5) satisfies Kolmogorov’s Axioms.
2.4 Equally-Likely Outcomes 22

Solution:
We need to check all three parts of Definition 2.1.

(a) If A ⊆ S then |A| ≥ 0 and

|A| 0
P(A) = ≥ = 0.
|S| |S|

(b) We straightforwardly have

|S|
P(S) = = 1.
|S|

(c) If A1 , A2 , . . . , An are pairwise disjoint subsets of S then

|A1 ∪ A2 ∪ · · · ∪ An | = |A1 | + |A2 | + · · · + |An |,

so
|A1 | + |A2 | + · · · + |An |
P(A1 ∪ A2 ∪ · · · ∪ An ) =
|S|
|A1 | |A2 | |An |
= + + ··· +
|S| |S| |S|
= P(A1 ) + P(A2 ) + · · · + P(An ).

Hence all three axioms are satisfied.

Since Definition 2.1 is fulfilled, all the other results in Sections 2.2 and 2.3 also
hold. In particular, the inclusion-exclusion formula for two events reads

|A ∪ B| |A| |B| |A ∩ B|
= + −
|S| |S| |S| |S|

which implies
|A ∪ B| = |A| + |B| − |A ∩ B|.

This last statement is a statement about the sizes of finite sets which can be proved
in other ways as well. This cross-fertilization of ideas between different branches
of mathematics is an important part of higher-level study (and research!).
Note that in this set-up of equally-likely outcomes, calculating probabilities
becomes counting! In the next chapter we will see some combinatorial arguments
for finding |A| and |S| in different situations.

Exercise 2.10: Getting out of jail

In the game of Monopoly one way to get out of jail is to throw “a double”. What it the
probability that when you roll two fair six-sided dice they both show the same number?

Exercise 2.11: Two Heads are better than one

You toss a fair coin five times. What is the probability that you see at least two Heads?
2.5 Further Exercises 23

2.5 Further Exercises

Exercise 2.12: Probability calculations
Let A and B be events with P(A) = 1/2, P(B) = 1/4, P(A ∩ B) = 1/10. Calculate the
following probabilities:

(a) P(B c ),

(b) P(A ∪ B),

(c) P(A ∩ B c ).

Exercise 2.13: Zeros

Let A and B denote two events with P(A ∪ B) = 0. Show that P(A) = 0 and P(B) = 0.

Exercise 2.14: More identities

Suppose that A and B are events.

(a) Using the inclusion-exclusion principle, or otherwise, show that

P(A ∩ B) ≥ P(A) + P(B) − 1.

(b) Using the events C = A4B and D = A ∩ B, or otherwise, show that

P(A4B) = P(A) + P(B) − 2P(A ∩ B).

Make sure that each step in your proofs is justified by a definition, axiom or result from
the notes (or is a simple manipulation).
Chapter 3

Sampling

3.1 Basics for Sampling

Throughout this chapter we focus again on the special case where all elements of
the sample space are equally likely, as in Section 2.4. In this situation, calculating
probability essentially boils down to counting the number of ways of making some
selection. Specifically, we are often interested in finding how many ways there are
of choosing r things from a collection of n things. This is called sampling from
an n-element set. The number of ways of doing this depends on exactly what
we mean by selection: is the order important and is repetition allowed? We will
see three distinct cases in these notes and illustrate them with examples.
Before getting into the details, it is worth mentioning a fundamental idea that
we use implicitly: the so-called “basic principle of counting” [Ros20]: if there are
m possible outcomes of experiment 1 and n possible outcomes of experiment 2,
then there are m × n outcomes of the two experiments together. For example,
if there are four main courses on a cafeteria menu and three desserts then there
are a total of twelve different meals which can be chosen. This principle can be
straightforwardly generalized to more than two experiments.
Let us also emphasize some notation which will be important in the following
sections. As we already know, a set {a1 , a2 , . . . , an } is an unordered collection of
n distinct objects. In contrast, an n-tuple (b1 , b2 , . . . bn ) is an ordered collection of
n objects which are not necessarily distinct – you can think of this as a coordinate
in an n-dimensional space. A 2-tuple is a “pair”, e.g., (1, 2); a 3-tuple is a “triple”,
e.g., (1,2,1), a 4-tuple is a “quadruple”, e.g., (1, 2, 1, 3), and so on.

24
3.2 Ordered Sampling with Replacement 25

3.2 Ordered Sampling with Replacement

Example 3.1: Words
How many ordered three-letter strings can be made from the letters A, B, C, D, E if
repetition is allowed? [We will call these “words” although they don’t have to be real words
in English, for example BDB, ACE, CAB, ABC, DDD, are all allowable possibilities.]

Solution:
This is an ordered selection of three things from a collection of five things with repetition
allowed – you can think of taking Scrabble tiles from a bag containing (one each of) A, B,
C, D, E, and replacing each letter after you have written it down. There are five choices
for the first letter. For each of these, there are five choices for the second letter and, for
each of these, there are five choices for the third letter. Hence, using the basic principle
of counting, there are a total of 5 × 5 × 5 = 53 = 125 possible words.
In formal terms, we let the set of letters be U = {A, B, C, D, E}. Then the experiment
is to pick three letters in order and each outcome can be written as a triple, e.g., (A, C, E).
The sample space of all words is the set of all such triples, i.e., S = {(s1 , s2 , s3 ) : si ∈ U }
and we have |S| = 53 .

The above example illustrates the general principle of this section:

• If we make an ordered selection of r things from a set U = {u1 , u2 , . . . , un }

with replacement (i.e., we allow repetition, an element can be selected more
than once) then the sample space is the set of all r-tuples consisting of
elements of U . That is

S = {(s1 , s2 , . . . , sr ) : si ∈ U }.

• If |U | = n there are n choices for u1 ; for each of these, there are n choices
for u2 , and so on. Hence, we have

|S| = |U |r = nr . (3.1)

To determine the probability of a given event, in the framework of equally-

likely outcomes, we also need to calculate the cardinality of that event. This can
often be done by straightforwardly applying the basic principle of counting, in a
similar way to the calculation of |S|. Some examples should illustrate the point.

Example 3.2: More words

Consider the experiment of Example 3.1 and suppose that all outcomes are equally likely
– there is no bias in the selection of letters from the bag. Find the probability of the
following events:
3.3 Ordered Sampling without Replacement 26

A: “A randomly-chosen word contains no vowels”;

B: “A randomly-chosen word begins and ends with the same letter”.

Solution:
If the word contains no vowels than it must be made up solely of the letters B, C, and D.
Hence there are three choices for the first letter, three choices for the second letter, and
three choices for the third letter. It follows that
|A| 3×3×3 27
P(A) = = = .
|S| 125 125

If the word begins and ends with the same letter then there are five choices for the first
letter and five choices for the second letter but only one choice for the third because it
must be the same as the first. It follows that
|B| 5×5×1 25 1
P(B) = = = = .
|S| 125 125 5

Exercise 3.3: Bank PIN

When I open a bank account I am allocated a 4-digit personal identification number (which
may begin with one or more zeros) at random.

(a) What is the cardinality of the sample space for this experiment?

(b) By computing the cardinality of each of these events find the probability that:

(i) Every digit of my number is even;

(ii) My number is palindromic (reads the same forwards as backwards);
(iii) No digit of my number exceeds 7;
(iv) The largest digit in my number is exactly 7.

3.3 Ordered Sampling without Replacement

Example 3.4: Yet more words
Consider again the experiment of Example 3.1 but now suppose that repetition of letters
is not allowed – you can still think of taking three letters from the bag of five but now
without replacement. How many three-letter words exist in this case?

Solution:
There are still five choices for the first letter but now there are only four choices for the
second letter (only four letters are “left in the bag”) and only three choices for the third
letter (only three letters “left in the bag”). Hence there are 5 × 4 × 3 = 60 different words.

Again this example illustrates a general principle:

• If we make an ordered selection of r things from a set U = {u1 , u2 , . . . , un }

without replacement (i.e., repetition is not allowed, each element can be
3.3 Ordered Sampling without Replacement 27

selected only once) then the sample space is the set of all ordered r-tuples of
distinct elements of U . That is

S = {(s1 , s2 , . . . , sr ) : si ∈ U with si 6= sj for all i 6= j}.

• To find the cardinality of S, notice that if |U | = n there are n choices for

s1 ; for each of these choices there are n − 1 choices for s2 ; for each of these
there are n − 2 choices for s3 , and so on. Hence,

|S| = n × (n − 1) × (n − 2) × · · · × (n − r + 1)
n × (n − 1) × (n − 2) × · · · × (n − r + 1) × (n − r) × (n − r − 1) × · · · × 2 × 1
=
(n − r) × (n − r − 1) × · · · × 2 × 1
n!
= (3.2)
(n − r)!

where k! = k × (k − 1) × · · · × 2 × 1 (known as “k factorial”) and we have

the convention 0! = 1.
Example 3.5: Letter permutations
How many permutations (rearrangements) of the letters A, B, C, D, E exist? In other
words, how many five-letter “anagrams” can we make from these letters without repeti-
tion?
Solution:
Considering how many choices there are for each letter, the number of permutations is
5 × 4 × 3 × 2 × 1 = 5! = 120.

In general, a permutation is an ordered sample of n things without replacement

from the set U = {u1 , u2 , . . . , un } of n things. Hence, we can find the number of
permutations using (3.2) with r = n: there are n!/0! = n! of them. This will be
an important result when we move on to unordered sampling in the next section.
Sometimes we’re interested in events with no repetition as a subset of a sample
space where repetition is allowed. In this case, we need to combine the methods
of this section and the last.
Example 3.6: Complicated words
Suppose we do the experiment of Examples 3.1 and 3.2, i.e., we sample letters with re-
placement. What is the probability that a randomly-chosen word has no repeated letters?
Solution:
The sample space of all possible words has |S| = 125 (Example 3.1). Let C be the event
that a word has no repeated letters. Then |C| = 60 (Example 3.4) so

|C| 60 12
P(C) = = = .
|S| 125 25
3.4 Unordered Sampling without Replacement 28

Exercise 3.7: *Bank PIN (revisited)

Consider the bank account example of Exercise 3.3. Find the probability that:

(a) The number has at least one repeated digit;

(b) The digits in the number are in strictly increasing order.

3.4 Unordered Sampling without Replacement

Example 3.8: Letters again
How many ways are there to choose three letters from five without replacement if the order
doesn’t matter? [You can think of drawing letters from a bag in a Scrabble-type game
where the important thing is just what letters you get, not what order you get them.]

Solution:
We are already know that there are 5!/2! = 60 ways to make an ordered selection without
replacement (see Example 3.4) but that is overcounting because now we want, e.g., ECB,
EBC, BEC, BCE, CBE, CEB to all count as the same outcome. For each choice of three
letters there are 3!=6 permutations (see Example 3.5) so the overcounting is by a factor
of six and there are 60/6 = 10 ways to make an unordered selection without replacement.

Once again, the above example illustrates a general idea:

• If we make an unordered selection of r things from a set U = {u1 , u2 , . . . , un }

without replacement then we obtain a subset of U of size r with, by definition,
distinct elements.

• The corresponding sample space is the set of all subsets of r elements of U :

S = {A ⊆ U : |A| = r}.

• An ordered sample is obtained by taking an element of this sample space

S and putting its elements in order. Each element of the sample space can
be ordered in r! ways and so, if |U | = n, then (using the formula (3.2) for
ordered selections without replacement) we must have that

n!
r! × |S| = ,
(n − r)!

and so
n!
|S| = . (3.3)
(n − r)!r!
3.4 Unordered Sampling without Replacement 29

Remark: We normally use the notation

 
n n!
= .
r (n − r)!r!

Here nr (read as “n choose r”) is called a binomial coefficient. By convention

n


r = 0 when r > n, which makes sense since we can’t choose more than r things
from n without replacement. Binomial coefficients appear in many other places,
for example, you may have encountered them in the binomial theorem:
         
n n n 0 n n−1 1 n n−2 2 n 1 n−1 n 0 n
(a + b) = a b + a b + a b + ··· + a b + a b
0 1 2 n−1 n
n  
X n k n−k
= a b .
k
k=0

Exercise 3.9: **Binomial theorem

Verify the binomial theorem formula for n = 2 and n = 3. Then prove it for general n.
[Hint: Use induction.]

Example 3.10: Course representation

How many ways are there to select four course representatives with two students from the
197 studying MTH4107, and two students from the 202 studying MTH4207?

Solution:
There are 197 202



2 ways to choose the MTH4107 students and 2 ways to choose the
MTH4207 students. Hence, by the basic principle of counting,
   
197 202
Number of selections = ×
2 2
197! 202!
= ×
195! 2! 200! 2!
197 × 196 202 × 201
= ×
2 2
= 197 × 98 × 101 × 201
= 391931106.

Exercise 3.11: Lotto

An entry into the UK lottery consists of a list of six numbers. During the draw, six
numbered balls are chosen at random and if a player matches all six (regardless of order)
they win the jackpot.
(a) Until 2015, the balls were numbered 1 to 49 (inclusive). What was the probability of
a particular entry winning?
(b) From 2015, the balls have been numbered 1 to 59 (inclusive). What is the probability
of winning now?
3.5 Sampling in Practice 30

(c) *What is the probability of matching five balls out of six in the current set-up? Can
you generalize to the probability of matching n balls?

3.5 Sampling in Practice

In the preceding sections we have shown the following.

Theorem 3.1. The number of ways of selecting (sampling) r objects from an

n-element set is

(a) Ordered with replacement (repetition allowed): nr ,

n!
(b) Ordered without replacement (no repetition): (n−r)! ,

n


(c) Unordered without replacement (no repetition): r .

Remarks:

• It is important when answering questions involving sampling that you read

the question carefully and decide what sort of sampling is involved. Specif-
ically, how many things are you selecting, what set are you selecting from,
does the order matter, and is repetition allowed or not?

• Sometimes we can consider an experiment as either ordered or unordered

sampling. In particular, if we are interested in an event where order doesn’t
matter, it is generally possible to consider a sample space of ordered out-
comes and then count how many of these ordered outcomes are in our event.
The important thing is to be consistent, i.e., to consider the same type of
outcomes when determining the cardinality of both sample space and event.

• We have not covered the case of unordered sampling with replacement (i.e.,
repetition allowed). This is rather more difficult to deal with but sometimes
the previous point provides a workaround.

Some examples should serve as further illustration.

Example 3.12: Coins

Suppose we have ten coins, seven gold ones and three copper ones, and we pick four of
these coins at random (i.e., such that all outcomes are equally likely). Let D be the event
that we pick four gold coins. Determine P(D) using both ordered and unordered sampling.

Solution:
Since we pick coins at random, P(D) = |D|/|S|. To calculate the probability we there-
fore need to determine the size of the sample space and the size of the event. The
3.5 Sampling in Practice 31

set U of objects contains seven different gold coins and three different copper coins,
U = {c1 , c2 , c3 , g1 , g2 , . . . , g6 , g7 }. It is important to note that the coins are all differ-
ent objects even if they may share the same colour – by definition the elements of a set
must be distinct.
Let us first consider the experiment as ordered sampling without replacement. In this
case, the outcomes are an ordered selection of four objects, i.e., a 4-tuple (s1 , s2 , s3 , s4 )
with sk ∈ U and no repetition (i.e., si 6= sj for i 6= j). Using n = 10 and r = 4 in
Theorem 3.1(b), the size of the sample space is

10! 10!
|S| = = = 10 × 9 × 8 × 7 .
(10 − 4)! 6!

A particular outcome in D is an ordered sample without replacement of four things from

the set of seven gold coins, for instance, (g5 , g2 , g6 , g3 ). How many such outcomes are
there? Well, again using Theorem 3.1(b), but now with n = 7 and r = 4, we have

7! 7!
|D| = = = 7 × 6 × 5 × 4,
(7 − 4)! 3!

and hence
|D| 7×6×5×4 1
P(D) = = = .
|S| 10 × 9 × 8 × 7 6
Now let us consider the experiment as unordered sampling – imagine revealing all the
picked coins at once rather than one-by-one. The outcomes are now subsets of U with
cardinality four (i.e., {s1 , s2 , s3 , s4 } with sk ∈ U ) and to calculate the size of the sample
space we need to use Theorem 3.1(c), with n = 10 and r = 4:
 
10 10! 10 × 9 × 8 × 7
|S| = = = .
4 6! 4! 4×3×2×1

To be consistent, in this framework the outcomes in D must also be considered as un-

ordered: each outcome is a four-element subset of the set of seven gold coins, for instance,
{g1 , g3 , g6 , g7 }. The number of such outcomes is given by Theorem 3.1(c), now with n = 7
and r = 4:  
7 7! 7×6×5×4
|D| = = = .
4 3! 4! 4×3×2×1
Hence, finally we have,
7


|D| 4 7×6×5×4 1
P(D) = = 10

= = ,
|S| 4
10 × 9 × 8 × 7 6

which, of course, is the same result as obtained using ordered sampling. [In the fortunate
circumstance that you are able to do a question using two different methods, comparison
of the results provides a good way to check your answer!]

Exercise 3.13: Coins again

Consider again the set-up of Example 3.12 and determine the probability of the following
3.6 Further Exercises 32

events:

(a) The event, E, that we pick two gold coins followed by two copper coins;

(b) The event, F , that we pick two gold and two copper coins in any order.

In each case, which of ordered and unordered sampling can be used?

Example 3.14: Poker dice

Suppose we roll five fair dice. What is the probability to roll “a pair” (i.e., for two of the
dice to show the same number while the other three are all different)?

Solution:
We are sampling r = 5 objects from the set U = {1, 2, 3, 4, 5, 6} of size n = 6. The exper-
iment allows repetition so we have to consider ordered sampling with replacement. [Re-
member that we haven’t really covered how to treat unordered sampling with replacement!]
Outcomes are thus 5-tuples and the size of the sample space is given by Theorem 3.1(a):

|S| = 65 .

Now let G be the event “we roll a pair”. We consider first those outcomes in the event
G which have the pair as the first two entries, i.e., outcomes of the form (p, p, r1 , r2 , r3 ).
There are obviously six choices for p; for each of those there are five choices for r1 ; for
each of those there are four choices for r2 ; for each of those there are three choices for
r3 . Hence by the basic principle of counting, there are 6 × 5 × 4 × 3 such outcomes.
However, the pair doesn’t have to appear as the first two entries of our 5-tuple; we need to
take other arrangements into account, e.g., (p, r1 , p, r2 , r3 ), (r1 , r2 , r3 , p, p). Since we are
effectively choosing two (distinct) dice from five to be the pair, there are 52 = 10 different

“patterns”, each corresponding to 6 × 5 × 4 × 3 outcomes. Hence, putting everything

together, we have
|G| = 10 × 6 × 5 × 4 × 3 .

and
|G| 10 × 6 × 5 × 4 × 3 25
P(G) = = = .
|S| 65 54

3.6 Further Exercises

Exercise 3.15: Real-life sampling
Think of a real-life situation involving sampling and discuss which of the methods from
this chapter would be appropriate to analyse it.

Exercise 3.16: More course representation

Look back at Example 3.10. Now suppose that four course representatives are chosen
at random from the 399 students. What is the probability that two are chosen from
MTH4107 and two from MTH4207? [Hint: Look at Exercise 3.13(b).]
3.6 Further Exercises 33

Exercise 3.17: Cricket squad

Each member of a squad of 18 cricketers is either a batsmen or a bowler. The squad
comprises 10 batsmen and 8 bowlers. An eccentric coach chooses a team by picking a
random set of 11 players from the squad.

(a) What is the probability that the team is made up of six batsmen and five bowlers?

(b) What is the probability that the team contains fewer than three bowlers?

Exercise 3.18: *Binomial identities

Let 1 ≤ r ≤ n. A subset of {1, 2, . . . , n} of cardinality r is chosen at random.

(a) Calculate the probability that 1 is an element of the chosen subset.

(b) Without using your answer to part (a), calculate the probability that 1 is not an
element of the chosen subset.

(c) Deduce that      

n n−1 n−1
= + .
r r r−1
Chapter 4

Conditional Probability

4.1 Introducing Extra Information

Additional information (a so-called “condition”) may change the probability as-
cribed to an event. This is important, for example, in medical testing/care and in
pricing life insurance.

Exercise 4.1: The power of knowledge

Think of some more real-life examples where extra information would change the proba-
bility given to an event.

Example 4.2: Information about a die

Consider rolling a fair six-sided die once and recording the number showing.

(a) Determine the probabilities of the following two events:

A: “The number shown is odd”;
B: “The number shown is smaller than four”.
(b) Now suppose somebody tells us that the number shown is odd, i.e., we know that
event A has happened (we say “event A is given”). What is now the probability of
event B (given A)?

Solution:

(a) Just as in Example 1.1 we can simply write the sample space as S = {1, 2, 3, 4, 5, 6}
and we have A = {1, 3, 5} and B = {1, 2, 3}. Hence, since outcomes are equally likely,

|A| 3 1
P(A) = = = ,
|S| 6 2
|B| 3 1
P(B) = = = .
|S| 6 2

34
4.1 Introducing Extra Information 35

(b) Again using the fact that outcomes are equally likely, we have

|{1, 3}| 2
P(B given event A) = = .
|{1, 3, 5}| 3

Notice that we can write

|{1, 3}|
P(B givenA) =
|{1, 3, 5}|
|{1, 3}|/|S|
=
|{1, 3, 5}|/|S|
|A ∩ B|/|S|
=
|A|/|S|
P(A ∩ B)
=
P(A)

This last expression gives a general definition of conditional probability (the proba-
bility of the event B conditioned on event A having occurring), which holds beyond
the special case of equally-likely outcomes.

Definition 4.1. If E1 and E2 are events and P(E1 ) 6= 0 then the conditional
probability of E2 given E1 , usually denoted by P(E2 |E1 ), is

P(E1 ∩ E2 )
P(E2 |E1 ) = .
P(E1 )

Remarks:

• The notation P(E2 |E1 ) is rather unfortunate since E2 |E1 is not an event.
Do not confuse the conditional probability P(E2 |E1 ) with P(E2 \ E1 ), the
probability for the event “E2 and not E1 ”.

• Note that the definition does not require that E2 happens after E1 , only
that we know about E1 but not about E2 . One way of thinking of this is
to imagine that the experiment is performed secretly and the fact that E1
occurred is revealed to you (without the full outcome being revealed). The
conditional probability of E2 given E1 is the new probability of E2 in these
circumstances.

Exercise 4.3: Coloured pens

You have three pens coloured blue, red, and green. You pick one at random and then you
pick another without replacement.

(a) Find the conditional probability that the second pen is blue, given the first pen is red.

(b) Find the conditional probability that the first pen is red, given the second pen is blue.
4.2 Implications of Extra Information 36

4.2 Implications of Extra Information

In the last section we introduced the idea of extra information; we here explore
further its implications. Note, in particular, that conditional probability can be
used to measure how the occurrence of some event influences the chance of another
event occurring:

• If P(E2 |E1 ) < P(E2 ) then E1 occurring makes E2 less probable;

• If P(E2 |E1 ) > P(E2 ) then E1 occurring makes E2 more probable;

• If P(E2 |E1 ) = P(E2 ) then the event E1 has no impact on the probability of
event E2 and we say the events are independent (see Chapter 5).

We now illustrate this with a long example.

Example 4.4: More information about a die

Consider rolling a fair six-sided die twice.

(a) Determine the probability of the following events:

A: “A ‘six’ occurs on the first roll”;
B: “A double is rolled”;
C: “At least one odd number is rolled”.
(b) Now find the conditional probabilities for the following:

(i) Rolling a double assuming the first roll is a six.

(ii) Rolling a six first assuming a double is rolled.
(iii) Rolling at least one odd number assuming a double is rolled.
(iv) Rolling a double assuming at least one odd number is rolled.

Solution:

(a) This situation can be treated as ordered sampling with replacement; we write the
outcomes as ordered pairs and |S| = 36 (see Exercise 1.3). Since outcomes are equally
likely we can simply count the number of outcomes in each event to find

|A| 6 1
P(A) = = = ,
|S| 36 6
|B| 6 1
P(B) = = = ,
|S| 36 6
|C| 27 3
P(C) = = = .
|S| 36 4

[If the last of these is not obvious, use the fact that there are 3 × 3 ways to get two
even numbers, together with Proposition 2.2.]
4.2 Implications of Extra Information 37

(b) (i) We have A ∩ B = {(6, 6)} so

|A ∩ B| 1
P(A ∩ B) = =
|S| 36

and hence
P(A ∩ B) 1/36 1
P(B|A) = = = .
P(A) 1/6 6
This fits with the intuition that rolling a six first does not change the probability
of a double.
(ii) Now we have
P(A ∩ B) 1/36 1
P(A|B) = = = .
P(B) 1/6 6
It is perhaps slightly less obvious that rolling a double does not change the
probability of getting a six on the first roll. However, remember that conditional
probability does not require the condition to happen “first”; indeed, here event
B happens “after” event A has happened.
(iii) Here B ∩ C = {(1, 1), (3, 3), (5, 5)} so

|B ∩ C| 3 1
P(B ∩ C) = = =
|S| 36 12

and hence
P(B ∩ C) 1/12 1
P(C|B) = = = .
P(B) 1/6 2
So rolling a double reduces the probability of having at least one odd number.
(iv) Now we have
P(B ∩ C) 1/12 1
P(B|C) = = = .
P(C) 3/4 9
Similarly, rolling at least one odd number reduces the probability of a double –
this is probably not obvious at all!

Notice from the last example that for events E1 and E2 , the probability of E1
given E2 does not have to be equal to the probability of E2 given E1 ; in general
these are different probabilities which is related to the discussion of Exercise 0.3.

Exercise 4.5: Conditional deductions

Let E1 and E2 be events with P(E1 ) > 0 and P(E2 ) > 0.

(a) Prove that if P(E1 |E2 ) > P(E1 ) then P(E2 |E1 ) > P(E2 ).

(b) Prove that P(E1c |E2 ) = 1 − P(E1 |E2 ).

(c) If P(E1 |E2 ) is known, what can be deduced about P(E1 |E2c )?
4.3 The Multiplication Rule 38

4.3 The Multiplication Rule

We have already seen how we can calculate the conditional probability P (E2 |E1 )
from knowledge of P(E1 ∩ E2 ). However, by straightforwardly rearranging Defini-
tion 4.1, one can write

P(E1 ∩ E2 ) = P(E1 )P(E2 |E1 ), (4.1)

which is a useful formula for calculating P(E1 ∩ E2 ) from knowledge of the con-
ditional probability P (E2 |E1 ). In fact, one can generalize this to an arbitrary
number of events as expressed in the following theorem.

Theorem 4.2. Let E1 , E2 , . . . , En be events, then

P(E1 ∩E2 ∩· · ·∩En ) = P(E1 )×P(E2 |E1 )×P(E3 |E1 ∩E2 )×· · ·×P(En |E1 ∩E2 ∩· · ·∩En−1 )

provided that all of the conditional probabilities involved are defined.

Proof:
For a given number of events, one can easily prove the theorem statement by plug-
ging in the definition of conditional probabilities and cancelling common factors
in the numerator and denominator. To prove it in general we will use induction.
As the base case, let us take n = 2. This is precisely what we already have
in (4.1) from direct rearrangement of Definition 4.1. Now, for the inductive step,
we assume we have shown that the statement of the theorem holds for n = k, i.e.,

P(E1 ∩E2 ∩· · ·∩Ek ) = P(E1 )×P(E2 |E1 )×P(E3 |E1 ∩E2 )×· · ·×P(Ek |E1 ∩E2 ∩· · ·∩Ek−1 )
(4.2)
and seek to prove the statement for n = k + 1. First note that we can write
E1 ∩ E2 ∩ · · · ∩ Ek+1 as F1 ∩ F2 with F1 = E1 ∩ E2 ∩ · · · ∩ Ek and F2 = Ek+1 . From
Definition 4.1, we have P(F1 ∩ F2 ) = P(F1 )P(F2 |F1 ), i.e.,

P(E1 ∩ E2 ∩ · · · ∩ Ek+1 ) = P(E1 ∩ E2 ∩ · · · ∩ Ek ) × P(Ek+1 |E1 ∩ E2 ∩ · · · ∩ Ek ). (4.3)

Substituting (4.2) into (4.3) yields

P(E1 ∩ E2 ∩ · · · ∩ Ek+1 ) = P(E1 ) × P(E2 |E1 ) × P(E3 |E1 ∩ E2 ) × · · ·

× P(Ek |E1 ∩ E2 ∩ · · · ∩ Ek−1 ) × P(Ek+1 |E1 ∩ E2 ∩ · · · ∩ Ek )

which means the statement of the theorem also holds for n = k + 1 and hence, by
the principle of induction, for all n ≥ 2.
4.4 Ordered Sampling Revisited 39

Exercise 4.6: Coins (revisited)

Reconsider the event D in Example 3.12 (picking four gold coins from a set of seven gold
and three copper coins). Demonstrate that one can also work out P(D) using conditional
probability.

4.4 Ordered Sampling Revisited

The exercise at the end of the previous section already gave a hint that conditional
probability provides an alternative approach to questions involving ordered sam-
pling. We now show how this approach can be used to confirm our earlier results
for the cardinality of the sample space in the cases of ordered sampling with and
without replacement.
Consider again that we pick at random r things in order from a set U of size
n = |U |. Let us denote a fixed outcome of this experiment by (v1 , v2 , . . . , vr ), with
vk ∈ U , and the corresponding event that this particular outcome occurs by A,
i.e., A = {(v1 , v2 , . . . , vr )}. Notice that A is a simple event so, since all outcomes
are equally likely, we have
1
P(A) = .
|S|
Hence, if we can calculate P(A) by conditional probability, it will give us an ex-
pression for |S|.
Let Ek denote the event that “the kth pick gives vk ” (with vk ∈ U ). The
event A can then be written as A = E1 ∩ E2 ∩ · · · ∩ Er and, by Theorem 4.2, the
probability of this event is given as

P(E1 ∩E2 ∩· · ·∩Er ) = P(E1 )×P(E2 |E1 )×P(E3 |E1 ∩E2 )×· · ·×P(Er |E1 ∩E2 ∩· · ·∩Er−1 ).

Note that this expression is valid whether we sample with or without replacement
but the form of the conditional probabilities is different in each case.

Example 4.7: Ordered sampling without replacement (revisited)

Suppose we sample without replacement and v1 , v2 , . . . , vr are distinct. Show that P(A) =
(n − r)!/n!.

Solution:
In this case,
1
P(E1 ) =,
n
as we pick the element v1 at random from the set U of size |U | = n. Similarly,

1
P(E2 |E1 ) = ,
n−1
4.5 Further Exercises 40

as we pick the element v2 at random from the set U \ {v1 } of size n − 1. In general,

1
P(Ei |E1 ∩ E2 ∩ · · · ∩ Ei−1 ) = ,
n−i+1

as we pick the element vi at random from the set U \ {v1 , v2 , . . . , vi−1 } of size n − i + 1.
Hence

P(A) = P(E1 ∩ E2 ∩ · · · ∩ Er )
1 1 1
= × × ··· ×
n n−1 n−r+1
(n − r)!
= .
n!

Exercise 4.8: Ordered sampling with replacement (revisited)

Suppose we sample with replacement. Show that P(A) = 1/nr .

Unsurprisingly, the results for P(A) in Example 4.7 and Exercise 4.8 agree
with 1/|S| when |S| is calculated from Theorem 3.1(b) and 3.1(a) respectively.
However, you may find the equally-likely assumption more transparent with the
method of this chapter.

4.5 Further Exercises

Exercise 4.9: More die rolling
A standard fair die is rolled twice.

(a) Find the probability that the sum of the two rolls is at least nine.

(b) Find the conditional probability that the first roll is a “four” given that the sum of
the two rolls is at least nine.

(c) Find the conditional probability that the first roll is not a “four” given that the sum
of the two rolls is at least nine.

(d) Find the conditional probability that the first roll is a “four” given that the sum of
the two rolls is less than nine.

(e) Find the conditional probability that the sum of the two rolls is at least nine given
that the first roll is a “four”.

Exercise 4.10: Travel difficulties

When you travel into university you notice whether your train is late and by how much
and also whether you are able to get a seat on it. Let A be the event “the train is not
late”, B be the event “the train is late but by not more than 15 minutes”, and C be the
event “you are able to get a seat”. Suppose that P(A) = 1/2, P(B) = 1/4, P(C) = 1/3
and P(A ∩ C) = 1/4.
4.5 Further Exercises 41

(a) Show that the conditional probability that the train is more than 15 minutes late
given that the train is late is equal to 1/2.

(b) Show that the conditional probability that you get a seat given that the train is late
is equal to 1/6.

Exercise 4.11: Medical testing

Two treatments for a disease are tested on a group of 390 patients. Treatment A is given
to 160 patients of whom 100 are men and 60 are women; 20 of these men and 40 of these
women recover. Treatment B is given to 230 patients of whom 210 are men and 20 are
women; 50 of these men and 15 of these women recover.

(a) For which of A and B is there a higher probability that a patient chosen randomly from
among those given that treatment recovers? Express this as an inequality between
two conditional probabilities.

(b) For which of A and B is there a higher probability that a man chosen randomly from
among those given that treatment recovers? Express this as an inequality between
two conditional probabilities.

(c) For which of A and B is there a higher probability that a woman chosen randomly from
among those given that treatment recovers? Express this as an inequality between
two conditional probabilities.

(d) Compare the inequality in part (a) with the inequalities in part (b) and (c). Are you
surprised by the result?
Chapter 5

Independence

5.1 Independence for Two Events – Basic Definition

The examples we have seen so far tell us that the probability P(E1 ∩ E2 ) being the
product P(E1 ) × P(E2 ) is a very special case.

Exercise 5.1: Finding the unusual

Look back through all the previous examples in the notes and find cases where the proba-
bility of the intersection of two events is the product of their individual probabilities, i.e.,
P(E1 ∩ E2 ) = P(E1 )P(E2 ). Can you identify what these situations have in common?

Example 5.2: Yet more die rolling

Consider again rolling a fair six-sided die twice. Let A denote the event “the first roll
shows an even number” and B denote the event “the number shown on the second roll is
larger than four”. Determine the probabilities P(A), P(B), P(A∩B), P(A|B), and P(B|A).

Solution:
Just as in Example 4.4, we can treat this situation as ordered sampling with replacement
and write the outcomes as ordered pairs. We obviously have |S| = 36 and, counting the
number of possibilities for each die roll we have |A| = 3 × 6 = 18 and |B| = 6 × 2 = 12.
Hence, since all outcomes are equally likely and (2.5) applies,

18 1 12 1
P(A) = = and P(B) = = .
36 2 36 3
For the event A∩B (“the first roll is even and the second roll is larger than four”), there are
obviously three possibilities for the first roll and two for the second so |A ∩ B| = 3 × 2 = 6
and
6 1
P(A ∩ B) = = .
36 6
Notice here that
1 1 1
P(A ∩ B) = = × = P(A)P(B).
6 2 3

42
5.2 Independence for Two Events – More Details 43

As you may already know, events with such a property are said to be independent. We
also have that
P(A ∩ B) 1/6 1
P(A|B) = = = ,
P(B) 1/3 2
and
P(A ∩ B) 1/6 1
P(B|A) = = = .
P(A) 1/2 3
Notice here that P(A|B) = P(A) and P(B|A) = P(B), i.e. the conditional probabilities do
not depend on the condition. We will further explore the connection between conditional
probabilities and independence in the next section.

Definition 5.1. We say that the events E1 and E2 are (pairwise) independent
if
P(E1 ∩ E2 ) = P(E1 )P(E2 ).

[If this equation does not hold we say that the events are dependent.]

Remarks:

• Be careful not to assume independence without good reason. You may as-
sume that two events E1 and E2 are independent in the following situations:

(i) They are clearly physically unrelated (e.g., they are associated with
different tosses of a coin);
(ii) You calculate their probabilities and find that P(E1 ∩E2 ) = P(E1 )P(E2 )
(i.e., you check Definition 5.1);
(iii) A question tells you that they are independent!

• Independence is not equivalent to being physically unrelated. Physically

unrelated events are always independent [see (i) above] but, as we shall see,
physically related events may or may not be independent.
Exercise 5.3: Different buses
Suppose there are two buses, the Alphabus and the Betabus, running from station P to
station Q along two different routes. Consider the events
A: “The Alphabus is running”;
B: “The Betabus is running”.
Assuming that these events are independent and have probabilities P(A) = 9/10 and
P(B) = 4/5, determine the probability that one can travel from P to Q by bus.

5.2 Independence for Two Events – More Details

As Example 5.2 already suggested, there is a connection between independence
and conditional probability.
5.2 Independence for Two Events – More Details 44

Theorem 5.2. Let E1 and E2 be events with P(E1 ) > 0 and P(E2 ) > 0. The
following are equivalent:
(a) E1 and E2 are independent,

(b) P(E1 |E2 ) = P(E1 ),

(c) P(E2 |E1 ) = P(E2 ).

Loosely speaking, this says that if E1 and E2 are independent then telling you that
E1 occurred does not change the probability that E2 occurred, and vice versa.
Proof:
It is sufficient to show that (a) implies (b), (b) implies (c), and (c) implies (a).
• (a) implies (b): We start by assuming that (a) is true, i.e., we suppose E1
and E2 are independent so, from Definition 5.1,

P(E1 ∩ E2 ) = P(E1 )P(E2 ).

Since P(E2 ) 6= 0, we can use Definition 4.1 to show

P(E1 ∩ E2 ) P(E1 )P(E2 )

P(E1 |E2 ) = = = P(E1 ).
P(E2 ) P(E2 )

Hence (b) is also true.

• (b) implies (c): We start by assuming (b) is true, i.e., P(E1 |E2 ) = P(E1 ).
Then, by Definition 4.1,

P(E1 ∩ E2 )
= P(E1 ).
P(E2 )

Since P(E1 ) 6= 0 (and E1 ∩ E2 = E2 ∩ E1 ), it follows that

P(E2 ∩ E1 )
= P(E2 )
P(E1 )

which means, again by Definition 4.1, P(E2 |E1 ) = P(E2 ), i.e., (c) is also true.

• (c) implies (a): We start by assuming (c) is true, i.e., P(E2 |E1 ) = P(E2 ).
Then, once again by Definition 4.1,

P(E2 ∩ E1 )
= P(E2 )
P(E1 )

which implies P(E2 ∩E1 ) = P(E1 )P(E2 ), i.e., E1 and E2 satisfy Definition 5.1
and are independent. Hence (a) is also true and the proof is complete.
5.3 Independence for Three or More Events 45

Independence can also be disguised in other ways as illustrated by the following

example.

Example 5.4: Another implication

Prove that the equality P(E1 ∪ E2 ) = P(E1 )P(E2c ) + P(E2 ) implies that E1 and E2 are
independent.

Solution:
We start from
P(E1 ∪ E2 ) = P(E1 )P(E2c ) + P(E2 )

and use the inclusion-exclusion formula (Proposition 2.7) on the left-hand side and the
obvious Proposition 2.2 on the right-hand side to obtain:

P(E1 ) + P(E2 ) − P(E1 ∩ E2 ) = P(E1 )[1 − P(E2 )] + P(E2 ).

Cancelling P(E1 ) + P(E2 ) on the two sides, this is equivalent to

−P(E1 ∩ E2 ) = −P(E1 )P(E2 )

which is trivially rearranged to the usual condition for independence:

P(E1 ∩ E2 ) = P(E1 )P(E2 ).

Hence the original equality implies that E1 and E2 are independent.

Exercise 5.5: More information about a die (revisited)

Look back at Example 4.4. Are the events A and B defined in that example independent?
Are B and C? Are A and C? [You may use any of the equivalent results from this section.]

5.3 Independence for Three or More Events

Exercise 5.5 may prompt you to ask whether it is actually possible to find three
events which are pairwise independent. You may also wonder whether just looking
at pairs is enough to define independence in the case of more than two events. In
this section we address such questions.

Example 5.6: Even more boring dice rolling

Consider (yet again) again rolling a fair six-sided die twice. Now look at the following
events:
D: “The first roll shows an odd number”;
E: “The second roll shows an odd number”;
F : “The sum of the two rolls is an odd number”.
Determine whether these events are pairwise independent.
5.3 Independence for Three or More Events 46

Solution:
Obviously |S| = 36 and |D| = |E| = 18 so P(D) = P(E) = 1/2 (cf. Example 5.2). The
cardinality of F is slightly less obvious but note that if the sum is odd, one roll must be odd
and the other even; there are nine ordered pairs which are odd followed by even and nine
ordered pairs which are even followed by odd. Hence, we also have P(F ) = 18/36 = 1/2.
Turning our attention to independence, we now consider each pair of events in turn.

• D and E relate to different die rolls so we can argue that they are physically unre-
lated. Hence D and E are independent. This is easily confirmed since

|D ∩ E| 9 1 1 1
P(D ∩ E) = = = = × = P(D)P(E).
|S| 36 4 2 2

• The event D ∩ F contains all pairs which are odd followed by even so

|D ∩ F | 9 1 1 1
P(D ∩ F ) = = = = × = P(D)P(F ).
|S| 36 4 2 2

Hence D and F are independent.

• Similarly, the event E ∩ F contains all pairs which are even followed by odd so

|E ∩ F | 9 1 1 1
P(E ∩ F ) = = = = × = P(E)P(F ).
|S| 36 4 2 2

Hence E and F are independent.

Each pair of events is independent so we say that D, E, and F are pairwise independent.
However, notice that the event D ∩ E ∩ F is impossible since when both outcomes are odd
the sum is even. Hence
1 1 1
P(D ∩ E ∩ F ) = P(∅) = 0 6= × × = P(D)P(E)P(F ).
2 2 2

In fact, for three or more events the notion of independence is slightly more
subtle than for two events. For example, for three events we have the following
definition.

Definition 5.3. Three events E1 , E2 , and E3 are called pairwise independent

if

P(E1 ∩ E2 ) = P(E1 )P(E2 ),

P(E1 ∩ E3 ) = P(E1 )P(E3 ),
P(E2 ∩ E3 ) = P(E2 )P(E3 ).

The three events are called mutually independent if in addition

P(E1 ∩ E2 ∩ E3 ) = P(E1 )P(E2 )P(E3 ).

5.4 Conditional Independence 47

Armed with this definition, we see that although the events in Example 5.6 are
pairwise independent they are not mutually independent.
Definition 5.3 can be generalized to four, five, and more events. The formal
definition looks awkward but the basic idea is that, for mutual independence, the
probability of the intersection of any finite subset of the events should factorize
into the probabilities of the individual events.

Definition 5.4. We say that the events E1 , E2 , . . . , En are mutually indepen-

dent if for any 2 ≤ t ≤ n and 1 ≤ i1 < i2 < · · · < it ≤ n we have

P(Ei1 ∩ Ei2 ∩ · · · ∩ Eit ) = P(Ei1 ) × P(Ei2 ) × · · · × P(Eit ).

Remark: If the term “independent” is used without qualification for three or

more events, it generally means mutually independent. However, to avoid ambigu-
ity we shall try to be careful in the use of “pairwise independent” and “mutually
independent” as appropriate.
Exercise 5.7: Tossing thrice again
Suppose you toss a fair coin three times and record the sequence of Heads/Tails, as in
Example 1.2. Now consider the following events:
A: “The first and the second toss show the same result.”;
B: “The first and the last toss show different results.”;
C: “The first toss is a Tail.”.
Determine whether these three events are pairwise independent and whether they are
mutually independent.

5.4 Conditional Independence

It is also possible to consider independence of two events given we know that a
third event happens. This more advanced concept is captured in the following
definition.

Definition 5.5. Two events E1 and E2 are said to be conditionally indepen-

dent given an event E3 if

P(E1 ∩ E2 |E3 ) = P(E1 |E3 )P(E2 |E3 ).

Example 5.8: Magic coins

A magician has two coins: one is fair; the other has probability 3/4 of coming up Heads.
She picks a coin at random and tosses it twice. Consider the following events:
F: “The magician picks the fair coin.”;
H1 : “The first toss is a Head.”;
H2 : “The second toss is a Head.”.
5.4 Conditional Independence 48

(a) Are H1 and H2 conditionally independent given F ?

(b) Are H1 and H2 conditionally independent given F c ?

(c) Are H1 and H2 independent?

Solution:

(a) Assuming we pick the fair coin, the probability of a Head on each toss is 1/2, i.e.,

1 1
P(H1 |F ) = , and P(H2 |F ) = .
2 2
Furthermore, coin tosses of the same coin are considered to be independent (see the
discussion below Definition 5.1) so

1 1 1
P(H1 ∩ H2 |F ) = P(H1 |F )P(H2 |F ) = × = ,
2 2 4
i.e., H1 and H2 are conditionally independent given F .

(b) Assuming we pick the biased coin, the probability of a Head on each toss is 3/4, i.e.,

3 3
P(H1 |F c ) = , and P(H2 |F c ) = .
4 4
Again, coin tosses of the same coin are considered to be independent so, by construc-
tion of the experiment,

3 3 9
P(H1 ∩ H2 |F c ) = P(H1 |F c )P(H2 |F c ) = × = ,
4 4 16
i.e., H1 and H2 are conditionally independent given F c .

(c) To check Definition 5.1 and determine whether H1 and H2 are independent, we need
to know P(H1 ), P(H2 ), and P(H1 ∩ H2 ). To calculate the first of these note that1

P(H1 ) = P(H1 ∩ F ) + P(H1 ∩ F c )

= P(H1 |F )P(F ) + P(H1 |F c )P(F c ),

where in the first line we have employed Kolmogorov’s third axiom, Definition 2.1(c),
and in the second line we have used the multiplication rule, Theorem 4.2. Now,
since the coin is picked at random, we obviously have P(F ) = P(F C ) = 1/2. Hence,
substituting numbers, we find

1 1 3 1 5
P(H1 ) = × + × = .
2 2 4 2 8
1
This is a taster of a general method which will appear in the next chapter.
5.5 Further Exercises 49

Similarly, we have

P(H2 ) = P(H2 ∩ F ) + P(H2 ∩ F c )

= P(H2 |F )P(F ) + P(H2 |F c )P(F c )
1 1 3 1
= × + ×
2 2 4 2
5
= ,
8
and

P(H1 ∩ H2 ) = P(H1 ∩ H2 ∩ F ) + P(H1 ∩ H2 ∩ F c )

= P(H1 ∩ H2 |F )P(F ) + P(H1 ∩ H2 |F c )P(F c )
1 1 9 1
= × + ×
4 2 16 2
13
= .
32
Since
26 5 5
P(H1 ∩ H2 ) = 6= × = P(H1 )P(H2 ),
64 8 8
we finally conclude that the events H1 and H2 are not independent. This is intuitively
reasonable, if we see a Head on the first toss we can reason that the magician is more
likely to be using the biased coin, and this will affect the probability of a Head on the
second toss.

Exercise 5.9: Are you bored of dice yet?

You roll a fair six-sided die twice. Let A be the event that the first roll is odd, B be the
event that you roll at least one “six”, and C be the event that the sum of the rolls is seven.

(a) Are the events A and B independent?

(b) Are the events A, B, and C mutually independent?

(c) Are the events A and B conditionally independent given C?

Exercise 5.10: *Challenging events

(a) Find an example with two events, A and B, which are independent but not condi-
tionally independent with respect to another event C.

(b) Find an example with two events, D and E, which are conditionally independent with
respect to an event F but not with respect to F c .

5.5 Further Exercises

Exercise 5.11: Integer selection
A positive integer from the set {1, 2, 3, . . . , 36} is chosen at random with all choices equally
5.5 Further Exercises 50

likely. Let E be the event “the chosen number is even”, O be the event “the chosen number
is odd”, Q be the event “the chosen number is a perfect square”, and Dk be the event
“the chosen number is a multiple of k”. Carefully justify your answers to the following.

(a) Are the events E and O independent?

(b) Are the events E and Q independent?

(c) Are the events O and Q independent?

(d) Are the events D3 and D4 independent?

(e) Are the events D4 and D6 independent?

Exercise 5.12: Top card

The top card of a thoroughly shuffled standard deck of playing cards is turned over. Let
A be the event “the card is an ace”, R be the event “the card belongs to a red suit (♦
or ♥)”, and M be the event “the card belongs to a major suit (♥ or ♠)”. Show that the
events A, R, and M are mutually independent.

Exercise 5.13: Practice with proofs

Prove the following statements.

(a) *If A, B, and C are mutually independent events then A and B ∪ C are independent.

(b) If E and F are mutually exclusive events with positive probabilities, then P(E|F ) = 0.
Chapter 6

Total Probability and Bayes’ Theorem

6.1 Law of Total Probability

We saw in Example 5.8 that conditional probabilities can be used to compute the
“total” probability of an event. To state this more formally, we need the idea of a
partition which we illustrate first with another example.

Example 6.1: Tossing thrice (revisited)

A coin is tossed three times and the sequence of Heads/Tails is recorded just as in Exam-
ple 1.2 (and Exercise 5.7). Consider the three events:
E1 : “The first toss is a Head”;
E2 : “The first toss is a Tail and the second toss is a Head”;
E3 : “The first and second tosses are Tails”.
State these events in set notation and consider how they relate to the sample space.

Solution:
With the same (obvious) notation as in Example 1.2, each outcome is a list of Heads (h)
and Tails (t) in the order in which they are seen. We have

E1 = {htt, hht, hth, htt},

E2 = {thh, tht},
E3 = {tth, ttt}.

Notice that every outcome in the sample space appears in exactly one of these sets. The
three events are pairwise disjoint (Ei ∩ Ej = ∅ for i 6= j) and S = E1 ∪ E2 ∪ E3 . Loosely
speaking, the three events “split” the sample space into three parts; more formally we say
that E1 , E2 , and E3 partition the sample space.

Definition 6.1. The events E1 , E2 , . . . , En partition S if they are pairwise dis-

joint (i.e., Ek ∩ E` = ∅ if k 6= `) and E1 ∪ E2 ∪ · · · ∪ En = S. We can also say
that the set {E1 , E2 , . . . , En } is a partition of S.

51
6.1 Law of Total Probability 52

Remarks:

• Some books explicitly require E1 , E2 , . . . , En to be non-empty sets; we will

not insist on that here, although in practice it will usually be true.

• Understanding the definition of a partition is important in seeing how to cal-

culate the (total) probability of an event A from the conditional probabilities
P(A|Ek ) (i.e., the probabilities under certain constraints) and the so-called
marginal probabilities P(Ek ).

Theorem 6.2 (Law of total probability). Suppose that E1 , E2 , . . . , En partition S

with P(Ek ) > 0 for k = 1, 2, . . . , n. Then for any event A we have

P(A) = P(A|E1 )P(E1 ) + P(A|E2 )P(E2 ) + · · · + P(A|En )P(En )

n
X
= P(A|Ek )P(Ek ).
k=1

Proof:
Let Ak = A ∩ Ek , for k = 1, 2, . . . , n. Note that, by Definition 6.1, the sets
E1 , E2 , . . . , En are pairwise disjoint and E1 ∪ E2 ∪ · · · ∪ En = S. Since Ak ⊆ Ek
the events A1 , A2 , . . . , An are also pairwise disjoint and, furthermore,

A1 ∪ A2 ∪ · · · ∪ An = A ∩ (E1 ∪ E2 ∪ · · · ∪ En ) = A ∩ S = A.

Hence, by Definition 2.1(c),

P(A) = P(A1 ) + P(A2 ) + · · · + P(An ). (6.1)

Now, since P(Ek ) > 0, we also have (for k = 1, 2, . . . , n)

P(A ∩ Ek )
P(Ak ) = P(A ∩ Ek ) = P(Ek ) = P(A|Ek )P(Ek ). (6.2)
P(Ek )

Substituting (6.2) in (6.1) yields the statement of the theorem.

In fact, we have already seen an example of the use of Theorem 6.2 in Exam-
ple 5.8: by definition F and F c partition S. More generally, the approach is very
widely applicable but for different problems one needs to think carefully about
what partition to use. The technique is called conditioning.

Exercise 6.2: Mind the gap

In a recent survey [YouGov, 11th–16th June 2020], 1088 adults were asked (amongst other
questions) if they thought Watford counted as part of London. The following excerpt
6.2 Total Probability for Conditional Probabilities 53

from the results shows the number of survey participants in different age categories and
the percentage of them saying that they did consider Watford as part of London.

Age
18-24 25-49 50–64 65+
Number of participants 124 544 247 173
Percentage saying Watford is in London 31 34 15 19

What is the probability a randomly-chosen participant thinks Watford is in London?

6.2 Total Probability for Conditional Probabilities

There is an analogue of Theorem 6.2 for conditional probabilities.

Theorem 6.3. If E1 , E2 , . . . , En partition S with P(Ek ) > 0 for k = 1, 2, . . . , n,

then for events A and B with P(B ∩ Ek ) > 0 for k = 1, 2, . . . , n, we have

P(A|B) = P(A|B ∩ E1 )P(E1 |B) + P(A|B ∩ E2 )P(E2 |B) + · · · + P(A|B ∩ En )P(En |B)
n
X
= P(A|B ∩ Ek )P(Ek |B).
k=1

Proof:
The idea is to use the definition of conditional probability together with the result
we proved in the previous section. Specifically, we start from Definition 4.1

P(A ∩ B)
P(A|B) = ,
P(B)

and apply Theorem 6.2 to P(A ∩ B) to yield

P(A ∩ B|E1 )P(E1 ) + P(A ∩ B|E2 )P(E2 ) + · · · + P(A ∩ B|En )P(En )

P(A|B) = .
P(B)
(6.3)
Now, for k = 1, 2, . . . , n, we have

1 1 P(A ∩ B ∩ Ek )
P(A ∩ B|Ek )P(Ek ) = P(Ek ) [by Definition 4.1]
P(B) P(B) P (Ek )
1 P(B ∩ Ek )
= P(A ∩ B ∩ Ek ) [using P(B ∩ Ek ) > 0]
P(B) P(B ∩ Ek )
P(A ∩ B ∩ Ek ) P(B ∩ Ek )
=
P(B ∩ Ek ) P(B)
= P(A|B ∩ Ek )P(Ek |B) [by Definition 4.1]. (6.4)

Substituting (6.4) in (6.3) yields the statement of the theorem.

6.2 Total Probability for Conditional Probabilities 54

Example 6.3: Magic coins (revisited)

Consider again the set-up of the magician in Example 5.8.

(a) Supposing there is a Head on the first toss, determine the probability that the coin is
fair.

(b) Use the result from (a) together with Theorem 6.3 to find the probability of getting a
Head on the second toss given there is a Head on the first toss.

Solution:

(a) We already have P(F ) = 1/2, P(H1 |F ) = 1/2, and P(H1 ) = 5/8 (see Example 5.8).
We expect P(F |H1 ) to be different to P(H1 |F ); to calculate the former, we start with
the definition of conditional probability. Using the results we already know, we have1

P(F ∩ H1 )
P(F |H1 ) = [by Definition 4.1]
P(H1 )
P(H1 |F )P(F )
= [by Theorem 4.2]
P(H1 )
(1/2) × (1/2)
=
5/8
2
= .
5

(b) Using Theorem 6.3 with the partition {F, F c } gives

P(H2 |H1 ) = P(H2 |H1 ∩ F )P(F |H1 ) + P(H2 |H1 ∩ F c )P(F c |H1 ). (6.5)

We have P(F |H1 ) = 2/5 [from (a)] and P(F c |H) = 1 − P(F |H1 ) = 3/5 [see Exer-
cise 4.5]. The other two conditional probabilities on the right-hand side of (6.5) look
more complicated at first sight. However, the property of conditional independence
discussed in Example 5.8, leads to considerable simplification. Starting once again
with the definition of conditional probability, we find

P (H2 ∩ H1 ∩ F )
P(H2 |H1 ∩ F ) = [by Definition 4.1]
P(H1 ∩ F )
P(H2 ∩ H1 |F )P(F )
= [by Theorem 4.2]
P(H1 |F )P(F )
P(H2 ∩ H1 |F )
=
P(H1 |F )
P(H2 |F )P(H1 |F )
= [by conditional independence of H1 and H2 given F ]
P(H1 |F )
= P(H2 |F )
1
= .
2
1
We’ll see this method again in the next section.
6.3 Bayes’ Theorem 55

Similarly, by the conditional independence of H1 and H2 given F c , we have

3
P(H2 |H1 ∩ F c ) = P(H2 |F c ) = .
4
Hence, putting everything together, we conclude

1 2 3 3 13
P(H2 |H1 ) = × + × = .
2 5 4 5 20

Exercise 6.4: Magic coins (re-revisited)

Show that the answer to Example 6.3(b) is consistent with the analysis in Example 5.8(c).

6.3 Bayes’ Theorem

As Example 6.3 reminded us, P(A|B) and P(B|A) are different conditional prob-
abilities. However, as seen in that example, we can determine one from the other
if we also know P(A) and P(B). The theorem is attributed to Thomas Bayes
(1702–1761) although not actually published until after his death [Bay63].

Theorem 6.4 (Bayes’ theorem). If A and B are events with P(A), P(B) > 0, then

P(A|B)P(B)
P(B|A) = .
P(A)

Proof:
Starting again from Definition 4.1 (and using that P(A), P(B) > 0) we have

P(B ∩ A)
P(B|A) =
P(A)
P(A ∩ B) P(B)
=
P(A) P(B)
P(A ∩ B) P(B)
=
P(B) P(A)
P(B)
= P(A|B) .
P(A)

[Instead of multiplying numerator and denominator by P(B) in the second line,

one could use the multiplication rule (Theorem 4.2) for P(A ∩ B).]

Remarks:

• Bayes’ theorem has many practical applications.

• We often need to use Theorem 6.2 (law of total probability) to calculate the
probability in the denominator of Theorem 6.4.
6.3 Bayes’ Theorem 56

Example 6.5: Medical test

Suppose there is a disease which 0.1% of the population suffers from. A test for the disease
has a 99% chance of giving a positive result for someone with the disease, and only a 0.5%
change of giving a positive result for someone without the disease (a “false positive”).
What is the probability that a randomly-chosen person who tests positive actually has the
disease?

Solution:
Let us define the events:
D: “The selected person has the disease”;
P : “The test for the selected person is positive”.
We know
1 99 5
P(D) = , P(P |D) = , and P(P |Dc ) = .
1000 100 1000
We want to compute P(D|P ) so, using Bayes’ theorem (Theorem 6.4), we write

P(P |D)P(D)
P(D|P ) = .
P(P )

To calculate P(P ) we can use Theorem 6.2 with the partition {D, Dc }:

P(P ) = P(P |D)P(D) + P(P |Dc )P(Dc )

= P(P |D)P(D) + P(P |Dc )(1 − P(D)) [using Proposition 2.2]
 
99 1 5 1
= × + × 1−
100 1000 1000 1000
99 1 5 999
= × + × .
100 1000 1000 1000
Hence, we find

(99/100) × (1/1000)
P(D|P ) =
(99/100) × (1/1000) + (5/1000) × (999/1000)
990
=
5985
22
=
133
= 0.1654 (to 4 decimal places).

So assuming the test is positive, there is only about 17% chance that the person has the
disease. In other words, about 83% of positive tests are false positives. Does this mean
the test is useless or is there anything one can do about the problem?

Exercise 6.6: *Double medical test

Consider the medical testing scenario of Example 6.5. Supposing a person tests positive
in two separate tests, what is the probability that they actually have the disease? State
clearly any assumptions you make.
6.4 From Axioms to Applications 57

6.4 From Axioms to Applications

The law of total probability and Bayes’ theorem are crucially important: not only
are they needed for many exam questions but they have wide-reaching applications
in real-life. At this point in the course it is worth pausing to see how far we have
come. After introducing the language of sets and events, we started in what may
have seemed quite an abstract way with Kolmogorov’s axioms (Definition 2.1)
specifying the properties that probability should have. These simple axioms and
the definition of conditional probability (Definition 4.1) are the basic ingredients
which have allowed us to build up to proving the more complex results in the
present chapter. This illustrates both the beauty and the power of the axiomatic
approach to probability. We are also finally in a position to revisit a burning
question from Chapter 0...

Example 6.7: Innocent or guilty (revisited)

Look back at Exercise 0.3. Based on the evidence there, a prosecution lawyer argues that
there is a 1 in 50,000 chance of the suspect being innocent.

(a) Why is such an argument flawed?

(b) Suppose that London has a population of 10 million and the murderer is assumed to
be one of these. If there is no evidence against the suspect apart from the fingerprint
match then it is reasonable to regard the suspect as a randomly-chosen citizen. Under
this assumption, what is the probability the suspect is innocent?

(c) How does the argument change if one knows that the suspect is one of only 100 people
who had access to the building at the time Professor Damson was killed?

Solution:

(a) Let us write I for the event “the suspect is innocent” and F for the event “the
fingerprints of the suspect match those at the crime scene”. The prosecutor notes that
P(F |I) = 1/50000 and deduces that P(I|F ) = 1/50000. This is nonsense. In general
there is no reason why two conditional probabilities P(A|B) and P(B|A) should be
equal or even close to equal.

(b) By Bayes’ theorem combined with the law of total probability (using partition {I, I c }),

P(F |I)P(I) P(F |I)P(I)

P(I|F ) = = .
P(F ) P(F |I)P(I) + P(F |I c )P(I c )

Now, we are told P(F |I) and it is reasonable to assume that P(F |I c ) = 1 since if the
suspect is guilty the fingerprints should certainly match.2 The quantity P(I) in Bayes’
theorem is the probability that the suspect is innocent in the absence of any evidence
2
In fact, you could set P(F |I c ) to be any reasonably large probability without changing the
general conclusions.
6.5 Further Exercises 58

at all. We are told that the suspect should be regarded as a randomly-chosen citizen
from a city of 10 million people so P(I c ) = 1/10000000 and P(I) = 1 − 1/10000000 =
9999999/10000000. This gives

(1/50000) × (9999999/10000000)
P(I|F ) =
(1/50000) × (9999999/10000000) + 1 × (1/100000000)
9999999
=
9999999 + 50000
= 0.9950 (to 4 decimal places).

Hence there is about a 99.5% chance that the suspect is innocent.

(c) This new information will decrease our initial value of P(I) which, remember, is the
probability that our suspect is innocent before we consider the fingerprint evidence.
From the information given it is now reasonable to treat the suspect as randomly-
chosen from among the 100 people with access to the building. Hence we take P(I c ) =
1/100 and P(I) = 99/100 which gives

(1/50000) × (99/100)
P(I|F ) =
(1/50000) × (99/100) + 1 × (1/100)
99
=
99 + 50000
= 0.0020 (to 4 decimal places).

Hence there is now only about a 0.2% chance that the suspect is innocent.

The above example illustrates the so-called “prosecutor’s fallacy” which is not just
of academic interest – it has been associated with several high-profile miscarriages
of justice.

Exercise 6.8: Wrongful conviction

Find and discuss some real-life examples where a suspect has been convicted on the basis
of faulty probabilistic arguments.

6.5 Further Exercises

Exercise 6.9: Football team
Two important members of a football team are injured. Suppose that their recoveries
before the match are independent events and each recovers with probability p. If both
are able to play then the team has probability 2/3 of winning the match, if only one of
them plays then the probability of winning is 5/12 and if neither play the probability of
winning is 1/6. Show that the condition p > 2/3 guarantees that the match is won with
probability greater than 1/2.
6.5 Further Exercises 59

Exercise 6.10: General partitioning

Which of the following partition S when A and B are arbitrary events? Justify your
answers.
(a) The four events A, Ac , B, B c ,
(b) The two events A, B \ A,
(c) The four events A \ B, B \ A, A ∩ B, (A ∪ B)c ,
(d) The three events A ∩ B, A4B, Ac ∩ B c ,
(e) The three events A, B, (A ∪ B)c .

Exercise 6.11: Lost key

Mimi and Rodolfo are looking for a key in the dark. Suppose that the key may be under
the table, behind the bookshelf or in the corridor, and has a 1/3 chance of being in each
of these places. Mimi searches under the table; if the key is there she has a 3/5 chance of
finding it. Rodolfo searches behind the bookshelf; if the key is there he has a 1/5 chance
of finding it.
(a) Calculate the probability that the key is found.
(b) Suppose that the key is found. Calculate the conditional probability that it is found
by Rodolfo.
(c) Suppose that the key is not found. Calculate the conditional probability that it is in
the corridor.
[Answers: 4/15, 1/4, 5/11]

Exercise 6.12: *Are you smarter than a pigeon? (revisited)

Consider the version of the Monty Hall problem presented in Exercise 0.2. Let the cards
be numbered 1 to 3 with your initial pick being card 1. Assume that in the case where
the ace is card 1, the street performer reveals card 2 with probability p, and card 3 with
probability 1 − p. Otherwise, the card (out of 2 and 3) which is not the ace is always
revealed.
(a) Assuming that you do not switch your choice, compute the probability of winning,
conditioned on the performer showing you card 2. Do the same conditioned on the
performer showing you card 3.
(b) Assuming that you do switch once the card has been revealed, compute the probability
of winning, conditioned on the performer showing you card 2. Again, do the same
conditioned on the performer showing you card 3. Check that regardless of the value
of p, and regardless of which card is revealed, deciding to switch is always at least as
good as deciding not to switch.
(c) Use the law of total probability to calculate the probability of winning in both cases
(a) and (b). Explain briefly why your result makes sense.
Chapter 7

Interlude (and Self-Study Ideas)

7.1 Looking Back and Looking Forward

In several of the situations seen in the first part of the course, we’ve been interested
in numerical values associated with the outcome of an experiment, e.g., the sum
of the numbers from two rolls of a die (Exercise 4.9) or the number of Heads when
tossing a coin (Exercise 2.11). This leads naturally onto the idea of a random
variable which will be the subject of the second half of the course. In order to
understand the following chapters, it is crucial that you have a good grasp of the
basic structure and definitions we have seen so far. You are thus encouraged to
spend some time in self-study and review – it is in your own interests to address
any lingering difficulties before going forward.

7.2 Tips for Reading the Lecture Notes

The lecture notes define the examinable content for the course so now would be a
good moment to reread Chapters 0 to 6 carefully. As you do that, you may find
the following helpful.

• Concentrate on understanding, not memorizing. In general, the more you

actually understand, the less you need to learn. For an open-book exam,
memorizing the notes word-by-word is especially pointless!

• Read actively, not passively. Highlight your notes or annotate them as you
read and ask yourself questions as if you’re an annoying lecturer! For exam-
ple, when you read a definition, see if you can think of cases which satisfy
it and cases which don’t. In a proof, check you understand how to get from
every line to the next; even better, cover the proof up and see if you can

60
7.3 Tips for Doing Examples/Exercises 61

work it out for yourself, uncovering to give yourself a clue where necessary.

• Watch the accompanying videos. If you need further help/explanation on a

particular topic then try rewatching the associated video; again you should
do this actively, pausing to check which bits you do or don’t understand and
then following up on queries as appropriate.

• Do the examples and exercises. As you read, you should try to redo the
examples/exercises which are already solved and attempt the “Further Ex-
ercises” if you have not done so. Remember that the starred exercises are
somewhat harder so could be skipped on a first pass of the notes; come
back to them later if you’re aiming for a high mark. For more advice on
examples/exercises, see the next section...

7.3 Tips for Doing Examples/Exercises

Perhaps the most important word in the title of this section is “doing”. University
study is not a spectator sport; to learn effectively you need to be actively developing
your own “mathematical muscles”. This means that it is not enough to read the
solution to an exercise/example or to watch a video – you must try and do it for
yourself. Some specific suggestions now follow:

• As you read a question, highlight the important words/concepts. For instance,

are you told that two events are independent or that something is chosen “at
random”? Usually such words are not there by accident but give important
clues as to how to proceed.

• Identify what you know and what you’re trying to find. If a problem is
phrased entirely in words, you will usually need to establish some notation
before you can do this (e.g., to define events) – often there are many valid
notational choices but you need to be clear and consistent.

• Think about the main tools that might help get from what you know to what
you’re trying to find. If you suspect a particular theorem or definition will
be useful, make sure you have the exact statement to hand.

• In a written solution, show all your working. For a proof you should try to
justify every step; for a more applied calculation you should indicate at least
the main methods you are using (e.g., “by inclusion-exclusion” or “using
Proposition 2.7”).
7.3 Tips for Doing Examples/Exercises 62

• Consider how to check your answer. This is a really important skill since
in the real world (and in exams!) there are rarely solutions to consult. For
instance, you can ask yourself whether a calculated probability is plausible
and whether there might be another way to do the same question.
Chapter 8

Introduction to Random Variables

8.1 Concept of a Random Variable

In many real-life experiments, you may be chiefly concerned with some numerical
quantity not with the actual outcome. For example, you might care about the
sum of the numbers on two dice when playing Monopoly, the number of questions
right in a multiple-choice quiz,1 or the percentage of the electorate voting for a
particular candidate. Loosely speaking, a random variable is a “machine” which
takes as input an outcome in the sample space and gives as output a single number.
More formally, a random variable is a function.

Definition 8.1. A random variable is a function from S to R.

Remarks:

• If S is uncountable then this definition is, in fact, not quite correct. It turns
out that some functions are too complicated to regard as random variables
(just as some sets are too complicated to regard as events). This subtlety is
well beyond the scope of this module and will not concern us at all.

• Random variables are usually denoted by capital letters but should not be
confused with events. To aid the distinction, in this course we generally use
letters from towards the beginning of the alphabet for events and letters from
towards the end for random variables.

Exercise 8.1: Real-life random variables

Think of some more examples of random variables in real life. Can you find two different
random variables associated with the same experiment?
1
Of course, you should care also about understanding which questions you got wrong.

63
8.1 Concept of a Random Variable 64

Random variables and events are different concepts but, as we now discuss,
events can be described in terms of random variables. If X is a random variable
then P(X) makes no sense as X is not an event. The set of all outcomes ω ∈ S
such that X(ω) = x is, however, an event. Note that we use a lowercase letter
(sometimes labelled with a subscript) to denote a particular value of a random
variable. We use the shorthand “X = x” for the set {ω ∈ S : X(ω) = x}.
Hence, for example, P(X = 2) does makes sense; it is the probability that the
random variable X takes the value two. Similarly, “X ≤ x” denotes the set
{ω ∈ S : X(ω) ≤ x} so we can write things like P(X ≤ 6).
Another type of event involves the relationship between the values of different
random variables for the same experiment. For example, if Y and Z are both
random variables on the same sample space (i.e., functions with the same domain),
then “Y > 2Z” is shorthand for the set {ω ∈ S : Y (ω) > 2Z(ω)}; in other words,
P(Y > 2Z) is the probability of the event that the value of the random variable Y is
more than double the value of the random variable Z. There will be more detailed
analysis of cases where several random variables are of interest in Chapters 11
and 12.

Example 8.2: Sum of two dice

Suppose that we roll two fair six-sided dice and record the numbers showing as an ordered
pair. Let X denote the sum of the two numbers.

(a) Describe X as a function, identifying its domain, co-domain, and range.

(b) Evaluate each of the following or explain why it does not make sense: X( (5, 2) ),
X( (6, 4) ), X( (4, 6) ), X( (2, 2) ), X(5, 6), X(∅).

(c) Determine the following probabilities: P(X = 5), P(X = 3), P(X = 1), P(X ≤ 2),
P(X ≤ 12).

Solution:
The sample space is the set of all ordered pairs with elements which are integers between
1 and 6 (inclusive), i.e., S = {(j, k) : j, k ∈ {1, 2, 3, 4, 5, 6}}, and |S| = 36 (cf. Exercise 1.3,
amongst others).

(a) To get the sum of the two dice, then for an outcome (j, k) we must write down j + k.
This recipe is a function (j, k) 7→ j + k from S, to the set of integers Z (or if you prefer
the set of natural numbers N. With this definition, we see that the domain is S and
the co-domain is Z (or N). It is also obvious that the range, i.e., the set of values that
X can actually take is {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}.

(b) The first four function values are easily obtained:

X( (5, 2) ) = 7, X( (6, 4) ) = 10 X( (4, 6) ) = 10, and X( (2, 2) ) = 4.

8.2 Distributions of Discrete Random Variables 65

[Note that the function X, in common with most random variables, is not injective.]
However, rather sneakily, X(5, 6) does not make sense as the input must be a pair
(an element of S), not just two numbers. Similarly, X(∅) does not make sense as the
empty set is not an element of S here.

(c) The event “X = 5” contains all outcomes (pairs) such that the sum of the two rolls
is five, i.e., it is the set {(1, 4), (2, 3), (3, 2), (4, 1)}. Since the cardinality of this set is
four and all outcomes are equally likely, we have P(X = 5) = 4/36 = 1/9. The other
probabilities are similarly determined:

2 1
P(X = 3) = P({(1, 2), (2, 1)}) = = ,
36 18
P(X = 1) = P(∅) = 0,
1
P(X ≤ 2) = P(X = 2) = P({(1, 1)}) = ,
36
P(X ≤ 12) = P(S) = 1.

Exercise 8.3: Head count

Suppose we toss a fair coin three times and denote outcomes in the sample space S by
listing, in order, the observed Heads (h) and Tails (t).

(a) Let X be the random variable counting the number of Heads, and Y be the random
variable counting the number of Tails.

(i) State the range of the functions X and Y , i.e., list the values they can take.
(ii) Evaluate X(hht) and Y (hht).

(b) Let Z be another random variable defined as Z = max{X, Y }.

(i) State the range of the function Z, i.e., list the values it can take.
(ii) Evaluate Z(hht).

(c) Determine the probability that we see more Tails than Heads.

8.2 Distributions of Discrete Random Variables

Recall from the previous section that we denote a random variable by an uppercase
letter and a particular value of the random variable by a lowercase letter (some-
times labelled with a subscript). We can classify random variables according to
the set of values that they take.

Definition 8.2. A random variable X is discrete if the set of values that X takes
is either finite or countably infinite.

In this case we can label the possible values x1 , x2 , x3 , etc. and use xk for a generic
value. In this course we only really consider such discrete random variables.
8.2 Distributions of Discrete Random Variables 66

The case of continuous random variables is more complicated2 but practically

important – you will certainly encounter it in future probability/statistics courses.

Exercise 8.4: Classifying real-life random variables

Look back at the examples you suggested in Exercise 8.1 and identify whether each of
them is a discrete or a continuous random variable.

Now let us turn to the central question of how to describe the probability
distribution of a discrete random variable. We need to associate probabilities to
the events of the random variable taking each possible value; this information is
encoded in the so-called probability mass function.

Definition 8.3. The probability mass function (p.m.f.) of a discrete random

variable X is the function which given input x has output P(X = x):

x 7→ P(X = x).

Remarks:

• The p.m.f. is sometimes denoted by p, i.e., we define p(x) = P(X = x); we

must have p(xk ) > 0 if xk is a possible value of the random variable.

• Do not confuse the lower case p, which is the “name” of a function, with the
P for probability; p(X = x) and P(x) are both wrong notation.

• In situations with more than one random variable we can label each p.m.f.
with a subscript, e.g., pX (x) = P(X = x) and pY (y) = P(Y = y).

In the next section we will return to general properties of the p.m.f.; for the present
we note that it can be given either by a closed-form expression or by a table, as
illustrated in the following examples and exercises.

Example 8.5: Sum of two dice (revisited)

Determine the probability mass function of the random variable X (sum of two dice rolls)
from Example 8.2.

Solution:
The random variable X takes values in the set {2, 3, 4, . . . , 12}. Since the dice are fair, we
can calculate probabilities from the cardinalities of the associated events just as we did
2
If X is a continuous random variable, what can you say about P(X = x)?
8.2 Distributions of Discrete Random Variables 67

previously in Example 8.2:

|{(1, 1)}| 1
P(X = 2) = = ,
36 36
|{(1, 2), (2, 1)}| 2 1
P(X = 3) = = = ,
36 36 18
|{(1, 3), (2, 2), (3, 1)}| 3 1
P(X = 4) = = = ,
36 36 12
|{(1, 4), (2, 3), (3, 2), (4, 1)}| 4 1
P(X = 5) = = = ,
36 36 9
|{(1, 5), (2, 4), (3, 3), (4, 2), (5, 1)}| 5
P(X = 6) = = ,
36 36
|{(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)}| 6 1
P(X = 7) = = = ,
36 36 6
|{(2, 6), (3, 5), (4, 4), (5, 3), (6, 2)}| 5
P(X = 8) = = ,
36 36
|{(3, 6), (4, 5), (5, 4), (6, 3)}| 4 1
P(X = 9) = = = ,
36 36 9
|{(4, 6), (5, 5), (6, 4)}| 3 1
P(X = 10) = = = ,
36 36 12
|{(5, 6), (6, 5)}| 2 1
P(X = 11) = = = ,
36 36 18
|{(6, 6)}| 1
P(X = 12) = = .
36 36
These results can simply be displayed in a table of the p.m.f.:

x 2 3 4 5 6 7 8 9 10 11 12
.
P(X = x) 1/36 1/18 1/12 1/9 5/36 1/6 5/36 1/9 1/12 1/18 1/36

Exercise 8.6: *Sum of two dice (re-revisited)

Find a formula for the p.m.f. in Example 8.2. [Hint: It may help to rewrite all the
probabilities with the same denominator.]

Example 8.7: Waiting for a Tail

Suppose you toss a fair coin until it comes up Tails. Determine the probability mass
function of the random variable T which counts the number of tosses.

Solution:
Denoting as usual a Head by h and a Tail by t, the sample space can be written as
{t, ht, hht, hhht} where, e.g., hhht means three Heads followed by a Tail. The random
variable T takes values 1, 2, 3, 4, ..., i.e., values from the countably infinite set of natural
numbers. It is easy to see that the function T is injective so to calculate the p.m.f., we have
to calculate the probabilities of simple events. Since the coin is fair and different tosses
are physically unrelated (so we can assume independence and multiply probabilities) we
8.3 Properties of the Probability Mass Function 68

have
1
P(T = 1) = P({t}) = ,
2
1 1 1
P(T = 2) = P({ht}) = × = ,
2 2 4
1 1 1 1
P(T = 3) = P({hht}) = × × = ,
2 2 2 8
..
.

It is easy to spot the pattern and we can write the p.m.f. as a table

n 1 2 3 4 ...
,
P(T = n) 1/2 1/4 1/8 1/16 ...

or as a compact formula 
1

2n for n ∈ N
P(T = n) =
0 otherwise.

[Note that the “0 otherwise” line is sometimes not written in p.m.f. formulae, it being
simply assumed that the probability is zero for values of the random variable which are
not explicitly listed.]

Exercise 8.8: Balls in a bag

Four balls are randomly selected, without replacement, from a bag that contains 10 balls
numbered from 1 to 10. Let X denote the largest number selected.

(a) List the values X takes.

(b) Calculate P(X = 5).

(c) Write a formula for the p.m.f. of X.

(d) Calculate P(X > 5).

8.3 Properties of the Probability Mass Function

Since the values assigned by the probability mass function are probabilities, they
must obey Kolmogorov’s axioms. In particular, this means they must add up to
one.

Proposition 8.4. If X is a discrete random variable which takes values x1 , x2 , x3 , . . .,

then
X
P(X = xk ) = P(X = x1 ) + P(X = x2 ) + P(X = x3 ) + · · · = 1
k

where the sum is over all values which X takes (a finite or infinite set).
8.3 Properties of the Probability Mass Function 69

Proof:
The random variable X takes the values xk (with k = 1, 2, 3, . . .); we let Ak be
the event “X = xk ”. The Ak ’s are pairwise disjoint which can easily be proved by
contradiction. [If Ai and Aj were not disjoint for some i 6= j then Ai ∩ Aj would
contain at least one common element, say ω, but that would mean that X(ω) = xi
and X(ω) = xj which is impossible if i 6= j.] Furthermore A1 ∪ A2 ∪ · · · = S since,
for any ω ∈ S, X(ω) takes some value. [X(ω) = xi means ω ∈ Ai so there is no
ω ∈ S which is not in one of the Ak ’s.] In other words, the Ak ’s partition the
sample space; together with Kolmogorov’s axioms, this yields
X
P(X = xk ) = P(X = x1 ) + P(X = x2 ) + P(X = x3 ) + · · ·
k
= P(A1 ) + P(A2 ) + P(A3 ) + · · ·
= P(A1 ∪ A2 ∪ A3 ∪ · · · ) [using Definition 2.1(c)]
= P(S)
=1 [using Definition 2.1(b)]

and so the result is proved.

Note that Proposition 8.4 provides a good way to check that a calculated p.m.f. is
at least reasonable.

Example 8.9: Checking probability mass functions

Check that Proposition 8.4 holds for the p.m.f. of X in Example 8.5, and for the p.m.f.
of T in Example 8.7

Solution:
For the p.m.f. of X, we have to check a finite sum:

12
X
P(X = x) = P(X = 2) + P(X = 3) + P(X = 4) + P(X = 5) + P(X = 6) + P(X = 7)
x=2
+ P(X = 8) + P(X = 9) + P(X = 10) + P(X = 11) + P(X = 12)
1 2 3 4 5 6 7 6 5 4 3 2
= + + + + + + + + + + +
36 36 36 36 36 36 36 36 36 36 36 36
36
=
36
= 1.
8.3 Properties of the Probability Mass Function 70

For the p.m.f. of T , we have to check an infinite sum:

∞
X
P(T = n) = P(T = 1) + P(T = 2) + P(T = 3) + P(T = 4) + · · ·
n=1 1 1 1 1
= + + + + ···
2  4 8 16 
1 1 1 1
= 1 + + + + ···
2 2 4 8
1 1
= ×
2 1 − (1/2)
1/2
=
1/2
=1

[We have used here the formula for the sum of a geometric series (see Exercise 8.14); you
will learn much more about infinite series in the module Calculus II.] Hence, in both cases
Proposition 8.4 holds, as of course it must.

The fact that the events “X = xk ” are pairwise disjoint means that we can
find the probabilities of other events by summing values of the probability mass
function. For example, if the random variable X takes values in the integers, then
P(0 ≤ X < 3) = P(X = 0) + P(X = 1) + P(X = 2). We can also find the p.m.f.
of another random variable, say Y , which is itself a function of X, by considering
which values of X are mapped to which values of Y .

Exercise 8.10: From X to Y

A random variable X has the following probability mass function:

x −2 −1 0 1 2
.
P(X = x) 1/10 2/5 1/4 1/5 1/20

Let Y be a new random variable defined by Y = X 2 + 4.

(a) List the values Y takes.

(b) Find the probability mass function of Y .

We conclude this chapter by remarking that a closely related function to the

probability mass function is the cumulative distribution function (c.d.f.), usu-
ally denoted by F , which given input x has output P(X ≤ x). This plays an
important role in more advanced probability theory.3

3
For continuous (non-discrete) random variables, one can still define the c.d.f. but the p.m.f.
is replaced by a probability density function (p.d.f.).
8.4 Further Exercises 71

8.4 Further Exercises

Exercise 8.11: Probability practice
Calculate the following probabilities for the random variable X of Exercise 8.10: P(X = 2),
P(X = 3), P(X ≤ 1), P(X 2 < 2).
[Answers: 1/20, 0, 19/20, 17/20]

Exercise 8.12: Choosing your marbles

A bag contains six red marbles and two blue marbles. You choose five at random without
replacement. Let B be the number of blue marbles you end up with and R be the number
of red marbles you end up with. Find the probability mass function of B. Without doing
any more calculations, write down the probability mass function of R.

Exercise 8.13: Unfair tossing

A coin which has probability p of coming up Heads is tossed three times. Let X be the
number of Heads observed.

(a) List the values which the random variable X takes.

(b) Compute the probability mass function of X.

(c) Confirm the statement in Proposition 8.4 for the p.m.f. calculated in (b).

Exercise 8.14: *Geometric series

(a) Let z 6= 1 be a real number and n be a positive integer. Show that

n−1
X 1 − zn
1 + z + z 2 + z 3 + · · · + z n−1 = zk = .
1−z
k=0

[Hint: Define Sn = 1 + z + z 2 + z 3 + · · · + z n−1 and take the difference of Sn and zSn .]

(b) Now assume that |z| < 1. By taking the limit n → ∞, derive the sum of the geometric
series
∞
X 1
1 + z + z2 + z3 + · · · = zk = .
1−z
k=0

(c) Suppose you toss the coin of Exercise 8.13 until either a Head appears or a total
of n Tails has been seen. Let the random variable T be the number of flips made.
Determine the probability mass function of T and use your result from part (a) to
verify that Proposition 8.4 holds.

Exercise 8.15: **Coin games

(a) Let the random variable Yn be the number of Tails appearing when a fair coin is
tossed n times. Determine the probability mass function of Yn and hence deduce a
8.4 Further Exercises 72

closed-form expression for the sum

n  
X n
.
k
k=0

(b) You play a game where you first choose a positive integer n and then flip a fair coin
n times. You win a prize if you get exactly two Tails. How should you choose n
to maximize your chances of winning? What is the probability of winning with an
optimal choice of n?
Chapter 9

Expectation and Variance

9.1 Expected Value

In this chapter we start thinking about how to characterize properties of the dis-
tributions of (discrete) random variables. Let us start by imagining that we toss
a fair coin 100 times. How many Heads should we expect to see? We’ll return
to this question when we introduce the binomial distribution in the next chapter
but intuition probably already gives you a good idea of the answer. This thought
experiment illustrates the idea of expected value or expectation which is defined
as follows.

Definition 9.1. If X is a discrete random variable which takes values x1 , x2 , x3 , . . .,

then the expectation of X (or the expected value of X) is defined by
X
E(X) = xk P(X = xk )
k

= x1 P(X = x1 ) + x2 P(X = x2 ) + x3 P(X = x3 ) + · · · .

Remarks:

• The expectation is sometimes called the mean and sometimes denoted by µ

(the Greek letter mu).

• The sum again ranges over all the possible values of the random variable;
there are some further subtleties in the case of infinite sums (largely beyond
the scope of this course).

• The expected value does not have to be one of the possible values of the
random variable.

73
9.1 Expected Value 74

Example 9.1: Expectation of a die

Let W be the number shown on rolling a fair six-sided die. Find E(W ).

Solution:
The random variable W obviously has p.m.f. P(W = w) = 1/6 for w ∈ {1, 2, 3, 4, 5, 6} so

6
X
E(W ) = wP(W = w)
w=1
1 1 1 1 1 1
=1× +2× +3× +4× +5× +6×
6 6 6 6 6 6
7
= .
2

Exercise 9.2: Expectation of the sum of two dice

Let X be the sum of the numbers when two fair six-sided dice are rolled. Use the p.m.f.
calculated in Example 8.5 to show that E(X) = 7. Why does this result make sense in
the light of Example 9.1? [We will see this more clearly in Chapter 11.]

Example 9.1 and Exercise 9.2 clearly illustrate that the expectation may or
may not be one of the values the random variable can actually take: it is possible
for the sum of two dice rolls to be seven (in fact the most likely sum we see) but
it is certainly not possible to roll 3.5 on a single die! What then can we say about
the expectation in general? Is there any way to check our calculations? Well, you
would (hopefully!) have been surprised if you had calculated E(W ) for the single
die as 7.1 or E(X) for the two dice as 1.5. This leads to the following proposition.

Proposition 9.2. If m ≤ X(ω) ≤ M for all ω ∈ S, then

m ≤ E(X) ≤ M.

Proof:
If every value xk (k = 1, 2, 3, . . . ) that X takes is less than or equal to M , xk ≤ M ,
we have that xk P(X = xk ) ≤ M P(X = xk ) [since probabilities are non-negative
by Definition 2.1(a)] and, using also Proposition 8.4,
X X X
E(X) = xk P(X = xk ) ≤ M P(X = xk ) = M P(X = xk ) = M.
k k k

Similarly, if every value that X takes is greater than or equal to m, xk ≥ m, we

have that
X X
E(X) = xk P(X = xk ) ≥ m P(X = xk ) = m.
k k

Hence, we have m ≤ E(X) ≤ M as required.

9.2 Expectation of a Function of a Random Variable 75

This proposition is admittedly less helpful in the case of a random variable

taking values from a countably infinite set. Determining the expectation in those
cases involves an infinite sum.
Exercise 9.3: *Waiting expectation
Let T be the number of tosses of a fair coin up to (and including) the first time you see a
Tail. Use the p.m.f. found in Example 8.7 to show that E(T ) = 2.

In general, it turns out that if a random variable can take infinitely many values,
its expectation may be infinite or even not well defined.1 From the point of view
of the present course, this is a complication that need not trouble you but see
Exercise 9.17 for a non-examinable challenge.

9.2 Expectation of a Function of a Random Variable

We now turn our attention to finding the expected value of a function of a random
variable. If we know the p.m.f. of a random variable X, how can we find the
expectation of a function of X, say f (X)?
Example 9.4: From X to Y (revisited)
Find the expectation of the random variable Y = X 2 + 4 where X has the p.m.f. given in
Exercise 8.10.

Solution:
Since we already calculated the p.m.f. of Y we can simply use that to determine E(Y ):

E(Y ) = 4 × P(Y = 4) + 5 × P(Y = 5) + 8 × P(Y = 8)

1 3 3
=4× +5× +8×
4 5 20
26
= .
5

Notice, however, that we can write this calculation in another way; with f (X) = X 2 + 4,
we have

E(Y ) = 4 × P(Y = 4) + 5 × P(Y = 5) + 8 × P(Y = 8)

= 4 × P(X = 0) + 5 × [P(X = −1) + P(X = 1)] + 8 × [P(X = −2) + P(X = 2)]
= 8 × P(X = −2) + 5 × P(X = −1) + 4 × P(X = 0) + 5 × P(X = 1) + 8 × P(X = 2)
= f (−2)P(X = −2) + f (−1)P(X = −1) + f (0)P(X = 0) + f (1)P(X = 1) + f (2)P(X = 2).

You should check that by using the values in the p.m.f. of X (see Exercise 8.10) we obtain
again E(Y ) = 26/5.
1
This is related to the question of whether or not an infinite series converges which you can
learn about in “Calculus II” and similar modules.
9.2 Expectation of a Function of a Random Variable 76

The above example illustrates a general principle which is stated in the next
proposition.

Proposition 9.3. If f is a real-valued function defined on the range of a discrete

random variable X, then
X
E( f (X) ) = f (xk )P(X = xk )
k

= f (x1 )P(X = x1 ) + f (x2 )P(X = x2 ) + f (x3 )P(X = x3 ) + · · ·

where the sum ranges over all possible values xk of X.

The proof is omitted; it is straightforward but requires some slightly cumbersome

notation. Proposition 9.3 has many useful consequences.

Example 9.5: Useful expectations

Let X be a discrete random variable, taking the values x1 , x2 , x3 , . . ., and c be a constant
(a real number). Show that:

(a) E(X + c) = E(X) + c,

(b) E(cX) = cE(X).

P P P
[You may use without proof the series properties k (ak + bk ) = k ak + k bk and
P P
k cak = c k ak .]

Solution:

(a) From Proposition 9.3, we have

X
E(X + c) = (xk + c)P(X = xk )
k
X X
= xk P(X = xk ) + cP(X = xk )
k k
X
= E(X) + c P(X = xk ) [using Definition 9.1]
k

= E(X) + c × 1 [using Proposition 8.4]

= E(X) + c.

(b) Similarly, Proposition 9.3 yields

X
E(cX) = cxk P(X = xk )
k
X
=c xk P(X = xk )
k

= cE(X) [using Definition 9.1].

9.3 Moments and Variance 77

Exercise 9.6: Profit margins

Suppose that the Great Expectations restaurant prepares four takeaway meals in advance
each evening at a cost of £4 each. Each takeaway meal is sold for £9 but any unsold
meal goes to waste. Let X denote the number of these meals sold in a given evening
and Y denote the profit made on them (in pounds). The restaurant owner observes that
P(X = 0) = 1/12, P(X = 3) = 1/6, P(X = 4) = 1/8, and E(X) = 2.

(a) Find the p.m.f. of X.

(b) Determine E(Y ).

9.3 Moments and Variance

An important special case of the treatment in the previous section is expectations
of X n (where n is a natural number); these expectations are the moments of the
random variable X.

Definition 9.4. The nth moment of the random variable X is the expectation
E(X n ).

Such expectations can easily be calculated for discrete random variables using
Proposition 9.3.2 Their values give information about the “shape” of the proba-
bility mass function. In particular, the second moment is related to the variance
which quantifies the spread of the distribution.

Definition 9.5. If X is a discrete random variable which takes values x1 , x2 , x3 , . . .,

then the variance of X is defined by
X
Var(X) = [xk − E(X)]2 P(X = xk )
k

= [x1 − E(X)]2 P(X = x1 ) + [x2 − E(X)]2 P(X = x2 )

+ [x3 − E(X)]2 P(X = x3 ) + · · · .

Remarks:

• Armed with Proposition 9.3, we see that Var(X) = E([X − E(X)]2 ), i.e., it
is the expectation of the square of the difference between X and E(X).

• The variance measures how sharply concentrated X is about E(X), with a

small variance meaning sharply concentrated and a large variance meaning
spread out.
2
For random variables taking infinitely many values, the moments may again be infinite or
not well defined; that subtlety does not concern us here.
9.3 Moments and Variance 78

• Since the square of any real number is non-negative and the values of the
p.m.f. are also non-negative [from Definition 2.1(a) of course], it is clear that
Var(X) ≥ 0.

• The square root of the variance is called the standard deviation. Mathe-
matically it is usually more convenient to work with the variance than the
standard deviation.

The concept of the variance as a measure of spread is illustrated by the following

example.

Example 9.7: Competing investments

Let X be the amount (in pounds) you get from one investment and Y be the amount (in
pounds) you get from a second investment. Suppose that X takes value 99 with probability
1/2 and value 101 with probability 1/2 while Y takes value 90 with probability 1/2 and
value 110 with probability 1/2.

(a) Compare E(X) and E(Y ).

(b) Compare Var(X) and Var(Y ).

Solution:

(a) From Definition 9.1 we have

1 1
E(X) = 99 × + 101 × = 100,
2 2
and
1 1
E(Y ) = 90 × + 110 × = 100.
2 2
[These results are also obvious from a symmetry argument.] Hence the expectations
of the amounts gained from the two investments are the same.

(b) From Definition 9.5 we have

1 1
Var(X) = (99 − 100)2 × + (101 − 100)2 × = (1)2 = 1,
2 2
and
1 1
Var(Y ) = (90 − 100)2 × + (110 − 100)2 × = (10)2 = 100.
2 2
So the variance of Y is much bigger than that of X; we can interpret this as the second
investment being, in some sense, riskier.

There is also a useful alternative formula for the variance.

Proposition 9.6. If X is a discrete random variable then

Var(X) = E(X 2 ) − [E(X)]2 .

9.3 Moments and Variance 79

Proof:
As usual we write x1 , x2 , x3 , . . . for the possible values of X. Starting from Defini-
tion 9.5 and the fact that [xk − E(X)]2 = (xk )2 − 2E(X)xk + [E(X)]2 , we have
X
Var(X) = [xk − E(X)]2 P(X = xk )
k
X X X
= (xk )2 P(X = xk ) − 2E(X)xk P(X = xk ) + [E(X)]2 P(X = xk )
k k k
X 2
X
2
= E(X ) − 2E(X) xk P(X = xk ) + [E(X)] P(X = xk ) [from Proposition 9.3]
k k

= E(X 2 ) − 2E(X) × E(X) + [E(X)]2 × 1 [using Definition 9.1 and Proposition 8.4]
= E(X 2 ) − [E(X)]2 .

Remarks:

• You can remember this expression for the variance as “the mean of the square
minus the square of the mean”.

• Since Var(X) ≥ 0, we must have E(X 2 ) ≥ [E(X)]2 .

Example 9.8: Variance of a die

Let W be the number shown on rolling a fair six-sided die (as in Example 9.1). Find
Var(W ).

Solution:
Using the formula in Proposition 9.6 we have

Var(W ) = E(W 2 ) − [E(W )]2

6  2
X 7
= w2 P(W = w) − [using result of Example 9.1]
w=1
2
 2
1 1 1 1 1 1 7
= 12 × + 22 × + 32 × + 42 × + 52 × + 62 × −
6 6 6 6 6 6 2
91 49
= −
6 4
35
= .
12

Exercise 9.9: Variance of the sum of two dice

Let X be the sum of the numbers when two fair six-sided dice are rolled. Use the p.m.f. cal-
culated in Example 8.5 to calculate Var(X) using both Definition 9.5 and Proposition 9.6.
Which method do you find easiest? [Answer (to first part): 35/6]

Exercise 9.10: Why oh why?

Let Y be a discrete random variable with E(Y ) = 2 and Var(Y ) = 6. Find E(2Y 2 ).
9.4 Useful Properties of Expectation and Variance 80

9.4 Useful Properties of Expectation and Variance

Linear functions of random variables are often found in applications. In this section
we summarize the particular properties of the expectation and variance in such
cases.

Proposition 9.7. If a, b ∈ R and X is a discrete random variable, then

E(aX + b) = aE(X) + b.

Remarks:

• The proof of Proposition 9.7 is left as an exercise; it is very similar to the

proofs done in Example 9.5.

• Setting a = 0 yields the special case E(b) = b. This corresponds to the

expectation of a so-called degenerate random variable which takes value b
with probability one.

• The property of “linearity of expectations” also extends to sums of higher

moments, for example, E(3X 2 − 4X + 5) = 3E(X 2 ) − 4E(X) + 5. We shall
see further generalization in Chapter 11.

Proposition 9.8. If a, b ∈ R and X is a discrete random variable, then

Var(aX + b) = a2 Var(X).

Proof:
Starting from Definition 9.5, we have

Var(aX + b) = E([aX + b − E(aX + b)]2 )

= E([aX + b − aE(X) − b)]2 ) [using Proposition 9.7]
= E(a2 [X − E(X)]2 )
= a2 E([X − E(X)]2 ) [using Proposition 9.7]
= a2 Var(X),

Remarks:

• Note that adding a constant to a random variable does not change the vari-
ance; this is intuitively reasonable as the spread of the distribution is un-
changed by a shift.

• An important special case is Var(b) = 0.

9.5 Further Exercises 81

Example 9.11: Empirical mean

Let X once again be the sum of the numbers when two fair six-sided dice are rolled. Now
define Y as the “empirical mean value” of the rolls, i.e., Y = X/2. Find the expectation
and variance of Y .

Solution:
Using the results of Exercises 9.2 and 9.9, together with Propositions 9.7 and 9.8, we have
 
X 1 7
E(Y ) = E = E(X) = ,
2 2 2

and    2
X 1 35/6 35
Var(Y ) = Var = Var(X) = = .
2 2 4 24
Notice that E(Y ) is the same as the expectation of the number on a single die (Example 9.1)
but Var(Y ) is smaller than the variance of the number on a single die (Example 9.8). Can
you understand why?

Exercise 9.12: Puzzling p.m.f.

Suppose that the discrete random variable X has E(3X + 7) = 10 and Var(3X + 7) = 36.
Give one example of a possible p.m.f. for X.

In the next chapter we will derive formulae for the expectation and variance of
various distributions which appear so frequently that they are given special names.

9.5 Further Exercises

Exercise 9.13: Calculating expectations and variances
Let X be a discrete random variable with E(X) = 5 and Var(X) = 2/3. Find the following:

(a) E(3X)

(b) Var(3X),

(c) E(4 − 3X),

(d) Var(4 − 3X),

(e) E(4 − 3X 2 ).

[Answers: 15, 6, −11, 6, −73]

Exercise 9.14: Unfair tossing (revisited)

A coin which has probability p of coming up Heads is tossed three times. Let X be the
number of Heads observed (see Exercise 8.13).

(a) Compute the expectation of X.

(b) Compute the variance of X.

9.5 Further Exercises 82

Exercise 9.15: *More series

Suppose |z| < 1. Using the geometric series, or otherwise, show that:

(a)
∞
X 1
1 + 2z + 3z 2 + 4z 3 + · · · = kz k−1 = ,
(1 − z)2
k=1

(b)
∞
X 2 1
1 + 4z + 9z 2 + 16z 3 + · · · = k 2 z k−1 = 3
− .
(1 − z) (1 − z)2
k=1

Exercise 9.16: *An e-zee proof ?

Let Z be a random variable taking values in the set {0, 1, 2, 3, . . . , n}.

(a) Show that

n
X
E(Z) = P(Z ≥ i).
i=1

(b) Deduce that if E(Z) < 1, then Z takes the value 0 with non-zero probability.

(c) Use part (a) to prove that for all 1 ≤ t ≤ n we have

E(Z)
P(Z ≥ t) ≤ .
t

Exercise 9.17: **European coin games

(a) Consider a game where you flip a fair coin until it comes up Heads, starting with £2
and doubling the prize fund with every appearance of Tails. Let the random variable
N be the number of flips; if N takes value n, you win £2n . Let X denote the prize
you win (in pounds). Find E(X) and discuss how much you would be prepared to pay
to enter such a game.

(b) Now consider the following two-player game. Angela and Boris flip a fair coin until
it comes up Tails. The number of flips needed is again a random variable N taking
values n = 1, 2, 3, . . . . If n is odd then Boris pays Angela 2n €; if n is even then Angela
pays Boris 2n €. Let Y denote Boris’ net reward (in euros). Show that E(Y ) does not
exist.
Chapter 10

Special Discrete Random Variables

10.1 Bernoulli Distribution

We begin our survey of special distributions with a very easy case which will
nevertheless introduce some important concepts.
Consider an experiment where there are only two possible outcomes labelled
“success” and “failure”. Note that no moral judgement is implied by these labels;
“success” could be something as mundane as a coin landing on Heads. This set-up
is called a Bernoulli trial. Now suppose that the probability of success is p,
i.e., P({success}) = p. Then the random variable defined by X(success) = 1 and
X(failure) = 0, has the probability mass function

k 0 1
.
P(X = k) 1−p p

This p.m.f. is called the Bernoulli distribution, with parameter p, and we write
X ∼ Bernoulli(p), where the symbol “∼” loosely means “has the distribution of”.
As with all the distributions in this chapter, we are interested in the expectation
and variance. In this case they are very easily calculated; from Definitions 9.1
and 9.5, we have

E(X) = 0 × (1 − p) + 1 × p = p, (10.1)
Var(X) = (0 − p)2 × (1 − p) + (1 − p)2 × p = p(1 − p)[p + (1 − p)] = p(1 − p).
(10.2)

Note that the Bernoulli distribution applies whenever the sample space of an exper-
iment is partitioned into “success” and “failure” and we are interested in whether
or not a “success” occurs.

83
10.2 Binomial Distribution 84

Example 10.1: Six success

Consider once more throwing a fair six-sided die. Let the random variable Y take the
value 1 when a “six” is rolled and be 0 otherwise. Find E(Y ) and Var(Y ).

Solution:
The sample space of the experiment is {1, 2, 3, 4, 5, 6}. Letting A be the event that we
roll a six (i.e., A = {6} and Ac = {1, 2, 3, 4, 5}), then we have Y (A) = 1 and Y (Ac ) = 0.
In this experiment, rolling a six is identified with “success”; the success probability is
P(Y = 1) = P(A) = 1/6 and so Y ∼ Bernoulli(1/6). Hence, from (10.1) and (10.2) above,
we obtain  
1 1 1 5
E(Y ) = and Var(Y ) = 1− = .
6 6 6 36

10.2 Binomial Distribution

Often we are interested in the number of “successes”, not from a single trial but
from multiple repeated trials (e.g., the number of Heads when we toss a coin 100
times). Building on the previous section, we thus consider performing n inde-
pendent Bernoulli trials, each with the same probability p of success. [Indepen-
dent trials means that if Ei denotes the event “the ith trial is a success” then
E1 , E2 , . . . , En are mutually independent events.] Let X be the number of “suc-
cesses” in these n trials. To determine the p.m.f. of X we need to evaluate the
probabilities P(X = k) for k = 0, 1, 2, . . . , n. We can do this by the following
argument.

• First consider the special outcome that the first k trials are successes and the
remaining n−k trials are failures. Using mutual independence the probability
of this simple event is pk (1 − p)n−k .

• The event “X = k” contains n

k different outcomes with k successes and
n−k failures.1 Since the trials are identical, each outcome occurs with the
same probability pk (1 − p)n−k .

Hence we obtain the p.m.f.

 
n k
P(X = k) = p (1 − p)n−k for k = 0, 1, 2, . . . , n.
k

We call this the binomial distribution, with parameters n and p, and write
X ∼ Bin(n, p). Note that the Bernoulli(p) distribution is just Binomial(1, p).
1
You can think of this as unordered sampling of k trials from n without replacement: the
order doesn’t matter but once a trial has been picked to be a success, it can’t be picked again.
10.2 Binomial Distribution 85

In dealing with the binomial distribution, the following identities are often
useful:
n  
X n
(a + b)n = ak bn−k [binomial theorem], (10.3)
k
k=0
   
n n n−1
= . (10.4)
k k k−1

Exercise 10.2: Identity check

(a) Use (10.3) to verify Proposition 8.4 for the binomial distribution.

(b) Starting from the definition of the binomial coefficient, prove (10.4).

Armed with (10.3) and (10.4), we now turn to the expectation and variance. The
expectation is calculated as
n
X
E(X) = kP(X = k)
k=0
n  
X n k
= k p (1 − p)n−k
k
k=1
n  
X n−1 k
= n p (1 − p)n−k [using (10.4)]
k−1
k=1
n  
X n − 1 k−1
= np p (1 − p)n−k
k−1
k=1
n−1
X n − 1
= np p` (1 − p)n−1−` [setting ` = k − 1]
`
`=0

= np[p + (1 − p)]n−1 [using (10.3) with a = p and b = 1 − p]

= np. (10.5)

Similarly, one can show that the variance is

Var(X) = np(1 − p). (10.6)

Exercise 10.3: *Variance of binomial distribution

Prove the expression (10.6) for the variance of the binomial distribution by using (10.4)
twice. [Hint: First calculate E(X 2 ) − E(X).]

In fact, we will see a much easier argument for (10.5) and (10.6) in the next chapter!
10.3 Geometric Distribution 86

Example 10.4: Ten tosses

Suppose you toss a fair coin ten times. Determine the expectation and variance of the
number of Heads seen.

Solution:
Let the number of Heads seen in ten tosses be Z. For a fair coin the probability of a Head
on each toss (i.e., the success probability in each trial) is 1/2. Hence Z ∼ Bin(10, 1/2)
and, from (10.5) and (10.6),
 
1 1 1 5
E(Z) = 10 × = 5 and Var(Z) = 10 × × 1 − = .
2 2 2 2

10.3 Geometric Distribution

Suppose we make an unlimited number of independent Bernoulli trials, each with
(non-zero) success probability p, and let T be the number of trials up to and
including the first success. In fact, we already saw a demonstration of this in
Example 8.7: tossing a coin repeatedly until a Tail appears. To find the probability
mass function for the general situation, we note that “T = k” is a simple event
consisting of a single outcome: k − 1 failures, each with probability 1 − p, followed
by one success with probability p. Since the trials are independent, we can multiply
probabilities to obtain

P(T = k) = (1 − p)k−1 p for k = 1, 2, 3, . . . .

We say that T has the geometric distribution with parameter p and write
T ∼ Geom(p). A word of warning is in order here: there is an alternative defini-
tion of the geometric distribution which involves counting the number of failures
(0, 1, 2, . . .) before the first success; in this course, we always use the definition
above but you should check carefully if consulting other books or websites.
In order to derive the expectation and variance of the geometric distribution
we will need the following two results for the sums of infinite series with |z| < 1:
∞
X 1
kz k−1 = 1 + 2z + 3z 2 + 4z 3 + . . . = , (10.7)
(1 − z)2
k=1
∞
X 2 1
k 2 z k−1 = 1 + 4z + 9z 2 + 16z 3 + . . . = 3
− . (10.8)
(1 − z) (1 − z)2
k=1

[One way to prove these is to start from the known formula for the sum of a
geometric series and differentiate both sides; the derivations belong more properly
in a calculus or analysis course but see Exercise 9.15 if you want to have a try.]
10.4 Poisson Distribution 87

Now the expectation of the geometric distribution is straightforwardly given by

∞
X
E(T ) = kP(T = k)
k=1
X∞
= k(1 − p)k−1 p
k=1
X∞
=p k(1 − p)k−1
k=1
1
=p× [using (10.7) with z = 1 − p]
[1 − (1 − p)]2
1
= , (10.9)
p

while for the second moment we have

∞
X
E(T 2 ) = k 2 P(T = k)
k=1
∞
X
= k 2 (1 − p)k−1 p
k=1
 
2 1
=p 3
− [using 10.8 with z = 1 − p]
[1 − (1 − p)] [1 − (1 − p)]2
2−p
= . (10.10)
p2

Substituting from (10.9) and (10.10) into Proposition 9.6, we have for the variance

Var(T ) = E(T 2 ) − [E(T )]2

 2
2−p 1
= −
p2 p
1−p
= .
p2

Exercise 10.5: *Tail c.d.f.

As in Example 8.7, let T be the number of tosses of a fair coin up to (and including) the
first time you see a Tail. Find a formula for the cumulative distribution function P(T ≤ k).
[Hint: First consider P(T > k).]

10.4 Poisson Distribution

Suppose we consider the binomial distribution and make the number of trials n
larger and larger whilst keeping the expectation the same – this means that the
10.4 Poisson Distribution 88

success probability must be of the form λ/n with λ a constant. In the n → ∞

limit the p.m.f. becomes

λk −λ
P(X = k) = e for k = 0, 1, 2, . . . .
k!

In this case we say that X has the Poisson distribution with parameter λ and
we write X ∼ Poisson(λ). If 0 < λ ≤ 1 then the p.m.f. P(X = k) is monotonically
decreasing; if λ > 1 then the p.m.f. has a maximum at a finite value k > 0.

Exercise 10.6: **Limit of Binomial

Write down the p.m.f. for a binomial distribution with n trials and success probability
λ/n. Carefully take the limit n → ∞ and show that the result is the above expression for
the p.m.f. of a Poisson distribution.

In proofs involving the Poisson distribution, a crucial ingredient is the Taylor

series of the exponential function:
∞
X xk x2 x3
ex = =1+x+ + + ··· . (10.11)
k! 2 3!
k=0

Exercise 10.7: Checking normalization

Use the identity (10.11) to show that Proposition 8.4 holds for the Poisson distribution.

With knowledge of (10.11), we can easily calculate the expectation of the

Poisson distribution:
∞
X
E(X) = kP(X = k)
k=0
∞
X λk
= k e−λ
k!
k=1
∞
−λ
X λ × λk−1
=e
(k − 1)!
k=1
∞
−λ
X λ`
= λe [setting ` = k − 1]
`!
`=0

= λe−λ × eλ [using (10.11)]

= λ. (10.12)

For the variance, we use the trick of first considering E(X 2 ) − E(X) which can be
10.4 Poisson Distribution 89

obtained as

E(X 2 ) − E(X) = E(X 2 − X)

∞
X
= (k 2 − k)P(X = k)
k=0
∞
X λk −λ
= k(k − 1) e
k!
k=2
∞
−λ
X λ2 × λk−2
=e
(k − 2)!
k=2
∞
2 −λ
X λ`
=λ e [setting ` = k − 2]
`!
`=0

= λ2 [using (10.11)]. (10.13)

Combining (10.12) and (10.13) with Proposition 9.6 we thus have

Var(X) = E(X 2 ) − [E(X)]2

= E(X 2 ) − E(X) + E(X) − [E(X)]2
= λ2 + λ − (λ)2
= λ.

The Poisson distribution can be used as an approximation in modelling situations

with many trials and a small probability of success. It also frequently appears
for counting events happening in continuous time, e.g., the number of clicks of a
Geiger counter (monitoring radioactive decay) in a minute.
Exercise 10.8: *Poisson presents
Let WS be the number of presents per hour produced by a Senior Elf in Santa’s workshop.
WS is a Poisson random variable with probability mass function

2k e−2
P(WS = k) = for k ∈ {0, 1, 2, 3, . . .}.
k!
A Junior Elf produces WJ presents per hour, also according to a Poisson distribution but
with parameter λ = 1. One morning, a snap inspection is organised to check whether
the elves are meeting the minimum performance criterion of each producing at least one
present per hour.
(a) Determine the probability that a Senior Elf fails to meet the criterion.
(b) For a team of one Senior Elf and two Junior Elves (all working independently), find
the probability that at least one elf fails to meet the criterion.
[You may leave powers of e in your answers.]
10.5 Distributions in Practice 90

10.5 Distributions in Practice

We can summarize the content of this chapter in the following table.

Distribution Values 
P(X = k) E(X) Var(X)
1 − p for k = 0
X ∼ Bernoulli(p) X = 0, 1 p p(1 − p)
p for k = 1
n
pk (1 − p)n−k


X ∼ Bin(n, p) X = 0, 1, . . . , n k np np(1 − p)
1 1−p
X ∼ Geom(p) X = 1, 2, 3, . . . (1 − p)k−1 p
p p2
λk −λ
X ∼ Poisson(λ) X = 0, 1, 2, . . . k! e λ λ

Note, however, that the first row is not really necessary since Bernoulli(p) is just
Bin(1, p).
An oft-asked question is how to determine which distribution applies in a par-
ticular situation. Of course, sometimes an exercise or exam problem will explicitly
give a distribution (especially in the Poissonian case) but, if not, a very good clue
is the range of the random variable in question – note the differences in the second
column of the table above. Loosely speaking, we can say the following.

• If a random variable takes only two possible values, it can always be related
to a Bernoulli random variable.

• If a random variable is counting the number of times something happens in

a fixed number of independent trials, it has a binomial distribution.

• If a random variable is counting the number of independent trials until some-

thing happens, it has a geometric distribution.

• If a random variable is counting the number of times something happens in

a fixed interval (continuous time), it probably has a Poisson distribution.

In real life of course, a random variable may not have any of the above distributions
but one of them may serve as a good approximation. You will see more of this in
future courses but, for now, note that it is always a good idea to state clearly any
assumptions you are making in modelling a situation.

Exercise 10.9: Real-life distributions

Look back at the discrete random variables you thought of in Exercise 8.1. Can you
suggest what distributions any of them might have? What assumptions might be needed?

Example 10.10: Shifted Bernoulli

Show that if W is a random variable taking the value a with probability 1 − p and the
value b with probability p, then X = (W − a)/(b − a) has a Bernoulli(p) distribution.
10.5 Distributions in Practice 91

Solution:
If W = a, then X = (a − a)/(b − a) = 0; if W = b then X = (b − a)/(b − a) = 1. Hence
P(X = 0) = P(W = a) = 1 − p and P(X = 1) = P(W = b) = p, i.e., X ∼ Bernoulli(p).
[Note that one can rearrange to get W = a + (b − a)X and obtain the expectation and
variance of W from the known Bernoulli results for E(X) and Var(X); in most cases,
however, it is probably easier to calculate E(W ) and Var(W ) directly.]

Example 10.11: Random red balls

Suppose you choose n balls at random from a bag containing N balls of which M are red.
[These numbers are fixed.] Let the random variable R denote the number of red balls
picked.
(a) If you pick the balls with replacement, what is the p.m.f. of R? State the expectation
and variance in this case.
(b) If you pick the balls without replacement, what is the p.m.f. of R?

Solution:
(a) Each random pick results in a red ball with probability M/N and the outcome of each
pick is independent of all the others (i.e. we perform n independent Bernoulli trials
with p = M/N ). Hence R ∼ Bin(n, M/N ) and we have
   k  n−k
n M M
P(R = k) = 1− for k = 0, 1, 2, . . . , n,
k N N
M
E(R) = n ,
N  
M M
Var(R) = n 1− .
N N

(b) Treating the situation as unordered sampling without replacement, the sample space
has cardinality N


n since we choose n balls from N . The event “R = k” corresponds
M


to choosing k red balls from M red balls, which can be done in k ways, and choosing
−M
n − k non-red balls from N − M non-red balls, which can be done in Nn−k


ways.
Hence, since the outcomes are all equally likely,
M N −M



k n−k
P(R = k) = N

. (10.14)
n

Obviously, R cannot be larger than M and n−R cannot be larger than N −M . Hence,
the possible values k of R satisfy max(0, n − (N − M )) ≤ k ≤ min(n, M ). However,
with the convention that ka = 0 for integers k > a ≥ 0, then the above formula for

the p.m.f. is valid for k = 0, 1, 2, . . . , n and gives zero probability to any impossible
values of R. [You can check that one gets the same p.m.f. by treating the situation as
ordered sampling without replacement.]
In fact, (10.14) gives the probability mass function of the so-called hypergeometric
distribution; you don’t need to know formulae for its expectation and variance but
10.6 Further Exercises 92

you can try to derive them for a challenge (Exercise 10.17).

Exercise 10.12: Typographical trials

Assume that, on average, a typographical error is found every 1000 typeset charac-
ters. Compute the probability that a 600-character page contains fewer than two errors.
[2016 exam question (part)]

10.6 Further Exercises

Exercise 10.13: Probabilities from distributions
Suppose that A ∼ Poisson(3), B ∼ Geom(1/3), and C ∼ Bin(4, 1/6) are random variables.
Find the following probabilities:

(a) P(A = 2),

(b) P(A > 2),

(c) P(B = 3),

(d) P(B ≤ 3),

(e) P(C = 2).

You may leave any powers of e in your answers but you should simplify all factorials and
other powers.
[Answers: 9/(2e3 ), 1 − 17/(2e3 ), 4/27, 19/27, 25/216]

Exercise 10.14: Lifetime of a component

An electrical component is installed on a certain day and is inspected on each subsequent
day. Let G be the number of days until inspection reveals that the component is broken.

(a) Suppose that G ∼ Geom(p). Show that for any k, ` ≥ 0,

P(G > k + `|G > k) = P(G > `).

(b) Say in words what the conclusion of part (a) means for the lifetime of the component.
Why do you think this is sometimes called the “memoryless property” of the geometric
distribution?

Exercise 10.15: *A fishy problem

Let X be the number of fish caught by a fisherman in one afternoon. Suppose that X is
distributed Poisson(λ). Each fish has probability p of being a salmon independently of all
other fish caught. Let Y be the number of salmon caught.

(a) Suppose that the fisherman catches m fish. What is the probability that k of them
are salmon?
10.6 Further Exercises 93

(b) Show that

∞
X
P(Y = k) = P(Y = k|X = m)P(X = m).
m=k

(c) Using part (b), find the probability mass function of Y . What is the name of the
distribution of Y ?

Exercise 10.16: An argument about money

A fair coin is tossed four times. Let N be the number of instances of a Head followed by
another Head in the sequence of tosses.

(a) Your friend proposes the following solution: There are three possible ways in which
we could have a Head followed by another Head (at the first and second, the second
and third, or third and fourth toss). We have a probability 1/2 × 1/2 = 1/4 of
getting a Head followed by another Head at each of these positions. Hence N is the
number of successes in three Bernoulli trials each with success probability 1/4 and so
N ∼ Bin(3, 1/4). Explain carefully what is wrong with this argument.

(b) Determine the correct probability mass function and then the expectation and variance
of N .

Exercise 10.17: **Hypergeometric distribution

Consider the p.m.f. of R in Example 10.11(b). Derive expressions for E(R) and Var(R),
and compare your results to the situation where the balls are picked with replacement.
Chapter 11

Several Random Variables

11.1 Joint and Marginal Distributions

In real-life situations, and exam questions, we are often interested in probabilities
of events which involve two (or more) random variables. For instance, you might
want to know the probability that the number of votes for Biden is greater than
the number of votes for Trump, or the probability that two different stocks both
go up in value. You may also wonder about the relation between two different
random variables (e.g., the number of storks in a town and the number of babies
born there); we will explore the concepts of independence and correlation at the
end of this chapter and in the next one. First, however, we need to establish some
basics.

Definition 11.1. Let X and Y be two discrete random variables defined on the
same sample space and taking values x1 , x2 , . . . and y1 , y2 , . . . respectively. The
function
(xk , y` ) 7→ P( (X = xk ) ∩ (Y = y` ) )

is called the joint probability mass function of X and Y .

Remarks:

• Usually we write P(X = xk , Y = y` ) instead of P( (X = xk ) ∩ (Y = y` ) ).

• The values of the joint p.m.f. must be non-negative and sum up to one:
P P P
k ` P(X = xk , Y = y` ) = 1 (also written as k,` P(X = xk , Y = y` ) = 1).

• Often we write the joint p.m.f. in the form of a table.

• The definition can be easily extended to joint distributions of three (or more)
random variables but three dimensional tables are more difficult to construct!

94
11.1 Joint and Marginal Distributions 95

Example 11.1: Coloured balls

A bag contains three red balls, two yellow balls, and two green balls. Suppose that we pick
three balls at random (without replacement). Let R denote the number of red balls we pick,
and Y the number of yellow balls we pick. Find P(R = 1, Y = 1) and P(R = 3, Y = 1).

Solution:
We can treat this situation as unordered sampling without replacement (see Section 3.4).
As we are picking three balls from a set of seven balls, we have
 
7 7!
|S| = = = 35.
3 4! 3!

The event (R = 1) ∩ (Y = 1) is the event that we pick one red ball (from three red balls),
one yellow ball (from two yellow balls), and one green ball (from two green balls). Since
all outcomes are equally likely, we can calculate the probability of this event as:
3

2
2

|(R = 1) ∩ (Y = 1)| 1 1 1 3×2×2 12
P(R = 1, Y = 1) = = = = .
|S| 35 35 35

It is even easier to calculate P(R = 3, Y = 1); it is impossible to draw three red balls and
one yellow ball if we only draw three balls in total so (R = 3) ∩ (Y = 1) = ∅ and

P(R = 3, Y = 1) = 0.

Exercise 11.2: Coloured balls (revisited)

In the set-up of Example 11.1, evaluate P(R = r, Y = y) for all possible values of R and
Y . Use your results to complete the following table for the joint p.m.f.:

HH R
H
0 1 2 3
Y H
HH
0 .
1 12/35 0
2

The next proposition relates the joint distribution P(X = xk , Y = y` ) and the
so-called marginals P(X = xk ) and P(Y = y` ).

Proposition 11.2. Let X and Y be two discrete random variables defined on the
same sample space and taking values x1 , x2 , . . . and y1 , y2 , . . . respectively. The
marginal distribution of X can be obtained from the joint distribution as
X
P(X = xk ) = P(X = xk , Y = y` ).
`
11.2 Expectations in the Multivariate Context 96

Similarly, the marginal distribution of Y is given by

X
P(Y = y` ) = P(X = xk , Y = y` ).
k

Loosely speaking, the idea is that if we only care about the probability of X
taking a particular value, we need to sum over all possible values of Y (and vice
versa, of course). The values of the marginals are the column sums and the row
sums in a table of the joint p.m.f.; they can be written in the margins, hence the
name.

Exercise 11.3: *Marginal proof

Give a proof of Proposition 11.2. [Hint: Use Definition 2.1(c) together with the fact that
the events “Y = yk ” (k = 1, 2, . . .) partition the sample space.]

Exercise 11.4: Coloured balls (re-revisited)

Find the marginal distributions of R and Y from Example 11.1 and hence calculate E(R)
and E(Y ).

11.2 Expectations in the Multivariate Context

After doing Exercise 11.4, you may wonder if it is possible to calculate say E(Y )
from a joint p.m.f. without first calculating the marginal distribution P(Y = yk ).
You may also wonder if one can define/calculate expectations of functions of two
random variables in a similar way to we did for functions of a single random
variable in Proposition 9.3. Both these questions are answered in the affirmative
by the following proposition.

Proposition 11.3. If g(X, Y ) is a real-valued function of the two discrete random

variables X and Y then the expectation of g(X, Y ) is obtained as
XX
E( g(X, Y ) ) = g(xk , y` )P(X = xk , Y = y` )
k `

where the sum ranges over all possible values xk , y` of the two random variables.

Remarks:

• Again we implicitly assume (in this course) that such expectations are well
defined, even when they involve infinite sums.

• Setting g(X, Y ) = 1 recovers the result that the values of the joint p.m.f.
sum to one; setting g(X, Y ) = Y gives an expression for E(Y ), and so on.
11.2 Expectations in the Multivariate Context 97

Example 11.5: Easy expectation?

Find E(U V + U ) if the random variables U and V have joint p.m.f. given by the table:

H U
HH
1 2
V HH
H .
1 1/2 1/6
3 1/3 0

Solution:
From Proposition 11.3, the required expectation can be written as a sum of four terms:

1 1 1
E(U V + U ) = (1 × 1 + 1) × + (2 × 1 + 2) × + (1 × 3 + 1) × + (2 × 3 + 2) × 0
2  6 3
1 1 1
=2× +4× + .
2 6 3
= 3.

Proposition 11.3 leads to the following important, if unsurprising, theorem.

Theorem 11.4. If X and Y are discrete random variables then

E(X + Y ) = E(X) + E(Y ).

Proof:
Starting from Proposition 11.3, and using series properties, we have
XX
E(X + Y ) = (xk + y` )P(X = xk , Y = y` )
k `
XX XX
= xk P(X = xk , Y = y` ) + y` P(X = xk , Y = y` )
k ` k `
! !
X X X X
= xk P(X = xk , Y = y` ) + y` P(X = xk , Y = y` )
k ` ` k
X X
= xk P(X = xk ) + y` P(Y = y` ) [from Proposition 11.2]
k `

= E(X) + E(Y ) [from Definition 9.1].

We can apply Theorem 11.4 repeatedly to obtain, for example,

E(X + Y + Z) = E(X + Y ) + E(Z) = E(X) + E(Y ) + E(Z).

In fact, using the properties of expectations (Proposition 9.7) one can show a useful
general result.
11.3 Independence for Random Variables 98

Corollary 11.5 (Linearity of expectation). If X1 , X2 , . . . , Xn are discrete random

variables and c1 , c2 , . . . , cn real-valued constants, then

E(c1 X1 + c2 X2 + · · · + cn Xn ) = c1 E(X1 ) + c2 E(X2 ) + · · · + cn E(Xn ).

Remark: This is a very general statement with no assumptions needed on X1 ,

X2 , etc.; it includes the case where the random variables are related in some way,
e.g., we could have X2 = (X1 + 3)2 .

Exercise 11.6: Ball expectations

For the set-up of Example 11.1, find E(R + Y ) and E(RY ) using Proposition 11.3. Check
that E(R + Y ) = E(R) + E(Y ). Is it also true that E(RY ) = E(R) × E(Y )?

11.3 Independence for Random Variables

We now turn to the question of independence for random variables. This builds
on the idea of independence for events.

Definition 11.6. Two discrete random variables X and Y are independent if

the events “X = xk ” and “Y = y` ” are independent for all possible values xk , y` ,
i.e., if
P(X = xk , Y = y` ) = P(X = xk )P(Y = y` )

for all xk and y` .

This generalizes to more than two random variables; X1 , X2 , . . . , Xn are inde-
pendent if their joint probability mass function factorizes into the product of the
marginal probability mass functions for all possible values of the random variables.

Exercise 11.7: *Three independent random variables

If the discrete random variables X, Y , and Z are independent then

P(X = xk , Y = y` , Z = zm ) = P(X = xk )P(Y = y` )P(Z = zm )

for all xk , y` and zm . Show that this implies

P(Y = y` , Z = zm ) = P(Y = y` )P(Z = zm )

for all y` and zm . [In this sense, the definition of independence for three random variables
is simpler than the definition of (mutual) independence for three events.]

Example 11.8: Easy independence?

Determine whether U and V in Example 11.5 are independent.
11.3 Independence for Random Variables 99

Solution:
We clearly have P(U = 2) = 1/6 and P(V = 3) = 1/3 but P(U = 2, V = 3) = 0 so
6 P(U = 2)P(V = 3) and hence U and V are not independent.
P(U = 2, V = 3) =

Independence of random variables has important consequences.

Theorem 11.7. If X and Y are independent discrete random variables then:
(a) E(XY ) = E(X)E(Y ),

(b) Var(X + Y ) = Var(X) + Var(Y ).

Proof:
(a) The proof relies on the fact that
! !
XX X X
ak b` = ak b` ,
k ` k `

which can be easily checked for sums over small numbers of values. Using this,
we have
XX
E(XY ) = xk y` P(X = xk , Y = y` )
k `
XX
= xk y` P(X = xk )P(Y = y` ) [by independence]
k `
! !
X X
= xk P(X = xk ) y` P(Y = y` )
k `

= E(X)E(Y ).

(b) Starting from Proposition 9.6, and employing Theorem 11.4/Corollary 11.5,
we have

Var(X + Y ) = E( (X + Y )2 ) − [E(X + Y )]2

= E(X 2 + 2XY + Y 2 ) − [E(X) + E(Y )]2
= E(X 2 ) + 2E(XY ) + E(Y 2 ) − [E(X)]2 − 2E(X)E(Y ) − [E(Y )]2
= E(X 2 ) − [E(X)]2 + E(Y 2 ) − [E(Y )]2 + 2 [E(XY ) − E(X)E(Y )]
= Var(X) + Var(Y ) [by part (a)].

Theorem 11.7 generalizes to three and more random variables. For instance, if
X, Y and Z are independent random variables, then,

Var(X + Y + Z) = Var(X) + Var(Y + Z) = Var(X) + Var(Y ) + Var(Z).

11.4 Binomial Distribution Revisited 100

Indeed, using Theorem 11.7 repeatedly together with properties of the variance
(Proposition 9.8), one arrives at the following corollary.

Corollary 11.8. If X1 , X2 , . . . , Xn are independent discrete random variables and

c1 , c2 , . . . , cn real-valued constants, then

Var(c1 X1 + c2 X2 + · · · + cn Xn ) = c21 Var(X1 ) + c22 Var(X2 ) + · · · + c2n Var(Xn ).

Remarks:

• Note that while Corollary 11.5 applies for all random variables, Corollary 11.8
applies for independent random variables.

• Independence of X and Y implies E(XY ) = E(X)E(Y ) but the converse

does not hold.

Exercise 11.9: *Converse counterexample

Find an example where E(XY ) = E(X)E(Y ) but X and Y are not independent.

Exercise 11.10: Ball independence

Determine whether the random variables R and Y of Example 11.1 are independent.

Exercise 11.11: Five dice

You roll five fair six-sided dice. Let X denote the sum of the numbers shown. Compute
the expectation and the variance of X.

11.4 Binomial Distribution Revisited

We now demonstrate how the results of this chapter can be leveraged to re-derive
the expressions for the expectation and variance of the binomial distribution which
we saw in Section 10.2. We start by considering n Bernoulli trials each with
probability p of success. We count the number of successes in each of these trials
with the random variables X1 , X2 , . . . , Xn ; the event “Xk = 1” corresponds to
success in the kth trial while “Xk = 0” corresponds to failure in the kth trial. Since
the trials are identical, we obviously have E(Xk ) = p and Var(Xk ) = p(1 − p) for
k = 1, 2, . . . , n.
Denoting the total number of successes by the random variable X, we have

X = X1 + X2 + · · · + Xn
11.5 Further Exercises 101

and, by Corollary 11.5,

E(X) = E(X1 + X2 + · · · + Xn )
= E(X1 ) + E(X2 ) + · · · + E(Xn )
= np.

This conclusion does not require the trials to be independent. However, if they
are independent, we can also employ Corollary 11.8 to obtain

Var(X) = Var(X1 + X2 + · · · + Xn )
= Var(X1 ) + Var(X2 ) + · · · + Var(Xn )
= np(1 − p).

Hence we easily recover (10.5) and (10.6) for the expectation and variance of the
number of successes in n independent Bernoulli trials, i.e., the expectation and
variance of the binomial distribution.

11.5 Further Exercises

Exercise 11.12: Expectations and variances
Suppose that X, Y , and Z are random variables with X ∼ Bin(7, 1/6), Y ∼ Geom(1/2),
and Z ∼ Poisson(6). Suppose further that X and Y are independent but that X and Z
are not independent. Which of the following can be determined from this information?
Find the value of those which can be determined.

(a) E(X + Y ),

(b) E(X + Z),

(c) E(X + 2Y + 3Z),

(d) E(X 2 + Y 2 + Z 2 ),

(e) Var(X + Y ),

(f) Var(X + Z),

(g) Var(X + 2Y + 3Z).

[Answers (in jumbled order): cannot be determined, 151/3, 19/6, 107/36, 43/6, cannot be
determined, 139/6 ]

Exercise 11.13: Ones and twos

Two fair six-sided dice are rolled. Let V be the number of “one”s seen in the outcome
and W be the number of “two”s seen. Find the joint distribution of V and W and the
two marginal distributions. Are V and W independent random variables?
11.5 Further Exercises 102

Exercise 11.14: *Joint deductions

Let X and Y be discrete random variables with probability mass functions given by

xk 0 1
,
P(X = xk ) 1/2 1/2

and
y` 0 1 2
.
P(Y = y` ) 1/3 1/3 1/3
Furthermore assume that

P(X = 0, Y = 0) = P(X = 1, Y = 2) = 0.

Find the joint probability mass function of X and Y .

Chapter 12

Covariance and Conditional Expectation

12.1 Covariance and Correlation

We know from the last chapter that if X and Y are independent then we always
have E(XY ) = E(X)E(Y ), while if X and Y are not independent we usually (but
not always) have E(XY ) 6= E(X)E(Y ). You may wonder if we can say any more
about the strength and type of relationship between two random variables. One
way to do this is by defining two new quantities.1

Definition 12.1. Let X and Y be discrete random variables defined on the same
sample space. The covariance of X and Y is defined by

Cov(X, Y ) = E( [X − E(X)][Y − E(Y )] ).

If Var(X) > 0 and Var(Y ) > 0, then the correlation coefficient of X and Y
is defined by
Cov(X, Y )
Corr(X, Y ) = p .
Var(X)Var(Y )
We shall consider how to interpret these quantities shortly. However, we can
immediately see by comparison with Definition 9.5, that Cov(X, X) = Var(X).
Just as there is an alternative formula for the variance (Proposition 9.6), there
is an alternative formula for the covariance which, in practice, is often easier for
calculations.

Proposition 12.2. If X and Y are discrete random variables defined on the same
sample space, then
Cov(X, Y ) = E(XY ) − E(X)E(Y ).
1
As in previous chapters, we assume here that all expectations are finite, even if they involve
infinite sums.

103
12.1 Covariance and Correlation 104

Proof:
From Definition 12.1, we have

Cov(X, Y ) = E( [X − E(X)][Y − E(Y )] )

= E( XY − XE(Y ) − Y E(X) + E(X)E(Y ) )
= E(XY ) − E(X)E(Y ) − E(X)E(Y ) + E(X)E(Y )
= E(XY ) − E(X)E(Y ),

where we use Proposition 9.7 / Corollary 11.5 in going from the second to the
third line.

Remarks:

• When Cov(X, Y ) = 0, we say that X and Y are uncorrelated. If X and

Y are independent then, from Theorem 11.7(a) and Proposition 12.2, we
always have Cov(X, Y ) = 0. Conversely, however, Cov(X, Y ) = 0 does not
necessarily mean X and Y are independent.

• Cov(X, Y ) > 0 if, on average, [X − E(X)] and [Y − E(Y )] have the same
sign. In other words, when [X − E(X)] is positive, [Y − E(Y )] tends to be
positive too; when [X − E(X)] is negative, [Y − E(Y )] tends to be negative
too. Loosely we can say in this case that X and Y tend to deviate together
above or below their expectation. An example of such positive correlation
might be the number of hours of revision and the final exam score.

• Cov(X, Y ) < 0 if, on average, [X − E(X)] and [Y − E(Y )] have opposite

signs. Loosely, we can say in this case that X and Y tend to deviate in
opposite directions from their expectations. An example of such negative
correlation might be temperature (recorded to the nearest degree, say) and
sales of hot chocolate.

• Correlation does not necessarily indicate causation. An oft-discussed exam-

ple in this respect is the positive correlation between the number of storks and
the number of babies in Germany [Sie88]. The “Spurious Correlations” web-
site and accompanying book [Vig15] also contain many entertaining graphs
(yes, really!) and bizarre facts.

Example 12.1: Correlated balls

Find the covariance of the random variables R and Y of Example 11.1. Discuss the
interpretation of the sign of the covariance in this case.
12.2 Conditional Expectation 105

Solution:
From Proposition 12.2, we have

Cov(R, Y ) = E(RY ) − E(R)E(Y )

6 9 6
= − × [see Exercises 11.4 and 11.6]
7 7 7
12
=− .
49
R and Y are negatively correlated so having more red balls means having fewer yellow
ball on average, and vice versa.

The correlation coefficient is a kind of scaled version of the covariance which

is insensitive, for example, to the choice of units in a measurement. In particular,
we have the following nice properties.

Proposition 12.3. Let X and Y be discrete random variables defined on the same
sample space and with Var(X) > 0 and Var(Y ) > 0.

(a) If a, b, c, d are real-valued constants with a > 0 and c > 0, then

Corr(aX + b, cY + d) = Corr(X, Y ).

(b) Corr(X, Y ) lies in [−1, 1], i.e., −1 ≤ Corr(X, Y ) ≤ 1.

Exercise 12.2: Proving properties

Suppose that X and Y are discrete random variables defined on the same sample space
(with Var(X) > 0 and Var(Y ) > 0) and that a, b, c, d are real-valued constants.

(a) Show that

Cov(aX + b, cY + d) = acCov(X, Y ).

(b) If a > 0 and c > 0, use the result of part (a) to show that

Corr(aX + b, cY + d) = Corr(X, Y ).

(c) Show that if Y = aX+b with a > 0, then Corr(X, Y ) = 1 (perfect positive correlation).
What relationship between X and Y would give Corr(X, Y ) = −1 (perfect negative
correlation)?

12.2 Conditional Expectation

Example 12.3: Conditional balls
Consider again the experiment of Example 11.1. Assuming that we pick one yellow ball
(i.e., that Y = 1), compute the probability that we pick r red balls for r = 0, 1, 2, 3.
12.2 Conditional Expectation 106

Solution:
By Definition 4.1 for conditional probability, we have

P( (R = r) ∩ (Y = 1) ) P(R = r, Y = 1)
P(R = r|Y = 1) = = .
P(Y = 1) P(Y = 1)

Hence the values of P(R = r|Y = 1) for different r can be obtained by dividing the values
in the second row of the joint probability mass function (Exercise 11.2) by the marginal
P(Y = 1). We can calculate P(R = 1|Y = 1), for example, as

P(R = 1, Y = 1)
P(R = 1|Y = 1) =
P(Y = 1)
12/35
= [see Exercise 11.4]
4/7
3
= .
5
The resulting conditional p.m.f. is given by the following table:
r 0 1 2 3
.
P(R = r|Y = 1) 1/10 3/5 3/10 0

The above example illustrates the general idea of conditioning an event involv-
ing a random variable on another event.
Definition 12.4. Let X be a discrete random variable and A be an event with
P(A) > 0. The conditional probability

P( (X = xk ) ∩ A)
P(X = xk |A) =
P(A)

defines the probability mass function of X given A. The corresponding ex-

pectation
X
E(X|A) = xk P(X = xk |A)
k

is called the conditional expectation.

Remarks:
• One may consider X|A as a random variable in its own right: “the random
variable X given event A”.

• The event A being conditioned on is often an event involving another random

variable, e.g., the event “Y = 1” in Example 12.3.

• If X and Y are independent random variables, then P(X = xk |Y = yl ) =

P(X = xk ) and we always have E(X|Y = yl ) = E(X). [If X and Y are not
independent, then we usually have E(X|Y = yl ) 6= E(X).]
12.3 Law of Total Probability for Expectations 107

Example 12.4: Conditional expectation of balls

Consider once again the experiment of Example 11.1. Assuming that we pick one yellow
ball, determine the expectation of the number of red balls.

Solution:
The conditional expectation E(R|Y = 1) is easily calculated from the table of the condi-
tional p.m.f. in Example 12.3:

3
X
E(R|Y = 1) = rP(R = r|Y = 1)
r=0
1 3 3
=0× +1× +2× +3×0
10 5 10
6
= .
5
Note that E(R|Y = 1) < E(R) = 9/7. This makes sense since, if we pick one yellow ball,
we have more than the expected number of yellow balls [E(Y ) = 6/7] so the expected
number of red balls is reduced. [R and Y are not independent, see Exercise 11.10.]

Exercise 12.5: Geometric and binomial

Let X be a random variable with X ∼ Geom(1/3). Let Y be another random variable
with Y ∼ Bin(n, 1/4) where n is the value taken by the random variable X.
(a) State the expectation of the random variable X.
(b) Assuming X takes the value n, compute the conditional expectation of the random
variable Y . [based on 2018 exam question (part)]

12.3 Law of Total Probability for Expectations

Recall that Theorem 6.2 enables us to write (total) probabilities in terms of con-
ditional probabilities. Analogously, the final theorem of our course enables us to
write (total) expectations in terms of conditional expectations.

Theorem 12.5. Suppose that E1 , E2 , . . . , En partition S with P(Ei ) > 0 for i =

1, 2, . . . , n. Then for any discrete random variable X we have
n
X
E(X) = E(X|Ei )P(Ei ).
i=1

Proof:
Using the law of total probability (Theorem 6.2) for the event “X = xk ” with
partition formed by E1 , E2 , . . . , En , we have
n
X
P(X = xk ) = P(X = xk |Ei )P(Ei ).
i=1
12.3 Law of Total Probability for Expectations 108

Then, using the definition of expectation (Definition 9.1), we get

X
E(X) = xk P(X = xk )
k
X n
X
= xk P(X = xk |Ei )P(Ei )
k i=1
n X
X
= xk P(X = xk |Ei )P(Ei )
i=1 k
Xn X
= P(Ei ) xk P(X = xk |Ei )
i=1 k
Xn
= P(Ei )E(X|Ei ) [using Definition 12.4].
i=1

Example 12.6: Last throw of the dice

We toss a fair coin. If the coin comes up Heads we roll a fair six-sided die. If it comes up
Tails we roll a fair four-sided die (with sides numbered 1, 2, 3, 4). Let X be the number
we roll. Find E(X).

Solution:
Let H be the event “the coin shows Heads”. Clearly {H, H c } is a partition of S and, since
the coin is fair, P(H) = P(H c ) = 1/2.
Now, the random variable X|H is the number shown by the die given that the coin
shows Heads, i.e., the number shown given we roll the six-sided die. Clearly its expectation
is
1+2+3+4+5+6 7
E(X|H) = = .
6 2
Similarly, X|H c is the number shown by the die given that the coin shows Tails, i.e., the
number shown given we roll the four-sided die, and so

1+2+3+4 5
E(X|H c ) = = .
4 2
Finally, Theorem 12.5 with the partition {H, H c } gives

7 1 5 1
E(X) = E(X|H)P(H) + E(X|H c )P(H c ) = × + × = 3.
2 2 2 2

Exercise 12.7: *Geometric and binomial (continued)

Continue Exercise 12.5 with the following.

(c) Using the law of total probability, or otherwise, compute the expectation of Y .

(d) Using the law of total probability, or otherwise, compute the expectation of the prod-
uct XY and hence compute the covariance of the random variables X and Y .
[based on 2018 exam question (part)]
12.4 Further Exercises 109

12.4 Further Exercises

Exercise 12.8: Variance and covariance
Suppose that X and Y are discrete random variables defined on the same sample space.
Show that
Var(X + Y ) = Var(X) + Var(Y ) + 2Cov(X, Y ).

Exercise 12.9: *An odd question

A fair six-sided die is rolled twice. Let X be the number showing on the first roll and A
be the event “at least one odd number is rolled”. Find the probability mass function of
X|A and hence calculate E(X|A). [Answer: 10/3]

Exercise 12.10: *A many-headed puzzle

A fair six-sided die is rolled. Let N be the number showing on the die. Now a fair coin is
tossed N times. Let X be the number of Heads seen. Find the conditional distribution of
X given N = n and hence find the expectation of X. [Answer: 7/4]

Exercise 12.11: *Coins and dice again

You have two fair six-sided dice and two coins, one of which is fair while the other always
shows Tails. First you roll the two dice, then you pick a coin at random and toss it. If
it shows Heads, your score is the product of the numbers on the two dice; if it shows
Tails, your score is the sum of the two numbers. Determine the expectation of your score.
[Answer: 133/16]

Exercise 12.12: **Random walk

Consider a “random walker” on an infinite one-dimensional lattice (with sites labelled by
integers). Let Xt denote the position of the walker at discrete time t where t = 0, 1, 2, 3, . . ..
At time t = 0 the walker is at site 0, i.e., X0 = 0. At each subsequent time step
the walker moves at random to one of the two neighbouring sites, i.e., if Xt−1 = ` then
Xt = `+1 with probability 1/2 and Xt = `−1 with probability 1/2. In terms of conditional
probabilities, this means P(Xt = ` + 1|Xt−1 = `) = 1/2, P(Xt = ` − 1|Xt−1 = `) = 1/2,
and P(Xt = k|Xt−1 = `) = 0 if k 6= ` ± 1.

(a) Use Theorem 12.5 to show that E(Xt ) = 0 for all t.

(b) Show further that Var(Xt ) = t for all t.

Chapter 13

Epilogue (and Exams etc.)

13.1 Tips for Revision and Exams

There is no silver bullet for exam success and effective revision will depend on
your own learning style. However, the earlier tips for reading the lecture notes
(see Section 7.2) should apply particularly as you revise – understanding is key,
especially as the results in later parts of the course continue to build on the earlier
ones. Similarly, for tackling questions in the exam itself, the tips for doing examples
and exercises (see Section 7.3) are relevant. In addition, there are some specific
elements of common sense and exam strategy, including those below.

• Read the question. Take a few moments to be clear about what is being asked
and, if necessary, work out how to convert the words to symbols. Make sure
you transcribe any relevant material correctly; if you copy something down
incorrectly you may even make a question impossible to solve!

• Show that you understand what you are doing. In a written question, you
must clearly show all your working except where you are simply asked to
“state” or “write down” a result. If you give a correct answer without jus-
tification, you may only be awarded partial marks. If you give an incorrect
answer without justification, you are likely to get zero marks! However, if
you give an incorrect answer but show signs of a correct method, you will
earn partial marks.

• Use your time wisely. Don’t spend ages struggling with a question (or ques-
tion part) worth only a few marks, when you know you can do something
else. Once the easier marks are safely “in the bag”, you can always come
back to the harder ones.

110
13.2 Probability in Perspective 111

• Check your answers. Try to avoid losing marks for “silly mistakes”! Don’t
just assume that you know how to do something and have done it correctly.
Think about whether each answer makes sense (e.g., is a calculated proba-
bility plausible) and if there is another way you could check it.

13.2 Probability in Perspective

I hope that you have enjoyed this course and seen something of the beautiful
structure of probability. The focus naturally turns to exams at this point but
there should also be a rather broader perspective. The aim of this module is not
merely to prepare you for a specific exam, nor even to prepare you for future
exams in courses which build on this one, but to give you tools to analyse prob-
abilistic scenarios and tackle unfamiliar problems wherever they may arise. This
could ultimately be in research-level mathematics or in a more applied career (e.g.,
those involving finance, medical statistics, insurance, weather prediction, etc.) but
just as importantly it could be in the decisions or discussions of everyday life –
probability matters!
Bibliography

[ASV18] David F. Anderson, Timo Seppäläinen, and Benedek Valkó, Introduction

to Probability, Cambridge University Press, Cambridge, 2018.

[Bay63] Thomas Bayes, An essay towards solving a problem in the doctrine of

chances. by the late Rev. Mr. Bayes, F. R. S. communicated by Mr.
Price, in a letter to John Canton, A. M. F. R. S., Phil. Trans. R. Soc.
53 (1763), 370–418.

[Ros20] Sheldon Ross, A First Course in Probability, 10th ed., Pearson Education
Limited, Harlow, 2020.

[Rot64] Gian-Carlo Rota, On the foundations of combinatorial theory i. theory

of möbius functions, Zeitschrift für Wahrscheinlichkeitstheorie und Ver-
wandte Gebiete 2 (1964), 340—-368.

[Sie88] Helmut Sies, A new parameter for sex education, Nature 332 (1988), 495.

[Tij12] Henk Tijms, Understanding probability, 3rd ed., Cambridge University

Press, New York, 2012.

[Vig15] Tyler Vigen, Spurious correlations, Hatchette Books, USA, 2015.

112
Appendix A

Errata

This appendix lists the points in these notes where there are non-trivial correc-
tions/clarifications from earlier released versions.

• Exercise 5.5 corrected to read “Look back at Example 4.4.”.

• Exercise 6.4 corrected to read “Show that the answer to Example 6.3(b) ...”.

• Exercise 8.3(b) corrected to read “Let Z be another random variable ...”.

• Solution to Example 8.9 corrected to read “For the p.m.f. of T ...”.

• Solution to Example 9.1 corrected to resolve more confusion with random

variables.

• Several embarassing typos in the solution to Example 9.7(b) also fixed.

• Explanatory note on the second-to-last line of the calculation leading to (10.13)

corrected to “[setting ` = k − 2]”.

• First remark below Corollary 11.8 clarified.

• Explicit reference to Corollary 11.5 added in proof of Proposition 12.2.

113