Chapter 3: Elements Of Probability

ST2334 Probability and Statistics1

(Academic Year 2014/15, Semester 1)

Department of Statistics & Applied Probability (DSAP)

National University of Singapore (NUS)1

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 1 / 57
 Outline

1 Introduction

2 Sample Space And Events

3 Venn Diagrams And The Algebra Of Events

4 Axioms Of Probability

5 Sample Spaces Having Equally Likely Outcomes

6 Conditional Probability

7 Bayes’ Formula

8 Independence

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 2 / 57
 Introduction I

Learning Outcomes

Questions to Address: © Understanding algebra of events © How

to compute/interpret probability © Various properties of probability © How
to count © Difference between combination & permutation © What
conditional probability is © What independence is
Concept & Terminology: © experiment © sample space © set
theory © event © complement/union/intersection © null/mutually
exclusive/disjoint/exhaustive events © Venn diagram
© commutative/associative/distributive/DeMorgan’s laws © axioms of
probability © addition/multiplication law © equiprobable outcomes
© basic principle of counting © permutation © combination © binomial
coefficient © conditional probability © law of total probability © tree diagram
© Bayes’ formula © independence © pairwise/mutual independence

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 3 / 57
 Introduction II

Mandatory Reading
/ Section 3.1 – Section 3.8

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 4 / 57
 Sample Spaces

Definition
An experiment is any action or process
whose outcome is subject to
uncertainty/randomness

When an experiment is performed/conducted, ONLY one of all

possible outcomes would occur or would be observed/realized
Assume that the set/collection of all possible outcomes is known

Definition
The sample space of an experiment, denoted by S, is the set/collection of
all possible outcomes of that experiment. Usually, we denote an
outcome/element of S by ω

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 5 / 57
 Example 1: Sample Spaces
The simplest experiment to which probability (prob) applies is one with 2
possible outcomes, e.g.,
1 When we flip/toss a coin & see which side faces up, S = {H , T },
where H & T denote head facing up & tail facing up, respectively
2 In an experiment of examining a light bulb to see if it is defective,
S={ , }

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 6 / 57
 Example 2: Sample Spaces
1 When we roll a die & see the upturned #, the sample space is
S = {1, 2, . . . , 6}

2 If we examine 3 light bulbs in sequence & note the result of each

examination, then an outcome for this experiment is any sequence of
’s & ’s of length 3, so

S={ , , , , , , , }
3 The amount of time between successive customers arriving at a
check–out counter is of interest. For such an experiment,

S = {t : t ≥ 0}
Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 7 / 57
 Events
To develop a mathematical framework for understanding an
experiment & studying prob, set theory is naturally adopted in the
mathematical theory of prob
Interested in not only the individual outcomes of S but also various
collections of outcomes from S
Definition
An event A is any collection of outcomes contained in the sample
space (i.e., any subset of S written as A ⊂ S). An event is simple if it
consists of exactly 1 outcome & compound if it consists of > 1 outcome

When an experiment is performed & an outcome ω ∈ S is

observed/realized:
An event A is said to occur if the observed outcome ω is included in
A (i.e., ω ∈ A )
Exactly 1 simple event occurs, but many compound events occur
simultaneously

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 8 / 57
 Example 3: Events
Toss 2 coins: S = {HH , HT , TH , TT }
The event that 2 heads are observed, A = {HH }, is a simple event as Ê
A ⊂ S & Ë A contains only 1 outcome
The event that 1 head is observed, B = {HT , TH }, is a compound event
Examine 3 fuses: Different compound events include, e.g.,

A ={ , , }
= the event that exactly 1 of the 3 fuses is not defective
B={ , , , }
= the event that at most 1 fuse is defective
C={ , }
= the event that all 3 fuses are in the same condition

Suppose that the experiment is performed, & the observed outcome is

. Then, the simple event { } has occurred & so have the
compound events B & C (but not A )

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 9 / 57
 Algebra of Events I

As an event is just a set, so set operations from elementary

set theory carry over directly into prob theory, which allows us to
create new & more “complex” events from given events

Definition
Given any event E , F ⊂ S,
1 The complement of E, denoted by E c , A , or E 0 , is the set of all outcomes in
S that are not contained in E
2 The union of A & B, denoted by E ∪ F & read “E or F”, is the event
consisting of all outcomes that are either in A or B or in both events
3 The intersection of E & F, denoted by E ∩ F (or, simply EF) & read “E & F”,
is the event consisting of all outcomes that are both in E & F

Commutative laws: E ∪ F = F ∪ E & E ∩ F = F ∩ E

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 10 / 57
 Algebra of Events II

Definition
Let ∅ denote the null event, the event consisting of no outcomes
whatsoever

Definition
When E ∩ F = ∅, E & F are said to be mutually exclusive or disjoint
events. It follows that E & E c must be disjoint for any event E ⊂ S

Definition
When E ∪ F = S, E & F are said to be exhaustive events. It follows that E
& E c must be exhaustive

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 11 / 57
 Venn Diagrams I
Venn diagrams: a useful tool for visualizing set operations

Note : Events can be represented by objects of any shape

Complement: E c is the shaded area

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 12 / 57
 Venn Diagrams II
Union: E ∪ F is the shaded area

Intersection: E ∩ F is the shaded area

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 13 / 57
 Venn Diagrams III
E & F are mutually exclusive or disjoint events:

E & E c (represented by the shaded area) are mutually exclusive &

exhaustive events:

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 14 / 57
 Example 4: Venn Diagrams

Toss 2 coins: S = {HH , HT , TH , TT }; F = {HH } is the right circle;

E = {HT , TH } is the left circle

How about C = {TT }?

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 15 / 57
 Example 5: Venn Diagrams & Algebra of Events

Assume that there are 7 possible outcomes in an experiment such

that (s.t.)
S = {0, 1, 2, 3, 4, 5, 6}
Let A = {0, 1, 2, 3, 4}, B = {3, 4, 5, 6} & C = {1, 2}. Then,

Ac = {5, 6}
A ∪B = {0, 1, 2, 3, 4, 5, 6} = S (i.e., A & B are exhaustive )
A ∪C = {0, 1, 2, 3, 4} = A (indeed, C is a subset of A )
A ∩B = {3, 4}
A ∩C = {1, 2}
(A ∩ C ) c
= {0, 3, 4, 5, 6}
B ∩C = ∅ (i.e., B & C are disjoint )

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 16 / 57
 Extensions to ≥ 3 Events

The operations of union & intersection can be extended to ≥ 3 events,

so can be the idea of disjointness & exhaustiveness
For any 3 events, A , B , C ⊂ S,
A ∪ B ∪ C is the event consisting of all outcomes that are in at least 1
of the 3 events
A ∩ B ∩ C is the event consisting of all outcomes that are in all the 3
events
A , B & C are said to be mutually exclusive or pairwise disjoint if no 2
events have any outcomes in common
A , B & C are said to be exhaustive if the event A ∪ B ∪ C consists of all
outcomes in S

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 17 / 57
 Algebra For ≥ 3 Events

Some Useful Laws

Given any event A , B , C , E1 , . . . , En ⊂ S,
1 Associative Laws:
(A ∪ B ) ∪ C = A ∪ (B ∪ C ) (A ∩ B ) ∩ C = A ∩ (B ∩ C )

2 Distributive Laws:
(A ∪ B ) ∩ C = (A ∩ C ) ∪ (B ∩ C ) (A ∩ B ) ∪ C = (A ∪ C ) ∩ (B ∪ C )

3 DeMorgan’s laws:
 n c n
 n c n
[  \ \  [
 Ei  = Eic  Ei  = Eic
i =1 i =1 i =1 i =1

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 18 / 57
 Axioms of Probability

Definition
Probability of an event E is defined by a function P from subsets of S to
the real numbers that satisfies the following axioms of probability
Ê If E ⊂ S, then P (E ) ≥ 0
Ë P (S ) = 1
Ì For any countably many sequence of disjoint events
E1 , E2 , . . . , En , . . ., ∞ 
[  X ∞
P  Ei  = P (Ei )
i =1 i =1

The above axioms do not completely determine an assignment of

probabilities (probs) to events. They serve only to rule out
assignments inconsistent with our intuitive notions of prob

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 19 / 57
 Relative–Frequency Interpretation of Probability

Definition of Probability Based on Relative Frequency

Prob of any event E can be defined in terms of its relative frequency, i.e.,
when the same experiment is repeatedly performed for n times under
exactly the same conditions, then

# of times that E occurs

P (E ) = lim
n→∞ n

P (E ) is defined as the “limiting” proportion of time that E occurs, or

the “limiting” relative frequency of E
Intuitively pleasing & theoretically correct
Serves as an interpretation of any prob

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 20 / 57
 Some More Properties Regarding Probability

Some Simple Propositions Regarding Probabilities

Given any event A , B ⊂ S,
1 P (∅) = 0 (i.e., prob that there is no outcome at all is zero)
2 P (A c ) = 1 − P (A )
3 P (A ) ≤ 1
4 If A ⊂ B, then P (A ) ≤ P (B )
5 Addition Law:

P (A ∪ B ) = P (A ) + P (B ) − P (AB )

The addition law serves as the general formula to compute prob of an

event of interest which is expressible as a union of events via
decomposing the event of interest into “smaller” events

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 21 / 57
 Example 6: Addition Law

In an undergraduate module, 60% of all students have statistics

background, 80% have calculus background, & 50% of all students have
both. If a student is selected at random, what is the prob that s/he has
background in Ê at least 1 subject & Ë exactly 1 subject?
(
A = {a selected student has statistics background}
Solution: Let
B = {a selected has calculus background}
We are given: P (A ) = .6, P (B ) = .8, & P (AB ) = .5

P (has background in at least 1 subject)

= P (A ∪ B ) = P (A ) + P (B ) − P (AB )
= .6 + .8 − .5 = .9
P (has background in exactly 1 subject)
= P (AB c ) + P (BA c ) = P (A ∪ B ) − P (AB )
= .9 − .5 = .4

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 22 / 57
 How to Compute Probabilities?

Depends on the nature of the sample space S: finite versus infinite

When the sample space is finite (i.e., S = {ω1 , ω2 , . . . , ωN }):
let
P ({ωi }) = pi , i = 1, 2, . . . , N
where N ≥ 1, called the cardinality of S, is a finite positive (+ve) integer
denoting the total # of outcomes in S/of the experiment
computing P (A ) is straightforward due to Axiom Ì of prob:

 
 [  X X
P (A ) = P  {ωi } = P ({ωi }) = pi
ω i ∈A ω i ∈A {i =1,...,N :ωi ∈A }
(1)

Simply add probs of all the ωi that are included in A !

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 23 / 57
 Sample Spaces Having Equally Likely Outcomes

Assume that
1 the sample space is finite (e.g., S = {1, 2, . . . , N })
2 all the N outcomes are equiprobable or equally likely to occur (a
reasonable attribute in many experiments)

Computing Probabilities For Sample Spaces Having Equally Likely

Outcomes
For an experiment satisfying the above 2 assumptions, the prob of any
event A is
# of outcomes in A
P (A ) =
# of outcomes in S

Simply count the total #’s of outcomes in A & in S!

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 24 / 57
 Example 7: Tossing 2 Fair Coins
S = {HH , HT , TH , TT } with cardinality N = 4
Let A denote the event that at least 1 head is observed
As A = {HH , HT , TH } = {HH } ∪ {HT } ∪ {TH }

P (A ) = P ({HH }) + P ({HT }) + P ({TH })

It remains to find the 3 individual probs

Furthermore, if the coin is fair (i.e., equally likely to observe head or
tail in any toss), all the 4 possible outcomes should be equally likely to
occur &
1
P ({HH }) = P ({HT }) = P ({TH }) = P ({TT }) =
4
Then,
# of outcomes in A 3
P (A ) = =
# of outcomes in S 4
Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 25 / 57
 Counting Methods

In reality, many experiments may be associated with a finite sample

space, but the cardinality N could be quite large
Understanding S (which involves listing all the outcomes S) is not an
easy task; sometimes, even obtaining N, the cardinality of S, can be
prohibitive
Even when the outcomes are equally likely, obtaining the cardinality
of A can also be prohibitive
Several counting methods or systematic ways for enumeration will be
introduced
Especially useful in computing probs when the sample space is finite
Some of the ideas can be borrowed or generalized to experiments with
countably infinite or uncountably infinite sample spaces

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 26 / 57
 Basic Principle of Counting

To deal with “complex” experiments defined as a sequence of several

“simpler” experiments

Basic Principle of Counting & Its Generalization

1 If 1 experiment has m > 0 outcomes & another experiment has n > 0
outcomes, then there are m × n possible outcomes for the 2
experiments
2 If there are p > 2 experiments & the first experiment has n1 possible
outcomes, the second n2 , . . . , & the pth np possible outcomes, then
there are a total of n1 × n2 × · · · × np possible outcomes for the p
experiments

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 27 / 57
 Example 8: Basic Principle of Counting
Draw a card from a deck of playing cards: 2 experiments with
m = 4 & n = 13 outcomes (as a card is defined by 4 different suits,
«, ª, ¨, ©, & 13 different face values, A, K, Q, J, 10, 9, ... , 3, 2)
þ 4 × 13 = 52 possible outcomes

Singapore Sweep on first Wednesday of every month:

7 experiments with n1 = n2 = · · · = n7 = 10 outcomes
þ n1 × n2 × · · · × n7 = 107 = 10, 000, 000 possible outcomes
Examine m light bulbs: m experiments with n1 = n2 = · · · = nm = 2
outcomes ( or ) þ n1 × n2 × · · · × nm = 2m possible outcomes

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 28 / 57
 Permutations & Combinations
Very often, we would like to address how many ways there are to
select a subset of size r from a group of n distinct/distinguishable
objects {c1 , c2 , . . . , cn }
e.g., when n = 6 & r = 4:

It depends on
1 whether we are allowed to duplicate objects:
Sampling without replacement versus Sampling with replacement
2 whether the sequence/order from which the r objects are selected
matters or not

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 29 / 57
 Permutations

Permutation
A permutation is an ordered arrangement of objects. Selecting a sample
of size r (= 0, 1, . . . , n) from a set of size n objects, there are
1 nr permutations under sampling with replacement
n!
2
n Pr ≡ = n(n − 1)(n − 2) · · · (n − r + 1) permutations under
(n − r )!
sampling without replacement

n! is called n factorial
When n is a +ve integer, n! = n(n − 1)(n − 2) · · · 1. We have the
convention 0! = 1
n!
The total # of ways in arranging n distinct objects is = n!
(n − n)!

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 30 / 57
 Example 9: Permutations

Refer to the picture at Page 29, how many different ordered arrangements
of 4 M&M’s selected from 6 M&M’s of different colors are there?

Solution: Here, n = 6 distinct M&M’s/colors (group size) & r = 4

selected M&M’s/colors (subset size). Note that the order of the 4 colors
matters & the M&M’s cannot be duplicated (i.e., sampling without
replacement). The # of permutations is

6! 6!
= = 6(5)(4)(3) = 360
(6 − 4)! 2!

Remark : This is equivalent to the total # of ways to assign 1 out of 6

pieces of distinct M&M’s to each of 4 different kids

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 31 / 57
 Example 10: Permutations
There are 10 teaching assistants (TAs) available for grading papers in a
test. The test consists of 5 questions, & the Professor wishes to select a
different TA to grade each question (at most 1 question per assistant). In
how many ways can the TAs be chosen for grading?

Solution: Here, this question is interested in finding the # of

permutations with n = 10 distinct TAs (group size) & r = 5 selected
TAs (subset size). Note that the order of the 5 selected TAs matters (as
the 5 questions are different) & each TA cannot grade > 1 question (i.e.,
sampling without replacement). The # of permutations is
10! 10!
= = 10(9)(8)(7)(6) = 30, 240
(10 − 5)! 5!

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 32 / 57
 Example 11: Permutations
There are n rewards to be distributed randomly by a teacher to n students
so that each student gets 1 reward. Suppose that 1 of the rewards is the
top prize. The students will queue up to receive a reward from the teacher.
Shall a student queue first so as to increase his/her chance of getting the
top prize?

Solution: First of all, all possible assignments of n rewards are equally

likely to happen and there are n! # of ways/permutations to distribute the
rewards
Assume that a student queues at the i-th (i = 1, . . . , n) position in the
queue. Imagine that the n rewards also line up and will be distributed
one-by-one accordingly. Now, for this student to have the top prize, the i-th
reward should be at the i-th position. There are (n − 1)! # of ways to
distribute the other n − 1 rewards into the other n − 1 positions. Hence, the
required prob is
(n − 1)! 1
=
n! n
Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 33 / 57
 Combinations
Sometimes, we may be no longer interested in how the objects are
arranged or selected, but in the constituents of the subset. For
instance, the following permutations are identical:

Combination
A combination is an unordered arrangement/collection of objects. For a
set of n distinct
! objects & a subset of size r, there are
n n!
n Cr ≡ ≡ different combinations under sampling without
r (n − r )!r !
replacement when r = 0, 1, . . . , n
!
n
The expression, n Cr or , is read as “n choose r” & also called
r
binomial coefficient

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 34 / 57
 Example 12: Combinations

Refer to the picture at Page 29, how many different combinations of 4

M&M’s selected from 6 M&M’s of different colors are there?

Solution:

Here, n = 6 distinct M&M’s/colors (group size) & r = 4 selected

M&M’s/colors (subset size). The order of the 4 colors does not matter &
we cannot duplicate the M&M’s (i.e., sampling without replacement). The
# of combinations is
6! 6!
= = 15
(6 − 4)!4! 2!4!
which is << 360 (the # of permutations obtained at Page 31)

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 35 / 57
 Example 13: Bridge Hand
A bridge hand consists of any 13 cards selected from a 52–card deck
without regard to order, wherein the 52 cards are distinct
!
52 52!
There are = = 635, 013, 559, 600 different bridge hands
13 13!39!
Since there are 13 cards in each of the 4 suits («, ª, ¨, ©), the # of
! consisting entirely of « or ¨ (no ª or © cards) is
hands
26 26!
= = 10, 400, 600
13 13!13!
Suppose that a bridge hand is dealt from a well–shuffled deck (i.e.,
every bridge hand/combination of 13 cards from S of cardinality
≈ 635 billion is equally likely to occur). So,

P (A ) = P (obtain a hand consisting entirely of « or ¨)

26
cardinality of A 13
= = 52 = .0000164
cardinality of S
13

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 36 / 57
 Example 14: Three of a Kind
A 5–card poker hand is said to be a “three of a kind” if it
consists of 3 cards of the same denomination & 2 other cards
of 2 different denominations other than the one for the previous
3 cards. What is the prob that a person is dealt a “three of a
kind”?
52
Solution: Note that all 5 possible poker hands are equally likely (i.e.,
order from which the 5 cards are drawn are irrelevant)

1 # of combinations to choose
the 3 cards of the same denomination, &
1244
2
2 1 1 # of combinations to choose the other 2 cards
1341244
1 3 2 1 1
Prob of a “three of a kind” is 52 ≈ .02113
5

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 37 / 57
 Conditional Probability: A Motivating Example I

In the process of grade assignment to students in an undergraduate

module, the proportion of students getting is regulated at 10%
þ The chance of getting for anyone is .1
However, in case it was known that a student has scored full mark in
the final examination

Would the chance of him/her getting A still be .1

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 38 / 57
 Conditional Probability: A Motivating Example II
Of course, it is intuitive that the chance of him/her getting

would probably be > .1!

The grade obtained by a student & his/her score in the final
examination are related
Once it is known that the student has scored full mark in the final
examination (or equivalently, the event that the student has scored
full mark in the final examination has occurred), it changes our belief
in (indeed, it practically increases) the prob of getting for the
student. Indeed, it changes the experiment
Given an experiment with all known characteristics & known probs of
outcomes, suppose now it happens that some additional information
about the experiment is available. It may affect the sample
space (i.e., the possible outcomes), & the probs associated with
each outcome

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 39 / 57
 Conditional Probability
Definition
Let A & B be 2 events with P (B ) > 0. The
conditional probability of A given B is defined to be

P (AB )
P (A |B ) = ∈ [0, 1]
P (B )

“Given B”: an event B has already occurred

Reduced sample space ⇔ Only outcomes in B, but not in B c ⊂ S,
are possible to occur
the sample space for this “new” experiment becomes B rather than S
& has a probably smaller cardinality
it is possible that it is easy to understand A once B has occurred as B
contains fewer outcomes than S
Multiplication law (for computing prob of an intersection in terms of
conditional prob): P (AB ) = P (B )P (A |B ) = P (A )P (B |A ) when
P (A ), P (B ) , 0
Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 40 / 57
 Example 15: Conditional Probability I

Suppose that of all individuals buying a certain digital camera, on the spot,
60% also buy a memory card, 40% buy an extra battery, & 30% buy both.
Consider randomly selecting a buyer, & let A be the event that a memory
card is purchased & B be the event that an extra battery is purchased.
Then, we have P (A ) = .6, P (B ) = .4, P (AB ) = .3. Given that the selected
buyer purchased an extra battery, prob that a memory card was also
purchased is
P (AB ) .3
P (A |B ) = = = .75
P (B ) .4
P (A c B ) P (B ) − P (AB ) .1
P (A c |B ) = = = = .25
P (B ) P (B ) .4

Note : P (A |B ) , P (A ) & P (A |B ) + P (A c |B ) = 1

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 41 / 57
 Example 16: Conditional Probability II
A bin contains 25 light bulbs, of which 5 are good & function at
least 30 days, 10 are partially defective & will fail in the second
day of use, while the rest are totally defective & won’t light up at
all. Given that a randomly chosen bulb initially lights up, what is
the prob that it will still be working after one week?

Solution: Let G be event that the randomly chosen bulb is in good

condition, & T be the event that the randomly chosen bulb is totally
defective. This implies that T c represents the event that the selected bulb
is either in good condition or partially defective (i.e., the selected bulb
lights up initially). The required conditional prob is

P (GT c ) P (G ) 5/25 1
P (G |T c ) = = = =
c
P (T ) c
P (T ) 15/25 3

Note : P (G |T c ) , P (G ) = 5/25

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 42 / 57
 Example 17: Conditional Probability And Reduced
Sample Space
A fair coin is flipped twice. All outcomes in

S = {HH , HT , TH , TT }

are equally likely. What is the prob that both flips

result in heads given that the first flip does?
Solution: Let A = {HH } & B = {HH , HT }. Then, P (B ) = 2/4 = 1/2, &
P (AB ) = P ({HH }) = 1/4. So,

P (AB ) 1/4 1
P (A |B ) = = =
P (B ) 1/2 2

Alternatively, since B has occurred, the reduced sample space is

B = {HH , HT }. Then, AB = A = {HH }, & hence, P (A |B ) = 1/2

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 43 / 57
 Law of Total Probability I
Definition
A collection of events B1 , B2 , . . . , Bn is called a partition of size n if
Ê Bi Bj = ∅ for i , j & Ë ∪ni=1 Bi = S

B1 , B2 , . . . , Bn are called mutually exclusive & exhaustive events

The simplest partition of size 2 including any event B is {B , B c }

Law of Total Probability

Let B1 , B2 , . . . , Bn be a partition with P (Bi ) > 0 for all i. Then,
for any event A ,
n
X
P (A ) = P (Bi )P (A |Bi )
i =1

To compute any P (A ): Freely choose a partition s.t. all the 2n probs

at the RHS are available or easily computed
Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 44 / 57
 Law of Total Probability II
e.g., with a partition of size n = 4:

The idea :
1 break A into 4 (or, in general, n) “smaller” events, namely,
A ∩ B1 , . . . , A ∩ B4 (or A ∩ Bn )
2 compute probs of the 4 (or n) intersections by multiplication law

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 45 / 57
 Example 18: Law of Total Probability
Of the items produced daily by a factory, 40% come from line 1 & 60%
from line 2. Line 1 has a defect rate of 8%, where line 2 has a defect rate
of 10%. If an item is chosen at random from the day’s production, find the
prob that it will not be defective



 D: item is defective
Solution: Define events:  L1: item comes from line 1



 L :
2 item comes from line 2
From the question, we have P (L1 ) = 1 − P (L2 ) = .4,
P (D c |L1 ) = 1 − P (D |L1 ) = .92, & P (D c |L2 ) = 1 − P (D |L2 ) = .9. Then,

P (an item is not defective) = P (D c )

= P (L1 )P (D c |L1 ) + P (L2 )P (D c |L2 )
= .4(.92) + .6(.9)
= .908

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 46 / 57
 Example 19: Law of Total Probability & Tree Diagram I
A chain of video stores sells 3 different brands of DVD players. Of its DVD
player sales, 50% are brand 1 & 30% are brand 2. Each manufacturer
offers 1–year warranty on parts & labor. It is known that 25% of brand 1’s
players require warranty repair work, whereas the corresponding
percentages for brands 2 & 3 are 20% & 10%, respectively. What is the
prob that a randomly selected purchaser has a DVD player that will need
repair while under warranty?

Solution: Let Ai = {brand i is purchased}, for i = 1, 2, 3, &

B = {needs repair}. Then, we have P (A1 ) = .5, P (A2 ) = .3, P (A3 ) = .2,
P (B |A1 ) = .25, P (B |A2 ) = .2, P (B |A3 ) = .1. Treating {A1 , A2 , A3 } as a
partition of size 3 & applying the law of total prob yield

P (B ) = P ({brand 1 & repair} or {brand 2 & repair} or {brand 3 & repair})

= P (A1 )P (B |A1 ) + P (A2 )P (B |A2 ) + P (A3 )P (B |A3 )
= .5(.25) + .3(.2) + .2(.1) = .205
Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 47 / 57
 Example 19: Law of Total Probability & Tree Diagram II
A tree diagram is a handy tool for computing probs in
experiments composing of several stages/generations
Components/ingredients in a tree diagram:
nodes & branches; total # in different generations depending on the
total # of possible outcomes
probs attached to each branch
Here, in this example,
the initial/1st generation branches correspond to different brands of
DVD players þ 3 branches in the 1st generation
the 2nd generation branches correspond to “needs repair” or “doesn’t
need repair” þ 2 branches at the end of each branch of the 1st
generation
probs attached to the branches in the 1st generation
probs attached to the branches in the 2nd generation: “conditional
probs” based on what has happened in the 1st generation

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 48 / 57
 Example 19: Law of Total Probability & Tree Diagram III

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 49 / 57
 Bayes’ Formula
Bayes’ Formula
Let B1 , B2 , . . . , Bn be a partition with P (Bi ) > 0 for all i. Then, for any event
A with P (A ) > 0,

P (Bj )P (A |Bj )
P (Bj |A ) = Pn j = 1, 2, . . . , n
i =1 P (Bi )P (A |Bi )

Numerator: P (ABj ); Denominator: P (A )

Reverse chronological order: usually Bj happens before A in time
what should have happened before A has occurred?

As Bj is the event of interest, it provides a hint that one needs to

look for a partition containing Bj s.t. the 2n probs in the denominator
are available or computable
No more complicated than the law of total prob!

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 50 / 57
 Example 20: Bayes’ Formula
Refer to Example 19 about DVD players at Page 47, if a
customer returns to the store with a DVD player that needs
warranty repair work, what is the prob that it is a brand 1
player? a brand 2 player? a brand 3 player?

Solution: One should apply the Bayes’ formula based on the partition
{A1 , A2 , A3 } to compute these required conditional probs. With
P (B ) = .205 obtained at Page 47, we have
P (A1 )P (B |A1 ) .5(.25)
P (A1 |B ) = = = .61
P (B ) .205
P (A2 )P (B |A2 ) .3(.2)
P (A2 |B ) = = = .29
P (B ) .205
P (A3 )P (B |A3 ) .2(.1)
P (A3 |B ) = = = .1
P (B ) .205

Note : P (A1 |B ) + P (A2 |B ) + P (A3 |B ) = 1

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 51 / 57
 Example 21: Calling a Cab
Three telephone lines, A , B & C, are available for calling a
cab. The failure rate in connection is 20% for A , 10% for B,
& 30% for C. However, line A is more popular & is used for
60% as it is more advertised, whereas line B is used for
30%. A business person failed to connect to a line. What is
the prob that s/he used line A ?

Solution: Let F be the event of failing to connect to a line. Then,

P (FA )
P (A |F ) =
P (F )
P (A )P (F |A )
=
P (A )P (F |A ) + P (B )P (F |B ) + P (C )P (F |C )
.6(.2)
=
.6(.2) + .3(.1) + .1(.3)
= .667

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 52 / 57
 Independence I

An important concept or notion in prob & statistics based on

conditional probs
Recall: The definition of conditional prob enables us to revise the
prob P (A ) originally assigned to A when we are subsequently
informed that another event B has occurred; the “new” prob of A is
P (A |B )
?
P (A ) = P (A |B )

Intuitively , P (A |B ) would be different from P (A ) unless knowing B

does not tell us anything about A (i.e., “occurrence of B has nothing
to do with occurrence of A ”).

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 53 / 57
 Independence II
Definition
Events A & B are said to be independent (indept) if

P (AB ) = P (A )P (B ), or equivalently,
P (A ) = P (A |B ), or equivalently,
P (B ) = P (B |A )

Otherwise they are said to be dependent

In general, multiplication law P (AB ) = P (A )P (B |A )

= P (B )P (A |B ) is always true for any intersection; the first formula
above, P (AB ) = P (A )P (B ), is a special case following from
independence (indep) of A & B
Indep & disjointness are 2 very different concepts
conclude disjointness from Venn diagram (no probs involved)
indep is defined in terms of probs

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 54 / 57
 Independence III
Indep of 2 Events
If A & B are indept, then so are Ê A & B c , Ë A c & B, & Ì A c & B c

Definition
Three events A , B & C, are said to be mutually indept if all the following 4
conditions hold:

P (ABC ) = P (A )P (B )P (C )
P (AB ) = P (A )P (B )
P (AC ) = P (A )P (C )
P (BC ) = P (B )P (C )

A is indept of any event formed by B & C (e.g., B c , BC, B ∪ C c )

Three events are pairwise indept if the last 3 conditions hold

Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 55 / 57
 Example 22: Independence
1 Draw a card from a deck of playing cards: Let A = {it is an ace} &
D = {it is a ©}. Intuitively, knowing the card being an ace should give
no information about its suit. Verify this with: P (A ) = 4/52 = 1/13,
P (D ) = 13/52 = 1/4 &
1 1 1
P (AD ) = P ({it is ©A}) = = × = P (A )P (D )
52 13 4
þ A & D are indept
2 Toss 2 coins: Suppose that A = {1st coin lands a head},
B = {2nd coin lands a head} & C = {exactly 1 head is observed}
It is clear that P (B |A ) = P (B ) as whatever happens to the 1st coin
does not influence the 2nd coin þ A & B are indept
Suppose that the coins are fair (i.e., P (A ) = P (B ) = .5). Compute
P (C ) = P ({HT , TH }) = 2/4 = .5 & thus
P (AC ) P ({HT }) .25
P (A |C ) = = = = .5 = P ( A )
P (C ) P ({HT , TH }) .5
þ A & C are indept
Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 56 / 57
 Example 23: Tossing 2 Dices
Toss 2 fair dice:
1 Let A6 denote the event that the sum of 2 dice is 6, & B denote the
event that
( the 1st die equals 4
A6 = { , , , , }
Then,
B = { , , , , , }
So, A6 B = { }, &
1 5 1
= P (A6 B ) , P (A6 )P (B ) = ×
36 36 6
þ A6 & B are dependent
2 Let A7 denote the event that the sum of 2 dice is 7. Then,
P (A7 ) = P ({ , , , , , }) = 6/36 = 1/6 &
P (A7 B ) = P ({ }) = 1/36. Hence,
1 1 1
= P (A7 B ) = P (A7 )P (B ) = ×
36 6 6
þ A7 & B are indept
Monday & Thursday, 12nn-2pm (UT AUD 3) Dr. HO Man-Wai ([email protected]) ST2334(NUS) Ch.3 57 / 57