WIA2003/WIB2003

PROBABILITY AND STATISTICS

Bayes’ Theorem

1
 WIA2003/WIB2003 Probability and Statistics

Revisit:
Probability and its notations
• Fundamentals of Probability
– Deterministic and random processes
– Events and sample space
– Permutation and combination
– Probability
• Intersection (mutual exclusive), union, conditional probability
(independence), complement
• Rule of subtraction, multiplication and addition
– Random Variables
• Discrete and continuous

2
Probability and its notations WIA2003/WIB2003 Probability and Statistics

How to calculate sample space and events?

• When the order is taken into account: Permutations

• When the order is not important: Combinations
• A permutation is an ordered combinations

– Permutation with repetition: 𝑛!

"!
– Permutation without repetition:
"$! !
"! "
– Combination without repetition: OR !
!! "$! !
!%"$& !
– Combination with repetition:
!! "$& !
where 𝑛 is the number of things to choose from, and 𝑟 of them are
chosen.
3
Probability and its notations WIA2003/WIB2003 Probability and Statistics

Permutation with repetition

• Given a travel lock that have 3 numbers to set from

choices from 0-9. What are the total possible orders
to set?

• 𝑛 = 10, 𝑟 = 3;
• Total possible orders à 𝑛! = 10" = 1000.

4
Probability and its notations WIA2003/WIB2003 Probability and Statistics

Permutation without repetition

• In a pool game, there will be a total of 16 different

pool balls. What are the total possible orders for the
first 3 pool balls be in?

• 𝑛 = 16, 𝑟 = 3.
• Total possible orders à
#! &'!
#%! !
= (&'%&")!
= 16 × 15 × 14 = 3360.

5
Probability and its notations WIA2003/WIB2003 Probability and Statistics

Combination without repetition

• In a pool game, there will be a total of 16 different

pool balls. What are the total possible outcome for
the first 3 pool balls be in, regardless of its order?

• 𝑛 = 16, 𝑟 = 3.
• Total possible outcome à
#! &'! &' × &+ ×&,
= = = 560.
!! #%! ! "!(&'%")! "×-×&

6
Probability and its notations WIA2003/WIB2003 Probability and Statistics

Combination with repetition?

• In a pool game, there will be a total of 16 different

pool balls. What are the total possible outcome for
the first 3 pool balls be in, regardless of its order, if
the pool ball will be put back once it get in?

• 𝑛 = 16, 𝑟 = 3.
• Total possible outcome à
(#.!%&)! (&'."%&)! &/! &/ × &0 ×&'
!! #%& !
= "!(&'%&)!
= "!&+! " × - × &
= 816.

7
Probability and its notations WIA2003/WIB2003 Probability and Statistics

Combination with repetition?

• In a pool game, there will be a total of 16 different

pool balls. What are the total possible outcome for
the first 3 pool balls be in, regardless of its order, if
the pool ball will be put back once it get in?

• 𝑛 = 16, 𝑟 = 3.
• Total possible outcome à
(#.!%&)! (&'."%&)! &/! &/ × &0 ×&'
!! #%& !
= "!(&'%&)!
= "!&+! " × - × &
= 816.

8
Probability and its notations WIA2003/WIB2003 Probability and Statistics

Combination with repetition

• Given that there are 5 choices of ice cream flavor.

How many combinations are they if we can have 3
scoops?

• 𝑛 = 5, 𝑟 = 3.
• Total possible outcome à
(#.!%&)! (+."%&)! 0!
!! #%& !
= "!(+%&)!
= "!,!
= 35.

9
Revisit: Probability and its notations WIA2003/WIB2003 Probability and Statistics

Try this!

1. What is the probability for the number of the first

dice roll to be higher than the second dice roll?

15
𝑃 𝑋>𝑌 =
36

10
Revisit: Probability and its notations WIA2003/WIB2003 Probability and Statistics

Try this!

2. Given that they are 50 students in a class, where 30

of them are males, and 20 of them are wearing
spectacles. What is the potential range of
probability for male students that are wearing
spectacles?

• There are two extreme cases, where the intersection between "males"
and "wearing spectacles" 𝑃 𝑀 ∩ 𝑊 is maximum or minimum.
• When 𝑃 𝑀 ∩ 𝑊 is maximum, means 20 of the males is wearing
spectacles. The probability is 20/30 = 0.67.
• When 𝑃 𝑀 ∩ 𝑊 is minimum, means none of the males is wearing
spectacles, which is possible because 20 of the students that wear
spectacles can be all female. The probability is 0/30 = 0.
11
• Therefore, the range now is 0 - 0.67.
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Example:
• Consider a fictitious computer company. We might
ask the following questions:

– What is the probability that consumers will actually buy a new

computer?
– What is the probability that consumers are planning to buy a new
computer?
– What is the probability that consumers are planning to buy and
actually will buy a new computer?
– Given that a consumer is planning to buy, what is the probability of a
purchase?

12
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Probability Review
• If we have an event 𝐴 there will be a complement to
𝐴 called 𝐴′ or 𝐵.
• Marginal probabilities
– Event 𝐴 consists of two outcomes, 𝑎& and 𝑎' :
𝐴 = {𝑎& , 𝑎' }
– The complement 𝐵 consists also of two outcomes, 𝑏& ahd 𝑏' :
𝐵 = {𝑏& , 𝑏' }
– Two events are mutually exclusive/disjoint if both events cannot
occur together
– A set of events is collectively exhaustive if one of the events must
occur

13
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Disjoint (mutual exclusive)

• If A and B are disjoint (mutually exclusive) events,
then
𝑃 𝐴 or 𝐵 = 𝑃 𝐴 + 𝑃(𝐵)

Example:
Let A: respondent lives in KL
Let B: respondents lives in Penang

If 𝑃 𝐴 = 0.087
𝑃 𝐵 = 0.106
Then 𝑃 𝐴 or 𝐵 = 0.193 14
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Probability of Combined Events

• More generally, if 𝐴 and 𝐵 are not mutually
exclusive,
𝑃 𝐴 or 𝐵 = 𝑃 𝐴 + 𝑃 𝐵 − 𝑃(𝐴 and 𝐵)

Example:
Let 𝐴 = event that respondent sleep less than 6 hours
per night.
Let 𝐵 = event that respondent reports trouble sleep
most or all of the time.
𝑃 𝐴 = 0.139; 𝑃 𝐵 = 0.152; 𝑃 𝐴 and 𝐵 = 0.061
Thus 𝑃 𝐴 or 𝐵 = 0.139 + 0.152 − 0.061 = 0.230
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Exhaustive Events
• Two or more events are said to be exhaustive (or
collective exhaustive) if at least one of them must
occur.
• 𝐴∪𝐵 =𝑆

• For example, if 𝐴 is the event that the respondent

sleeps less than 6 hours per night and 𝐵 is the event
that the respondent sleeps at least 6 hours per night,
then 𝐴 and 𝐵 are exhaustive.

16
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Exhaustive VS mutual exclusive

• For a dice roll:
• 1,2,3,4,5 and 6 (all possible values): exhaustive and mutual exclusive
• 2 and 6: mutual exclusive but not exhaustive
• “even” and “not 6”: not mutual exclusive but exhaustive

• For a coin toss?

17
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Independence
• Two events are independent if the occurrence of one
in no way affects the probability of the other.

• If events 𝐴 and 𝐵 are independent, then

𝑝 𝐴 and 𝐵 = 𝑝 𝐴 𝑝 𝐵

• If events 𝐴 and 𝐵 are not independent, then

𝑝 𝐴 and 𝐵 = 𝑝 𝐴 𝑝(𝐵|𝐴)

18
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Example
• The sample space (of 1000 customers) looks like this:

Actually purchase
Yes (𝑏! ) No (𝑏" ) Total
Plan to Yes (𝑎! ) 200 50 250
purchase No (𝑎" ) 100 650 750
Total 300 700 1000

19
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Example
• If 𝐴 = planned to purchase and 𝐵 = actually
purchased:
𝑃(planned to buy)= 𝑃(planned & did) + 𝑃(planned & did not)
= 𝑃 𝑎! and 𝑏! + 𝑃(𝑎! and 𝑏" )
Actually purchase
Yes (𝑏! ) No (𝑏" ) Total
Plan to Yes (𝑎! ) 200 50 250
purchase No (𝑎" ) 100 650 750
Total 300 700 1000

"## $# "$#
= + = = 0.25
!### !### !###
20
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Probability Review
• Example, there are 40 female students in a class of
100. 10 of them are from some foreign countries. 20
male students are also foreign students.

– 𝐴: student from a foreign country

– 𝐵: a female student

21
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Venn Diagrams

30+10 = 40 20+10 = 30

Foreign
Female (10) Student
(30)
(20)

Male non-foreign student Female foreign student (10)

(40)

22
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Probability Review
• Complement: 𝑝 𝐴3 = 1 − 𝑝 𝐴 ; 𝑝 𝐵3 = 1 − 𝑝(𝐵)

Non Female
Female

Non Foreign
Foreign
Student
student

23
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Conditional probability
n The conditional probability of 𝐴 given 𝐵 is the joint
probability of 𝐴 and 𝐵, divided by the marginal
probability of 𝐵:
𝑃 𝐴, 𝐵
𝑃 𝐴𝐵 =
𝑃 𝐵
n Thus if 𝐴 and 𝐵 are statistically independent,
𝑃 𝐴, 𝐵 𝑃 𝐴 𝑃 𝐵
𝑃 𝐴𝐵 = = = 𝑃(𝐴)
𝑃 𝐵 𝑃 𝐵
n However, if 𝐴 and 𝐵 are statistically dependent, then
𝑃 𝐴 𝐵 ≠ 𝑃(𝐴)

24
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Conditional Probabilities
• Conditional probabilities can be written as
𝑃 𝑋&𝑌 Joint probability of 𝑋 & 𝑌
= = 𝑃(𝑋|𝑌)
𝑃 𝑋 Marginal probability 𝑜𝑓 𝑋

– This is read as “The probability of X given Y”

– For example: The probability that someone earns between $200 and
$300, given that he/she has completed 10 years of education

25
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Marketing Example
Actually purchase
Yes (𝑏! ) No (𝑏" ) Total
Plan to Yes (𝑎! ) 200 50 250
purchase No (𝑎" ) 100 650 750
Total 300 700 1000

• Let’s look at the probability you purchased a computer given

that you planned to purchase:
𝑃 actually purchased planned to purchase
= 𝑃(𝑏&|𝑎&)
4 5! ,7!
= 4 7
-88
=
-+8
= 0.8 26
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Useful Properties of Conditional

Probabilities
• Property 1. The Conditional Probability for
Independent Events
– If A and B are independent events, then:
𝑃 𝐴∩𝐵 𝑃 𝐴 𝑃 𝐵
𝑃 𝐴𝐵 = = = 𝑃(𝐴)
𝑃 𝐵 𝑃 𝐵

• Property 2. The Multiplication Rule for Conditional

Probabilities
– In an experiment involving two non-independent events 𝐴 and 𝐵, the
probability that both 𝐴 and 𝐵 occurs can be found in the following two
ways:
𝑃 𝐴∩𝐵 =𝑃 𝐵 𝑃 𝐴 𝐵
or
27
𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝑃(𝐵|𝐴)
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem/Rule
n Someone is shooting at a target. If it is windy, he has
a 40% chance hitting the target. If there is no wind,
his chance is 70%.

n A. If there is a 12% chance of being windy, what is his

chance of hitting the target?

n B. If he hits the target, what is the chance of it being

windy? à Bayes Theorem

28
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem
• Introduced by Reverand Thomas Bayes (1702-1761)
• Bayes Theorem for probability events A and B
𝑃 𝐵𝐴 𝑃 𝐴
𝑃 𝐴𝐵 =
𝑃 𝐵
Or for a set of mutually exclusive and exhaustive
events (i.e. 𝑝 ⋃M 𝐴M = ∑M 𝑝 𝐴M = 1), then
𝑃 𝐵 𝐴 𝑃 𝐴(
𝑃 𝐴( 𝐵 =
∑) 𝑃 𝐵 𝐴) 𝑃(𝐴) )

29
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem
• Think of the Bayes Theorem as probability in reverse
– You can update your probabilities in light of new information
• Suppose you have a product with a known
probability of success
𝑃 success = 𝑃 𝑆 = 0.4
𝑃 failure = 𝑃 𝑆 * = 0.6
• We also know that a consumer group will write
either a favorable or unfavorable report on the
product
𝑃 𝐹 𝑆 = 0.8; 𝑃 𝐹 𝑆 * = 0.3

30
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem
• We want to find the probability that the product will
be successful given a favorable report
𝑃(𝑆|𝐹)
• In this case, Bayes says
𝑃 𝐹𝑆 𝑝 𝑆
𝑃 𝑆𝐹 =
𝑃 𝐹 𝑆 𝑃 𝑆 + 𝑃 𝐹 𝑆 * 𝑃(𝑆 * )
• We can plug values into the above equation to find
0.8 0.4 0.32
𝑃 𝑆𝐹 = = = 0.64 = 64%
0.8 0.4 + (0.3)(0.6) 0.50
• We can use the theorem to update the probability of
a successful product given that the product gets a
favorable report
31
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem/Rule
• Bayes theorem deals with sequential events,
whereby new additional information is obtained for a
subsequent event, and that new information is used
to revise the probability of the initial event.

– Prior probability- is an initial probability value originally

obtained before any additional information is obtained.

– Posterior probability- is a probability value that has been

revised by using additional information that is later
obtained.
32
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem/Rule - example

The Gallup organization randomly selects an adult
American for a survey about credit card usage. Use
subjective probabilities to estimate the following.

a) What is the probability that the selected subject is a male?

b) After selecting a subject, it is later learned that this person
was smoking a cigar during the interview. What is the
probability that the selected subject is a male?
c) Which of the preceding two results is a prior probability?
Which is a posterior probability?

33
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem/Rule - example

a) Roughly half of all Americans are males, so we estimate the
probability of selecting a male subject to be 0.5. Denoting a
male by 𝑀, we can express this probability as follows:
𝑃 𝑀 = 0.5.

b) Although some women smoke cigars, the vast majority of

cigar smokers are males. A reasonable guess is that 85% of
cigar smokers are males. Based on this additional subsequent
information that the survey respondent was smoking a cigar,
we estimate the probability of this person being a male as
0.85. Denoting a male by 𝑀 and denoting a cigar smoker by
𝐶, we can express this result as follows: 𝑃 𝑀 𝐶 = 0.85.
34
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem/Rule - example

c) In part a), the value of 0.5 is the initial probability, so we
refer to it as the prior probability. Because the probability of
0.85 in part b) is a revised probability based on the additional
information that the survey subject was smoking a cigar, this
value of 0.85 is referred to a posterior probability.

35
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem
• In Orange County, 51% of the adults are males. One
adult is randomly selected for a survey involving
credit card usage.

a) Find the prior probability that the selected person is a male.

b) It is later learned that the selected survey subject was
smoking a cigar. Also, 9.5% of males smoke cigars, whereas
1.7% of females smoke cigars.
c) Use this additional information to find the probability that
the selected subject is a male.

36
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem
• Let's use the following notation:
𝑀 = male 𝑀′ = female (or not male)
𝐶 = cigar smoker C′ = not a cigar smoker.

a. Before using the information given in part b, we know only

that 51% of the adults in Orange County are males, so the
probability of randomly selecting an adult and getting a male
is given by 𝑝 𝑀 = 0.51.

37
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem
b. Based on the additional given information, we have the
following:
– 𝑃(𝑀) = 0.51 because 51% of the adults are males
– 𝑃(𝑀’) = 0.49 because 49% of the adults are females (not
males)
– 𝑃(𝐶|𝑀) = 0.095 because 9.5% of the males smoke cigars
(The probability of getting someone who smokes cigars,
given that the person is a male, is 0.095.)
– 𝑃(𝐶|𝑀’) = 0.017. because 1.7% of the females smoke
cigars (The probability of getting someone who smokes
cigars, given that the person is a female, is 0.017.)

38
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ Theorem
c. Initially we knew that the survey subject smoked a cigar,
there is a 0.51 probability that the survey subject is male,
however, after learning that the subject smoked a cigar, we
revised the probability to 0.853. This makes sense, because
the likelihood of a male increases dramatically with the
additional information that the subject smokes cigars
(because so many more males smoke cigars than females).
4 9 ⋅4(;|9)
𝑃 𝑀𝐶 =
4 9 ⋅4(;|9).4 9" ⋅4(;|9" )
8.+&⋅8.8>+
=
8.+&⋅8.8>+ .[8.,>⋅8.8&0]
= 0.85329341
= 0.8533 (rounded)
39
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

More Examples

• Given:
– A doctor knows that meningitis (𝑀) causes stiff neck (𝑆) 50%
of the time
– Prior probability of any patient having meningitis is 1/50,000
– Prior probability of any patient having stiff neck is 1/20

• If a patient has stiff neck, what’s the probability he/she

has meningitis?
1 1
𝑃 𝑆𝑀 𝑃 𝑀 0.5× 0.5×
𝑃 𝑀𝑆 = = 50000 = 50000 = 0.0002
𝑃(𝑆) 𝑃(𝑆) 1
20
40
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

An example using Bayes’ theorem

• Suppose the probability (for anyone) to have AIDS is:
𝑃(AIDS)=0.001; 𝑃(no AIDS)=0.999 ← prior probabilities, i.e.,
before any test carried out
Consider an AIDS test: result is + or –

𝑃 + 𝐴𝐼𝐷𝑆 = 0.98; 𝑃 − 𝐴𝐼𝐷𝑆 = 0.02; ← probabilities to (in)correctly

identify an infected person

𝑃 + 𝑛𝑜 𝐴𝐼𝐷𝑆 = 0.03; 𝑃 − 𝑛𝑜 𝐴𝐼𝐷𝑆 = 0.97 ← probabilities

to (in)correctly
identify an
uninfected
person

Suppose your result is +. How worried should you be?

41
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics

Bayes’ theorem example (cont.)

• The probability to have AIDS given a + result is
• Posterior probability:
A(.|BCDE)A BCDE
𝑃 𝐴𝐼𝐷𝑆 + = A(.|BCDE)A BCDE .A(.|#F BCDE)A(#F BCDE)
8.>/×8.88&
= 8.>/×8.88&.8.8"×8.>>>
= 0.032

• Your viewpoint: my degree of belief that I have AIDS

is 3.2%
• Your doctor’s viewpoint: 3.2% of people like this will
have AIDS 42
The End

43