Probability

1 Sample Spaces and Events

1.1 Random Experiments
A random experiment is a procedure that produces one outcome from a set of possible outcomes. Each performance of
the experiment is called a trial.
• Fair experiment: When all the outcomes of a random experiment are equally likely, i.e., have the same chances of
happening.
• Biased Experiment: When all the outcomes of a random experiment are not equally likely, i.e., some outcomes have
greater/smaller chances of happening than others.
1.2 Sample Spaces
The sample space S of a random experiment is the set of all possible outcomes. For example, for a coin toss, S = {H, T }.
1.3 Events
An event is a subset of the sample space. For example, in a die roll, the event of rolling an even number is E = {2, 4, 6}.
• Simple Event: An event that consists of exactly one outcome. For example, rolling a 3 on a die.
• Compound Event: An event that consists of more than one outcome. For example, rolling an even number on a die.
• Mutually Exclusive Events: Events that cannot occur at the same time. If A and B are mutually exclusive, then
A ∩ B = ∅.
• Exhaustive Events: A set of events is exhaustive if their union covers the entire sample space. For example, in a die
roll, the events {1, 2}, {3, 4}, and {5, 6} are exhaustive.
• Independent Events: Two events are independent if the occurrence of one does not affect the occurrence of the other.
For A and B to be independent, P (A ∩ B) = P (A)P (B).
1.4 Counting Techniques
For finite sample spaces, the number of outcomes in an event A can be counted directly through the following rules.
• Multiplication rules: If an event A can occur in a series of steps, where each step can be done in n1 , n2 , n3 , ...nk ways,
then the total number of ways the events can occur is ways.
Total ways = n1 × n2 × n3 × ... × nk

• Permutation rule: A permutation is an arrangement of all or part of a set of objects. The number of permutations of n
objects taken r at a time is given by:
n!
P (n, r) =
(n − r)!
where n! (n factorial) is the product of all positive integers up to n. Note, 0! = 1.
• Combination rule: A combination is a selection of items from a larger pool where the order does not matter. The number
of combinations of n objects taken r at a time is given by:
 
n n!
C(n, r) = =
r r!(n − r)!

- Permutations and combinations are useful for counting the number of ways to arrange or select items.
2 Interpretations and Axioms of Probability
2.1 Interpretations of Probability
• Classical Interpretation: Probability as the ratio of favorable outcomes to the total number of outcomes. It is written
as P (A):
n
P (A) =
N
, where n = total number of ways the event A can happen, and N = the size of the sample space S.
• Subjective Interpretation: Probability as a measure of belief about the occurrence of an event.
2.2 Axioms of Probability
Let P be a probability measure on a sample space S:
• P (A) ≥ 0 for any event A ⊆ S.
• P (S) = 1.
• For any sequence of mutually exclusive events {Ai }:
!
[ X
P Ai = P (Ai )
i i

3 Addition Rules
3.1 Mutual Exclusivity
Events A and B are mutually exclusive if A ∩ B = ∅. For mutually exclusive events:

P (A ∪ B) = P (A) + P (B)

3.2 General Addition Rule

For any two events A and B:
P (A ∪ B) = P (A) + P (B) − P (A ∩ B)

3.3 Complementary Rule

For any event A :
P (A′ ) = 1 − P (A)

4 Conditional Probability
The conditional probability of A given B is:
P (A ∩ B)
P (A|B) =
P (B)
provided P (B) > 0.
5 Multiplication Rules
For any two events A and B:
P (A ∩ B) = P (A)P (B|A)
If A and B are independent:
P (A ∩ B) = P (A)P (B)

6 Independence
Events A and B are independent if:
P (A ∩ B) = P (A)P (B)

7 Mutual Exclusivity vs Conditional Independence

Mutual Exclusivity Conditional Independence

Definition A∩B =∅ P (A ∩ B|C) = P (A|C)P (B|C)
P (A ∪ B) P (A ∪ B) = P (A) + P (B) P (A ∪ B) = P (A) + P (B) − P (A ∩ B)
P (A ∩ B) P (A ∩ B) = 0 P (A ∩ B) = P (A)P (B) (if A and B are independent)
Observability from Venn diagram only through multiplications of individual event probabilities
8 Some probability rules based on Set theory
8.1 De Morgan’s Laws
De Morgan’s Laws provide relationships between the union and intersection of sets through complementation.
8.1.1 First Law
The complement of the union of two sets is the intersection of their complements, i.e., (A ∪ B)′ = A′ ∩ B ′ . Therefore, we
have the following:
P ((A ∪ B)′ ) = P (A′ ∩ B ′ )
8.1.2 Second Law
The complement of the intersection of two sets is the union of their complements, i.e, (A ∩ B)′ = A′ ∪ B ′ :

P ((A ∩ B)′ ) = P (A′ ∪ B ′ )

8.2 Only one event

8.2.1 Probability of ONLY A
When there are two event A and B, then the probability of only the event A is
P (A ∩ B ′ ) and that can be written as:

P (A ∩ B ′ ) = P (A) − P (A ∩ B)

Figure 1. Probability of ONLY A

8.2.2 Probability of ONLY B
When there are two event A and B, then the probability of only the event B is
P (A′ ∩ B) and that can be written as:

P (A′ ∩ B) = P (B) − P (A ∩ B)

Figure 2. Probability of ONLY B

8.2.3 Probability of subsets
If A and B are two events of a sample space S of a random experiment such that A ⊂ B, then

P (A) = P (A ∩ B)

And, If B ⊂ A, then
P (B) = P (A ∩ B)
Therefore, If A ⊆ B and B ⊆ A, then
P (A) = P (B) = P (A ∩ B)
8.3 Practice derivations

1) P (A′ ∪ B) = P (A′ ) + P (B) − P (A′ ∩ B) [Addition rule]

′
= P (A ) + P (B) − [P (B) − P (A ∩ B)]
= P (A′ ) − P (A ∩ B)

2) P (A ∪ B ′ ) = P (A) + P (B ′ ) − P (A ∩ B ′ ) [Addition rule]

′
= P (A) + P (B ) − [P (A) − P (A ∩ B)]
= P (B ′ ) − P (A ∩ B)

3) P [(A ∪ B)′ ] = 1 − P (A ∪ B) [Complementary rule]

′ ′
= P (A ∩ B ) [De morgan’s law]
 P (A′ ∩ B)
4) P (B|A′ ) = [Conditional prob. rule]
P (A′ )
P (B) − P (A ∩ B)
= [Rule for only B]
P (A′ )
P (B) − P (A ∩ B)
= [Complementary rule]
1 − P (A)

5) P (B ′ |A) = 1 − P (B|A) [Complementary rule]

P (A ∩ B)
=1− [Conditional rule]
P (A)

9 Total Probability Rule and Bayes’ Theorem

9.1 Total Probability Rule
If {Ei } is a partition of the sample space S:
X
P (B) = P (B|Ei )P (Ei )
i
Figure 3. Total Probability, P(B)
Probability Tree Diagram A probability tree diagram can be used to visually rep-
resent the total probability rule. Example:

Probability Level of Con- Probability

of Failure tamination of Level
0.10 High 0.2
0.01 Medium 0.3
0.001 Low 0.5

9.2 Bayes’ Theorem

If {Ei } is a partition of the sample space S, and the event B has happened already, then:

P (B|Ei )P (Ei )
P (Ei |B) =
P (B)

10 Types of Sampling
Sampling methods are crucial for statistical analysis and determining probabilities. There are two primary types: sampling
with replacement and sampling without replacement.
10.1 Sampling with Replacement
In sampling with replacement, each member of the population is replaced after being selected, so it can be chosen more
than once. If there are N objects in the population, the probability of selecting any particular object in a single draw is
1 k
N . For k draws: P (Selecting the same object each time) = N
1

For example, if you draw a card from a deck of 52 cards and replace it each time, the probability of drawing the Ace of
1 3
Spades in each of three consecutive draws is: P (Ace of Spades each time) = 52


10.2 Sampling without Replacement
In sampling without replacement, each member of the population is not replaced after being selected, so it cannot be
chosen more than once. If there are N objects in the population and you are drawing k objects:
1
P (Selecting a specific sequence of k objects) = N


k

For example, if you draw three cards from a deck of 52 cards without replacing them, the probability of drawing the Ace
of Spades first, the King of Hearts second, and the Queen of Diamonds third is: P (Specific sequence) = 52
1 1
× 51 1
× 50