Lecture 1

Elementary Probability

1 / 79
What is probability?

2 / 79
A simple question

• A box contains black and red balls (unknown). We randomly

draw a ball.
• Repeat this 1000 times, we obtain 898 blacks and 102 reds. In
the next draw, what can we say about the probability of
drawing a black ball?

3 / 79
A simple question

• A box contains black and red balls (unknown). We randomly

draw a ball.
• Repeat this 1000 times, we obtain 898 blacks and 102 reds. In
the next draw, what can we say about the probability of
drawing a black ball?

• “Too easy!
I don’t need to study probability theory to answer this question.
The probability is about 0.9 ”.

4 / 79
Another question

• November 8, 2016 is the day of the US presidential election.

• On FiveThirtyEight.com, Nate Silver (who correctly forecast the
results of the presidential election several times before) wrote, at
9am:
”The probability of Hillary Clinton winning the election is 71%.”
• That evening, Donald Trump was declared the winner to the
surprise of many.

So... does the probability 71% above mean anything?

5 / 79
The Monty Hall problem (1)

Do you want to switch?

6 / 79
The Monty Hall problem (2)

In another situation...

• Suppose the host suddenly forgets where the car is. Since the
show is still going on, he has to open a door randomly anyway.
Luckily, that door is the goat!

• Now the host asks you if you want to switch. Do you want to
change?

7 / 79
Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT

Course outline
1. Elementary probability

2. Conditional probability

3. Discrete random variables

4. Continuous random variables

5. Joint distributions

6. Properties of Expectation and advanced topics

7. Law of Large Numbers, Central Limit Theorem

8. Markov Chains
18//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Combinatorics / the basic principle of counting

In this part we’ll learn how to count in some typical scenarios.

The starting point is the following:
◮ Suppose an experiment has n outcomes; and another
experiment has m outcomes.
◮ Then the two experiments jointly have n · m outcomes.

Example
Rolling a die and flipping a coin can have a total of 6 · 2 = 12
different outcomes, combined.

39//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

1. Permutations

Definition
Let H = {h1 , h2 , . . . , hn } be a set of n different objects. The
permutations of H are the different orders in which one can
write all of the elements of H. There are n! = 1 · 2 · 3 · · · n of
them. We set 0! = 1.

4 //281
10 79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

1. Permutations

Definition
Let H = {h1 , h2 , . . . , hn } be a set of n different objects. The
permutations of H are the different orders in which one can
write all of the elements of H. There are n! = 1 · 2 · 3 · · · n of
them. We set 0! = 1.

Example
The results of a horse race with horses
H = {A, B, C, D, E, F, G} are permutations of H. A possible
outcome is (E, G, A, C, B, D, F) (E is the winner, G is second,
etc.). There are 7! = 5 040 possible outcomes.

4 //281
11 79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

2. Permutations with repetitions

Definition
Let H = {h1 . . . h1 , h2 . . . h2 , . . . , hr . . . hr } be a set of r different
types of repeated objects: n1 many of h1 , n2 of h2 , . . . nr of hr .
The permutations with repetitions of H are the different orders
in which one can write all of the elements of H. There are
 
n n!
:=
n1 , n2 , . . . , nr n1 ! · n2 ! · · · nr !

of them, where n = n1 + · · · + nr is the total number of objects.

This formula is also known as the multinomial coefficient.

5 //281
12 79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

2. Permutations with repetitions

Example
We can make
 
11 11!
= = 83 160
5, 2, 2, 1, 1 5! · 2! · 2! · 1! · 1!

different words out of the letters A, B, R, A, C, A, D, A, B, R, A.

6 //281
13 79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

3. k-permutations
Definition
Let H = {h1 , h2 , . . . , hn } be a set of n different objects. The
k-permutations of H are the different ways in which one can
n!
pick and write k of the elements of H in order. There are (n−k )!
of these k-permutations.

7 //281
14 79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

3. k-permutations
Definition
Let H = {h1 , h2 , . . . , hn } be a set of n different objects. The
k-permutations of H are the different ways in which one can
n!
pick and write k of the elements of H in order. There are (n−k )!
of these k-permutations.

Example
The first three places of a horse race with horses
H = {A, B, C, D, E, F, G} form a 3-permutation of H. A
possible outcome is (E, G, A) (E is the winner, G is second, A
7!
is third.). There are (7−3)! = 210 possible outcomes for the first
three places.

7 //281
15 79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. k-combinations

Definition
Let H = {h1 , h2 , . . . , hn } be a set of n different objects. The
k-combinations of H are the different ways in which one can
pick k of the elements of H without order. There are
 
n n!
:=
k k! · (n − k)!

of these k-combinations. This formula is also known as the

binomial coefficient (“n choose k”).

9 //281
16 79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. k-combinations
Example
There are  
30 30!
= = 142 506
5 5! · (30 − 5)!
possible ways to form a committee of 5 students out of a class
of 30 students.

10
17//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. k-combinations
Example
There are  
30 30!
= = 142 506
5 5! · (30 − 5)!
possible ways to form a committee of 5 students out of a class
of 30 students.

Remark
In a similar way, there are
 
n n!
:=
k1 , k2 , . . . , kr k1 ! · k2 ! · · · kr !

many ways to form unordered groups of sizes k1 , k2 , . . . kr of n

objects (n = k1 + · · · + kr ). Thus, the multinomial coefficient
generalizes the binomial coefficient.
10
18//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

6. The Binomial coefficient

◮ Recall the definition, for n, k non-negative integers,

   
n n! n
:= = for 0 ≤ k ≤ n.
k k! · (n − k)! n−k


We extend this by kn ≡ 0 in all other cases. (It is possible
to define these coefficients for any real n, but we won’t
need that.)

11
19//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

6. The Binomial coefficient

Theorem (Pascal’s Identity)
For any k and 1 ≤ n integers,
     
n n−1 n−1
= + .
k k −1 k

12
20//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

6. The Binomial coefficient

Theorem (Pascal’s Identity)
For any k and 1 ≤ n integers,
     
n n−1 n−1
= + .
k k −1 k

Proof.
Either write out the factorials, or count the number of
k-combinations of n objects in two ways:
◮ the first object is chosen, and the remaining k − 1 objects
need to be picked out of n − 1, or
◮ the first object is not chosen, and all k objects need to be
picked out of n − 1.

12
21//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

7. The Binomial Theorem

Theorem (Newton’s Binomial Theorem)

For any real numbers x and y, and n ≥ 1, we have
n  
X
n n
(x + y) = · x k · y n−k .
k
k =0

13
22//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

8. The Multinomial Theorem

The Binomial Theorem generalises to

Theorem (Multinomial Theorem)

Let x1 , x2 , . . . , xr be real numbers, n ≥ 1. Then
X  
n
(x1 + x2 + · · · + xr ) = n
· x1n1 · x2n2 · · · xrnr .
n 1 n 2 . . . nr
n1 , ..., nr ≥0
n1 +···+nr =n

14
23//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Sample space
Here we are (almost) going to define a mathematical model for
various experiments. To do it properly, we would need some
tools from measure theory. This will be skipped for now, but
you are welcome to revisit this point some time later during your
studies!

15
24//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Sample space
Here we are (almost) going to define a mathematical model for
various experiments. To do it properly, we would need some
tools from measure theory. This will be skipped for now, but
you are welcome to revisit this point some time later during your
studies!
◮ We always consider an experiment. Ω will denote the set of
all possible outcomes of this experiment.
◮ An event will be a collection of possible outcomes.
Therefore, and event E will be considered a subset of Ω:
E ⊆ Ω.

15
25//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Sample space
Here we are (almost) going to define a mathematical model for
various experiments. To do it properly, we would need some
tools from measure theory. This will be skipped for now, but
you are welcome to revisit this point some time later during your
studies!
◮ We always consider an experiment. Ω will denote the set of
all possible outcomes of this experiment.
◮ An event will be a collection of possible outcomes.
Therefore, and event E will be considered a subset of Ω:
E ⊆ Ω.
◮ Sometimes Ω is too large, and not all its subsets can be
defined as events. This is where measure theory helps...
◮ It makes perfect sense to define the union E ∪ F and the
intersection E ∩ F of two events, E and F .

15
26//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Sample space

Ω Ω
E F E F

E ∪F E ∩F

Notation: sometimes E ∪ F = E + F , E ∩ F = EF .

16
27//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

1. Examples

Example
Experiment: Is it going to rain today?
Sample space: Ω = {r, n}.
|Ω| = 2.
An event: E = {r}.
|E| = 1.

17
28//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

1. Examples
Example
Experiment: Finishing order of a race of 7 horses.
Sample space: Ω = {permutations of A, B, C, D, E, F, G}.
|Ω| = 7!.
An event: E = {horse B wins}
= {permutations that start with B}.
|E| = 6!.
Another event: F = {G wins, D is second}.
= {permutations starting as (G, D, . . . )}.
|F | = 5!.

Notice E ∩ F = ∅ in this example. We call ∅ the null event. This

is the event that never happens.
18
29//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

1. Examples
Example
Experiment: Flipping two coins.
Sample space: Ω = {ordered pairs of the two outcomes}.
= {(H, H), (H, T ), (T , H), (T , T )}.
|Ω| = 4.
An event: E = {the two coins come up different}
= {(H, T ), (T , H)}.
|E| = 2.
Another event: F = {both flips come up heads}.
= {(H, H)}.
|F | = 1.
Notice: E ∪ F = {(H, T ), (T , H), (H, H)}
= {at least one H}.
19
30//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

1. Examples
Example
Experiment: Rolling two dice.
Sample space: Ω = {ordered pairs of the two outcomes}
= {(i, j) : i, j = 1 . . . 6}.
|Ω| = 36.
An event: E = {the sum of the rolls is 4}
= {(1, 3), (2, 2), (3, 1)}.
|E| = 3.
Another event: F = {the two rolls are the same}.
= {(i, i) : i = 1 . . . 6}.
|F | = 6.
Notice: E ∩ F = {(2, 2)}.

20
31//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

1. Examples

Example
Experiment: Repeatedly rolling a die until we first see 6.
Sample space: Ω = {sequences of numbers between 1}
and 5, and then a 6}.
|Ω| = ∞.
An event: E = {roll 4 first, get 6 on the third roll}
= {(4, k, 6) : k = 1 . . . 5}.
|E| = 5.

21
32//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

1. Examples
Example
Experiment: Lifetime of a device (measured in years).
Sample space: Ω = [0, ∞)
|Ω| = ∞ (uncountable).
An event: E = {shouldn’t have bought it} = {0}
|E| = 1.
Another event: F = {device lasts for at least 5 years}
= [5, ∞).
|F | = ∞.
Another event: G = {device is dead by its 6th birthday}
= [0, 6).
|G| = ∞.
Notice: F ∩ G = [5, 6), F ∪ G = [0, ∞) = Ω.
22
33//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

2. The union and the intersection

Inspired by the above:

Remark
The union E ∪ F of events E and F always means E OR F .
The intersection E ∩ F of events E and F always means E
AND F .

Similarly:

Remark
S
The union i Ei of events Ei always means at least one of the
Ei ’s. T
The intersection i Ei of events Ei always means each of the
Ei ’s.

23
34//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

2. The union and the intersection

Definition
If E ∩ F = ∅, then we say that the events E and F are
mutually exclusive events.

If the events E1 , E2 , . . . satisfy Ei ∩ Ej = ∅ whenever i 6= j, then

we say that the Ei ’s are mutually exclusive events.

Mutually exclusive events cannot happen at the same time.

24
35//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

3. Inclusion and implication

Remark
If the event E is a subset of the event F , E ⊆ F , then the
occurrence of E implies that of F .

Example
The experiment is rolling a die.
E = {rolling 1 on a die} ⊆ {rolling an odd no. on a die} = F .
Ω
F
2
3
6 E
1

5 4

E ⊆F

25
36//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

4. Complementary events

Definition
The complement of an event E is E c : = Ω − E. This is the
event that E does not occur.

Ω
E

Ec

Notice: E ∩ E c = ∅, E ∪ E c = Ω.

Notation: sometimes E c = Ē = E ∗ .

26
37//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Commutativity: E ∪ F = F ∪ E,
E ∩ F = F ∩ E.

27
38//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Associativity: E ∪ (F ∪ G) = (E ∪ F ) ∪ G = E ∪ F ∪ G,
E F

E ∩ (F ∩ G) = (E ∩ F ) ∩ G = E ∩ F ∩ G.
E F

28
39//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G

29
40//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
E ∪F

29
41//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
(E ∪ F ) ∩ G

29
42//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
(E ∪ F ) ∩ G E ∩G

29
43//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
(E ∪ F ) ∩ G F ∩G

29
44//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
(E ∪ F ) ∩ G (E ∩ G) ∪ (F ∩ G)

29
45//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
(E ∪ F ) ∩ G (E ∩ G) ∪ (F ∩ G)

(E ∩ F ) ∪ G = (E ∪ G) ∩ (F ∪ G).
E F E F

G G

29
46//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
(E ∪ F ) ∩ G (E ∩ G) ∪ (F ∩ G)

(E ∩ F ) ∪ G = (E ∪ G) ∩ (F ∪ G).
E F E F

G G
E ∩F
29
47//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
(E ∪ F ) ∩ G (E ∩ G) ∪ (F ∩ G)

(E ∩ F ) ∪ G = (E ∪ G) ∩ (F ∪ G).
E F E F

G G
(E ∩ F ) ∪ G
29
48//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
(E ∪ F ) ∩ G (E ∩ G) ∪ (F ∩ G)

(E ∩ F ) ∪ G = (E ∪ G) ∩ (F ∪ G).
E F E F

G G
(E ∩ F ) ∪ G E ∪G
29
49//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
(E ∪ F ) ∩ G (E ∩ G) ∪ (F ∩ G)

(E ∩ F ) ∪ G = (E ∪ G) ∩ (F ∪ G).
E F E F

G G
(E ∩ F ) ∪ G F ∪G
29
50//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

Distributivity: (E ∪ F ) ∩ G = (E ∩ G) ∪ (F ∩ G),
E F E F

G G
(E ∪ F ) ∩ G (E ∩ G) ∪ (F ∩ G)

(E ∩ F ) ∪ G = (E ∪ G) ∩ (F ∪ G).
E F E F

G G
(E ∩ F ) ∪ G (E ∪ G) ∩ (F ∪ G)
29
51//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

30
52//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

E ∪F

30
53//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

(E ∪ F )c

30
54//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

(E ∪ F )c Ec

30
55//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

(E ∪ F )c Fc

30
56//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

(E ∪ F )c Ec ∩ F c

30
57//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

(E ∪ F )c Ec ∩ F c
(E ∩ F )c = E c ∪ F c .
E F E F

30
58//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

(E ∪ F )c Ec ∩ F c
(E ∩ F )c = E c ∪ F c .
E F E F

E ∩F

30
59//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

(E ∪ F )c Ec ∩ F c
(E ∩ F )c = E c ∪ F c .
E F E F

(E ∩ F )c

30
60//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

(E ∪ F )c Ec ∩ F c
(E ∩ F )c = E c ∪ F c .
E F E F

(E ∩ F )c Ec

30
61//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

(E ∪ F )c Ec ∩ F c
(E ∩ F )c = E c ∪ F c .
E F E F

(E ∩ F )c Fc

30
62//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

5. Simple properties of events

De Morgan’s Law: (E ∪ F )c = E c ∩ F c .
E F E F

(E ∪ F )c Ec ∩ F c
(E ∩ F )c = E c ∪ F c .
E F E F

(E ∩ F )c Ec ∪ F c

30
63//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Probability

Finally, we can now define what probability is.

Definition (axioms of probability)

The probability P on a sample space Ω assigns numbers to
events of Ω in such a way, that:
1. the probability of any event is non-negative: P{E} ≥ 0;
2. the probability of the sample space is one: P{Ω} = 1;
3. for any finitely or countably infinitely many mutually
exclusive events E1 , E2 , . . .,
n[ o X
P Ei = P{Ei }.
i i

32
64//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Probability

Notation:
n
[
Ei = E1 ∪ E2 ∪ · · · ∪ En , or
i=1
[∞
Ei = E1 ∪ E2 ∪ . . . ,
i=1
n
X
P{Ei } = P{E1 } + P{E2 } + · · · + P{En } , or
i=1
∞
X
P{Ei } = P{E1 } + P{E2 } + . . . .
i=1

33
65//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

A few simple facts

Proposition
For any event, P{E c } = 1 − P{E}.

Proof.
We know that E and E c are mutually exclusive, and
E ∪ E c = Ω. Therefore by Axiom 3, and then 2,

P{E} + P{E c } = P{E ∪ E c } = P{Ω} = 1.

Corollary
We have P{∅} = P{Ωc } = 1 − P{Ω} = 1 − 1 = 0.

For any event E, P{E} = 1 − P{E c } ≤ 1.

34
66//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

A few simple facts

Proposition (Inclusion-exclusion principle)

For any events E and F , P{E ∪ F } = P{E} + P{F } − P{E ∩ F }.

Proposition (Boole’s inequality)

For any events E1 , E2 , . . . , En ,
n[
n o Xn
P Ei ≤ P{Ei }.
i=1 i=1

35
67//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

A few simple facts

Proof by induction.
When n = 2,

P{E1 ∪ E2 } = P{E1 } + P{E2 } − P{E1 ∩ E2 } ≤ P{E1 } + P{E2 }.

Now suppose true for n. Then

[ o
nn+1 n [
n  o n[
n o
P Ei = P Ei ∪ En+1 ≤ P Ei + P{En+1 }
i=1 i=1 i=1
n
X n+1
X
≤ P{Ei } + P{En+1 } = P{Ei }.
i=1 i=1

36
68//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

A few simple facts

Proposition (Inclusion-exclusion principle)

For any events E, F , G,

P{E ∪ F ∪ G} = P{E} + P{F } + P{G}

− P{E ∩ F } − P{E ∩ G} − P{F ∩ G}
+ P{E ∩ F ∩ G}.

37
69//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

A few simple facts

Example
In the sports club,
36 members play tennis, 22 play tennis and squash,
28 play squash, 12 play tennis and badminton,
18 play badminton, 9 play squash and badminton,
4 play tennis, squash and badminton.

How many play at least one of these games?

38
70//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

A few simple facts

Example
36 members play tennis, 22 play tennis and squash,
28 play squash, 12 play tennis and badminton,
18 play badminton, 9 play squash and badminton,
4 play tennis, squash and badminton.

Solution
Introduce probability by picking a random member out of those
N enrolled to the club. Then
T : = {that person plays tennis},
S : = {that person plays squash},
B : = {that person plays badminton}.

39
71//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

A few simple facts

Example
36 members play tennis, 22 play tennis and squash,
28 play squash, 12 play tennis and badminton,
18 play badminton, 9 play squash and badminton,
4 play tennis, squash and badminton.

Solution (. . . cont’d)

P{T ∪ S ∪ B} = P{T } + P{S} + P{B}

− P{T ∩ S} − P{T ∩ B} − P{S ∩ B}
+ P{T ∩ S ∩ B}
36 28 18 22 12 9 4 43
= + + − − − + = .
N N N N N N N N
Our answer is therefore 43 members.
40
72//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

A few simple facts

Proposition (Inclusion-exclusion principle)
For any events E1 , E2 , . . . , En ,
X
P{E1 ∪ E2 ∪ · · · ∪ En } = P{Ei }
1≤i≤n
X
− P{Ei1 ∩ Ei2 }
1≤i1 <i2 ≤n
X
+ P{Ei1 ∩ Ei2 ∩ Ei3 }
1≤i1 <i2 <i3 ≤n

− ···
+ (−1)n+1 P{E1 ∩ E2 ∩ · · · ∩ En }.

41
73//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

A few simple facts

Proposition

If E ⊆ F , then P{F − E} = P{F } − P{E}.

Corollary
If E ⊆ F , then P{E} ≤ P{F }.

Example
E = {rolling 1 on a die} ⊆ {rolling an odd no. on a die} = F .
Ω
F
2
3
6 E
1
6 = P{E} ≤ P{F } = 12 . 1

5 4

E ⊆F
42
74//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Equally likely outcomes

A very special but important case is when the sample space is
finite: |Ω| = N < ∞, and each outcome of our experiment has
equal probability. Then necessarily this probability equals N1 :

1
P{ω} = ∀ω ∈ Ω.
N

Definition
These outcomes ω ∈ Ω are also called elementary events.

Let E ⊆ Ω be an event that consists of k elementary events:

|E| = k. Then
|E| k
P{E} = = .
|Ω| N
We thus see why counting will be important.
43
75//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Equally likely outcomes

Example
Rolling two dice, what is the probability that the sum of the
numbers shown is 7?

Defining E = {sum is 7} = {(1, 6), (2, 5), . . . , (6, 1)} in the

sample space Ω = {(i, j) : i, j = 1 . . . 6}, and noting that each
pair of numbers is equally likely, we have

|E| 6 1
P{E} = = = .
|Ω| 36 6

A wrong solution would be to say that 7 is one out of the

1
possible values 2, 3, . . . , 12 for the sum, and the answer is 11 .
These sums are not equally likely, e.g., 12 only occurs as one
case out of 36. The pairs of numbers above are equally likely.
44
76//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Equally likely outcomes

Often there are more than one ways of solving a probability

problem. Sometimes we can count in order, or without order.

Example
An urn contains 6 red and 5 blue balls. We draw three balls at
random, at once (that is, without replacement). What is the
chance of drawing one red and two blue balls?

45
77//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Equally likely outcomes

Solution (with order)

Define the sample space Ω as ordered choices of 3 out of the
11 balls (3-permutations of them). Then each choice is equally
11!
likely, and |Ω| = (11−3)! = 11 · 10 · 9. Now, our event E consists
of:
◮ drawing red-blue-blue, in 6 · 5 · 4 many ways, or
◮ drawing blue-red-blue, in 5 · 6 · 4 many ways, or
◮ drawing blue-blue-red, in 5 · 4 · 6 many ways.
Thus, |E| = 6 · 5 · 4 + 5 · 6 · 4 + 5 · 4 · 6 = 3 · 6 · 5 · 4, and the
answer is
|E| 3·6·5·4 4
P{E} = = = .
|Ω| 11 · 10 · 9 11

46
78//281
79
 Prob. Cond. Discr. Cont. Joint E, cov LLN, CLT Combi. Sample sp. Probability Equally l.

Equally likely outcomes

Solution (without order)

Define the sample space Ω as unordered choices of 3 out of
the 11 balls (3-combinations
of 11·10·9
them). Then each choice is
equally likely, and |Ω| = 11
3 = 6 . Now, our event E
consists of picking 1 out of the 6 red balls and 2 out of the 5
blue balls,

with
no respect to order. Thus,
|E| = 61 · 52 = 6 · 10, and the answer is

|E| 6 · 10 4
P{E} = = = .
|Ω| 11 · 10 · 9/6 11

Both solutions are fine. Sometimes in order is easier,

sometimes harder than without order. But never mix the two.

47
79//281
79