Probability for Data Science – Master’s Degree in Data Science – A.Y.

2023/2024
Academic Staff: Francesca Collet, Paolo Dai Pra

EXERCISE CLASS

Exercise 1. At the foyer of a bank there are two ATMs. Each machine can be used by one person
at a time, so a queue forms when three or more customers are present. The foyer is limited in
size and can hold only five persons. Arriving customers will leave if the foyer is full. The service
times of the two ATMs are independent and exponentially distributed with rate 3 customers per
minute; while customers arrive at the bank according to a Poisson process with rate 2 persons per
minute.
Define an appropriate continuous-time Markov chain to describe the number of customers at the
foyer; give the holding time parameters and the transition matrix of the embedded jump chain.

Solution. Let
X(t) = number of customers at the foyer at time t
denote the state of the system at time t. The stochastic process (X(t))t≥0 is a continuous-time
Markov chain on S = {0, 1, 2, 3, 4, 5}. We have to determine the holding time parameters and the
transition matrix of the embedded jump chain.
Let Ts be the service times of an ATM and let Ta be the customer arrival times. We have
Ts ∼ Exp(3) and Ta ∼ Exp(2). We obtain the following characterization.

Holding time parameters:

v0 = 2 (the waiting time for an arrival is Ta ∼ Exp(2))

(the waiting time for either a departure or an arrival is min{Ts , Ta } ∼ Exp(5), since only
v1 = 5 one ATM is busy when the system is in state 1)
n o
(1) (2)
v2 = v3 = v4 = 8 (the waiting time for either a departure or an arrival is min Ts , Ts , Ta ∼ Exp(8), since
both ATMs are busy when the system is in state i ∈ {2, 3, 4})
n o
(1) (2)
v5 = 6 (the waiting time for a departure is min Ts , Ts ∼ Exp(6))

Jump chain transition matrix (non-zero entries):

P01 = 1
3
P10 = P (Ts < Ta ) = P (Exp(3) < Exp(2)) =
5
2
P12 = P (Ta < Ts ) = P (Exp(2) < Exp(3)) =
5
 n o  3
Pi,i−1 = P min Ts(1) , Ts(2) < Ta = P (Exp(6) < Exp(2)) = , if i ∈ {2, 3, 4}
4
 n o 1
Pi,i+1 = P Ta < min Ts(1) , Ts(2) = P (Exp(2) < Exp(6)) = , if i ∈ {2, 3, 4}
4
P54 = 1,
that is  
0 1 0 0 0 0
3 0 2
0 0 0
5 5 

0 3 1 
4 0 4 0 0
P=
0 3 1
.
 0 4 0 4 0
 3 1
0 0 0 4 0 4

0 0 0 0 1 0

Remark. Observe that the process is a birth and death process. In particular, (X(t))t≥0 is a birth
and death process on S with birth rates
λ0 = λ1 = λ2 = λ3 = λ4 = 2 (the waiting time for an arrival is Ta ∼ Exp(2))

and with death rates

µ1 = 3 (the service time of a busy ATM is Ts ∼ Exp(3))

n o
(1) (2)
µ2 = µ3 = µ4 = µ5 = 6 (the waiting time for a departure is min Ts , Ts ∼ Exp(6)).

Exercise 2. Consider a culture of two one-celled organisms. Each organism can be in one of two
states–either A or B. An individual in state A will change to state B at an exponential rate 1;
whereas, an individual in state B will change to state A at an exponential rate 2. We are interested
in the number of organisms in state A.
(a) Define an appropriate continuous-time Markov chain for this model; give the holding time
parameters and the transition matrix of the embedded jump chain.
(b) If it exists and it is unique (explain why or why not!), determine the stationary distribution
of the embedded chain. Do limiting probabilities for the jump chain exist?
(c) Determine the stationary distribution of the continuous-time Markov chain. In the long-run,
what is the average number of cells in state A?
Solution.
(a) Let
X(t) = number of organisms in state A at time t
denote the state of the system at time t. The stochastic process (X(t))t≥0 is a continuous-
time Markov chain on S = {0, 1, 2}. We have to determine the holding time parameters and
the transition matrix of the embedded jump chain.

Holding time parameters:

(the waiting time for one of the two cells in state B to change its state to A
v0 = 4 is the minimum between two independent exponential random variables both
with parameter 2)

(the waiting time for either a cell in state B to change its state to A or a cell
v1 = 3 in state A to change its state to B is the minimum between two independent
exponential random variables with respective parameters 2 and 1)

(the waiting time for one of the two cells in state A to change its state to B
v2 = 2 is the minimum between two independent exponential random variables both
with parameter 1)
 Jump chain transition matrix (non-zero entries):

1
P01 = 1 P10 = P (Exp(1) < Exp(2)) =
3
2
P12 = P (Exp(2) < Exp(1)) = P21 = 1,
3
that is  
0 1 0
P =  13 0 2
3 .


0 1 0

(b) From the transition matrix P we deduce

2
the transition graph shown in the figure 1 3

aside. All states communicate with each " "

other (0 → 1 → 2 → 1 → 0), therefore •0 b •1 b •2
the jump chain has only one communica-
tion class. 1 1
3

The embedded chain is irreducible. Moreover, as the state space is a finite set, it is positive
recurrent. As a consequence, we have existence and uniqueness of the stationary distribution
for this chain. To determine the stationary distribution π̃, we solve the following system of
linear equations (
π̃ = π̃ P
P2
i=0 π̃i = 1,

which is equivalent to

 π̃0 = 31 π̃1 
1
 π̃0 =

 

 π̃ = π̃ + π̃  6
1 0 2 1
2
⇐⇒ π̃1 = 2

 π̃2 = 3 π̃1 

 π̃ = 1
3.

 2
π̃0 + π̃1 + π̃2 = 1


The limiting probabilities for the embedded jump chain do not exist, as it is a periodic chain
with period d = 2 (an even number of steps is needed to reenter any given state).
(c) Method 1. We get the stationary distribution π of the continuous-time process via the
stationary distribution π̃ of the discrete-time embedded Markov chain (see lecture 39). Since
P2
we calculate i=0 π̃vii = 38 , we obtain
1 1 1
24 1 6 4 6 4
π0 = 3 = , π1 = 3 = and π2 = 3 = .
8
9 8
9 8
9

Method 2. We analyze the process for what it is, a birth and death process, and we use
directly the expression for the stationary distribution we derived in class (see lecture 40). In
particular, (X(t))t≥0 is a birth and death process on S with birth rates
(the waiting time for one of the two cells in state B to change its state to A
λ0 = 4 is the minimum between two independent exponential random variables both
with parameter 2)

(the waiting time for the only cell in state B to change its state to A is an
λ1 = 2 exponential random variable with parameter 2)
and with death rates
(the waiting time for the only cell in state A to change its state to B is an
µ1 = 1 exponential random variable with parameter 1)

(the waiting time for one of the two cells in state A to change its state to B
µ2 = 2 is the minimum between two independent exponential random variables both
with parameter 1)

Therefore, we obtain
1 1 1
π0 = λ0 λ0 λ1
= 4 4·2 =
1+ µ1 + µ1 µ2
1+ 1 + 1·2
9

λ0 4 4
π1 =  = =
µ1 1 + λ0
+ λ0 λ1 1·9 9
µ1 µ1 µ2

λ0 λ1 4·2 4
π2 =  = = .
µ1 µ2 1 + λ0
+ λ0 λ1 1·2·9 9
µ1 µ1 µ2

As a consequence, in the long-run, the average number of cells in state A is given by

4 4 4
π1 + 2π2 = + 2 · = ≈ 1.33.
9 9 3