STAT3903 Stochastic Models The University of Hong Kong

The Mid-Term Test Solution to Questions

March 2018 Signature

1. Let (X, Y ) be a pair of random variables with values in {1, 2, 3, . . .} × (0, 1)such that

• Y ∼ Beta(α, β), a beta distribution with parameters α > 2 and β > 1;

• Given Y = y ∈ (0, 1), X has the geometrical distribution with parameter y.

Calculate E(X) and var(X). [10 marks]

Hint. (1) Geometric distribution with success probability p has mean 1/p and variance
(1 − p)/p1 . (2) Beta(α,β) distribution has density function
α αβ
b(α, β)xα−1 (1 − x)β−1 1(0,1) (x), mean and variance 2
.
α+β (α + β) (α + β + 1)
Γ(α + β)
Here b(α, β) = .
Γ(α)Γ(β)

Solution.
1−Y
Since X|Y = y ∼ Geoy, E(X|Y ) = 1/Y and var(X|Y ) = Y .

1 b(α, β) Γ(α + β) Γ(α − 1)Γ(β) α+β−1

• E[X] = E[E[X|Y ]] = E[ ]= = · = .
Y b(α − 1, β) Γ(α)Γ(β) Γ(α + β − 1) α−1
• Using var(X) = E[var(X|Y )] + var(E(X|Y )):

1−Y b(α, β) Γ(α + β) Γ(α − 1)Γ(β + 1) β

E[var(X|Y )] = E[ ]= = · = .
Y b(α − 1, β + 1) Γ(α)Γ(β) Γ(α + β) α−1

For var(E(X|Y )) = var( Y1 ):

1 b(α, β) Γ(α + β) Γ(α − 2)Γ(β) (α + β − 1)(α + β − 2)

· E( 2
)= = · = ,
Y b(α − 2, β) Γ(α)Γ(β) Γ(α + β − 2) (α − 1)(α − 2)
α+β−1 2
 
1 (α + β − 1)(α + β − 2) β(α + β − 1)
· var( ) = − =
Y (α − 1)(α − 2) α−1 (α − 1)2 (α − 2)

So finally,

β β(α + β − 1)
var(X) = + .
α − 1 (α − 1)2 (α − 2)

1−Y
Remark. With the correct formula of var(X|Y ) = Y2
, we have

1−Y b(α, β) Γ(α + β) Γ(α − 2)Γ(β + 1) β(α + β − 1)

E[var(X|Y )] = E[ 2
]= = · = .
Y b(α − 2, β + 1) Γ(α)Γ(β) Γ(α + β − 1) (α − 1)(α − 2)

And
β(α + β − 1) β(α + β − 1) αβ(α + β − 1)
var(X) = + = .
(α − 1)(α − 2) (α − 1)2 (α − 2) (α − 1)2 (α − 2)

1
Typo: the variance is in fact (1 − p)/p2 but we will continue with this (wrong) assumption.

1
2. Consider a Markov chain consisting of the six states 0, 1, 2, 3, 4, 5, and having transition
probability matrix

state 0 1 2 3 4 5
 
0 0.2 0 0.2 0 0 0.6
 
1  0.6 0.2 0.2 0 0
 0 
P =
 
2  0
 0 0 0 1 0 .
3 0.25 0.25 0.25 0 0 0.25
 
 
 
4  0
 0 1 0 0 0 

5 0 0 0 0 0 1

Specify the classes and determine whether they are transient or recurrent.

[16 marks]

Solution.

• Class {0}: pn00 = 0.2n for all n ≥ 1, so n

P
n p00 < ∞ and “0” is transient.
• Class {1}: pn11 = 0.2n for all n ≥ 1, so n pn11 < ∞ and “1” is transient.
P

• Class {3}: pn33 = 0 for all n ≥ 1, so n pn33 < ∞ “3” is transient.

P

• Class {5}: Absorbing state, recurrent ( pn55 = 1 for all n ≥ 1, so n pn55 = ∞).
P

• Class {2, 4}: p2k n 2k

P P
22 = 1 for all k ≥ 1, so n≥1 p22 = k≥1 p22 = ∞, and the state “2”
then the class is recurrent.

3. Jerry the mouse likes very much resting at the four corners of a square table. The corners
are labeled as A, B, C and D (clockwise). Once a minute, she moves from where she is
to one of the two nearest corners randomly with equal probability.

• Give the transition matrix of the Markov chain representing the successive positions
of Jerry.
• Is the Markov chain ergodic?
• Find the limiting probabilities of the chain.
• Initially, Jerry is at corner D. Find the expected number of moves until she returns
to D.

[24 marks]

2
Solution.

• The transition matrix is

state A B C D
 
1 1
A 0 2 0 2
 
P = B  1 0 1
0 
 2 2
.

1 1 

C  0
 2 0 2 
1 1
D 2 0 2 0
• Clearly this is only one class: the chain is irreducible.
2k+1
But the chain is 2-periodic: indeed PAA = 0, that is return to a state is possible
only after an even number of moves.
The chain is then not ergodic (ergodic = irreducible + aperiodic).
n will not has a limit when n → ∞. There are no
• Since the chain is periodic, PAA
limiting probabilities.
Remark. Invariant probabilities do exist. By symmetry the equation πP = π has
an unique solution given by π = ( 14 , 14 , 14 , 14 ), that is, the uniform distribution on the
four states.
• We consider a related chain by making D an absorbing state with transition matrix:
state A B C D
 
1 1
A 0 2 0 2
 
P0 = B  1
 2 0 1
2 0 
.

1 1 

C  0
 2 0 2 
D 0 0 0 1
For this Markov chain, the three states A, B and C are transient. The associated
S-matrix is found as follows;
     
0 12 0 1 − 12 0 3
1 1

0
2 2
S 0 = (I − PT0 )−1 = 
   
1 1  1
I − PT =   2 0 2  = − 2 1 − 21 
, 1 2 1 ,

0 21 0 0 − 12 1 1 3
2 1 2

If we call m0iD the expected number of moves before absoption at D when this chain
starts at i, then we have
3 1
m0AD = SAA
0 0
+ SAB 0
+ SAC = + 1 + = 3,
2 2
1 3
m0CD = SCA
0 0
+ SCB 0
+ SCC = + 1 + = 3.
2 2
Now we go back to the original chain. By first-step analysis, for the first step, Jerry
will move to A or C with probability 12 . Then the expected number before returning
to D is equal to
1 1
E(N ) = (1 + m0AD ) + (1 + m0CD ) = 4.
2 2
Note. As an alternative solution, one can apply First Step Analysis to derive a system
of equations involving first entrance times to state D with different initial states.