HOMEWORK 4: SOLUTIONS

1.

A Markov chain with state space {1, 2, 3} has transition probability matrix
 
1/3 1/3 1/3
P =  0 1/2 1/2
0 0 1

Show that state 3 is absorbing and, starting from state 1, find the expected time until
absorption occurs.
Solution. Let ψ(i) be the expected time to reach state 3 starting from state i, where
i ∈ {1, 2, 3}. We have

ψ(3) = 0
1 1
ψ(2) = 1 + ψ(2) + ψ(3)
2 2
1 1 1
ψ(1) = 1 + ψ(1) + ψ(2) + ψ(3).
3 3 3
We solve and find ψ(3) = 0, ψ(2) = 2, ψ(1) = 5/2.

2.
Smith is in jail and has 3 dollars; he can get out on bail if he has 8 dollars. A guard
agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars
with probability 0.4 and loses A dollars with probability 0.6. Find the probability that
he wins 8 dollars before losing all of his money if
(a) he bets 1 dollar each time (timid strategy).
(b) he bets, each time, as much as possible but not more than necessary to bring his
fortune up to 8 dollars (bold strategy).
(c) Which strategy gives Smith the better chance of getting out of jail?
Solution. (a) The Markov chain (Xn , n = 0, 1, . . .) representing the evolution of
Smith’s money has diagram

0.4 0.4 0.4 0.4 0.4 0.4 0.4

1
1 0 1 2 3 4 5 6 7 8
0.6 0.6 0.6 0.6 0.6 0.6 0.6

Let ϕ(i) be the probability that the chain reaches state 8 before reaching state 0,
starting from state i. In other words, if Sj is the first n ≥ 0 such that Xn = j,

ϕ(i) = Pi (S8 < S0 ) = P (S8 < S0 |X0 = i).

Using first-step analysis (viz. the Markov property at time n = 1), we have

ϕ(i) = 0.4ϕ(i + 1) + 0.6ϕ(i − 1), i = 1, 2, 3, 4, 5, 6, 7

ϕ(0) = 0
ϕ(8) = 1.

1
 We solve this system of linear equations and find

ϕ = (ϕ(1), ϕ(2), ϕ(3), ϕ(4), ϕ(5), ϕ(6), ϕ(7))

= (0.0203, 0.0508, 0.0964, 0.1649, 0.2677, 0.4219, 0.6531, 1).

E.g., the probability that the chain reaches state 8 before reaching state 0, starting
from state 3 is the third component of this vector and is equal to 0.0964. Note that
ϕ(i) is increasing in i, which was expected.
(b) Now the chain is

0.6 0.4
0.4
1
1 0 1 2 3 4 5 6 7 8

0.6
0.6
0.4

and the equations are:

ϕ(3) = 0.4ϕ(6)
ϕ(6) = 0.4ϕ(8) + 0.6ϕ(4)
ϕ(4) = 0.4ϕ(8)
ϕ(0) = 0
ϕ(8) = 1.

We solve and find

ϕ = (0, 0, 0.256, 0.4, 0, 0.64, 0).

(c) By comparing the third components of the vector ϕ we find that the bold strategy
gives Smith a better chance to get out jail.

3.
A fair coin is tossed repeatedly and independently. Find the expected number of tosses
till the pattern HTH appears.
Solution. Call HTH our target. Consider a chain that starts from a state called
nothing ∅ and is eventually absorbed at HTH. If we first toss H then we move to state
H because this is the first letter of our target. If we toss a T then we move back to ∅
having expended 1 unit of time. Being in state H we either move to a new state HT if
we bring T and we are 1 step closer to the target or, if we bring H, we move back to
H: we have expended 1 onit of time, but the new H can be the beginning of a target.
When in state HT we either move to HTH and we are done or, if T occurs then we
move to ∅. The transition diagram is
1/2 1

H HT HTH
1/2 1/2 1/2

1/2 1/2

2
 Rename the states ∅, H, HT, HTH as 0, 1, 2, 3, respectively. Let ψ(i) be the expected
number of steps to reach HTH starting from i. We have
1
ψ(2) = 1 + ψ(0)
2
1 1
ψ(1) = 1 + ψ(1) + ψ(2)
2 2
1 1
ψ(0) = 1 + ψ(0) + ψ(1).
2 2
We solve and find ψ(0) = 10.
4.
A. Assume that an experiment has m equally probable outcomes. Show that the
expected number of independent trials before the first occurrence of k consecutive
occurrences of one of these outcomes is
mk − 1
.
m−1
Hint: Form an absorbing Markov chain with states 1, 2, . . . , k with state i representing
the length of the current run. The expected time until a run of k is 1 more than the
expected time until absorption for the chain started in state 1.
B. How many tosses of a fair die are required, on the average, until 10 consecutive sixes
are obtained?
C. It has been found that, in the decimal expansion of π = 3.14159 . . ., starting with
the 24,658,601st digit, there is a run of nine 7’s. What would your result say about
the expected number of digits necessary to find such a run if the digits are produced
randomly?
Solution. A. Let the outcomes be a, b, c, . . . (m of them in total). Suppose that a is
the desirable outcome. We set up a chain as follows. Its states are
∅, (a), (aa), (aaa), ..., (aa · · · a)
| {z }
k times

Or, more simply, 0, 1, 2, . . . , m. State k means that you are currently at the end of
a run of k a’s. If you see an extra a (with probability 1/m) you go to state k + 1.
Otherwise, you go to ∅. Let ψ(k) be the expected number of steps till state m is
reached, starting from state k:
ψ(k) := Ek Sm .
We want to find ψ(0). We have
ψ(k) = 1 + (1 − 1/m)ψ(0) + (1/m)ψ(k + 1).
Solving these, we find
mk − 1
ψ(0) = 1 + m + m2 + · · · + mk−1 = .
m−1
B. So to get 10 consecutive sixes by rolling a die, you need more than 12 million rolls
on the average (12, 093, 235 rolls to be exact).
C. They are not random. If they were, we expect to have to pick (109 − 1)/9 digits
before we see nine consecutive sevens. That’s about 100 million digits. The actual
position (24 million digits) is one fourth of the expected one.

3
5.
A rat runs through the maze shown below. At each step it leaves the room it is in by
choosing at random one of the doors out of the room.

2 3 4

5 6

(a) Find the transition matrix for this Markov chain.

(b) Show that it is irreducible and find its period.
(c) Find the stationary distribution.
(d) Now suppose that a piece of mature cheddar is placed on a deadly trap in Room 5.
The rat starts in Room 1. Find the expected number of steps before reaching Room 5
for the first time, starting in Room 1.
(e) Supppose again that the rat starts in Room 1 and moves until it dies in Room 5.
Every time the rat is in room i it gnaws an amount of wood equal to i2 . Find the
expected amount of wood gnawed by the rat before its death. (Assume when it enters
room 5 it dies immediately before chewing any wood at all.)
(f) Supppose again that the rat starts in Room 1 but there is no trap in Room 5.
Explain why it will, for sure, return to room 1. Find the average time until it returns
to room 1.
Solution
(a) The transition matrix P for this Markov chain is as follows:
 
0 0 1 0 0 0
 0 0 1 0 0 0 
 
1/4 1/4 0 1/4 1/4 0 
P=  0
.
 0 1/2 0 0 1/2
 0 0 1/2 0 0 1/2
0 0 0 1/2 1/2 0

(b) The chain is irreducible, because it is possible to go from any state to any other
(n)
state. However, it is not aperiodic, because for any n even p6,1 will be zero and for
(n)
any n odd p6,5 will also be zero (why?). This means that there is no power of P that
would have all its entries strictly positive.
(c) The stationary distribution is
1 1 4 2 2 2
π=( , , , , , ).
12 12 12 12 12 12

You should carry out the calculations and check that this is correct.
(d) Let
ψ(i) = E(number of steps to reach state 5 | X0 = i).

4
 We have

ψ(5) = 0
ψ(6) = 1 + (1/2)ψ(5) + (1/2)ψ(4)
ψ(4) = 1 + (1/2)ψ(6) + (1/2)ψ(3)
ψ(3) = 1 + (1/4)ψ(1) + (1/4)ψ(2) + (1/4)ψ(4) + (1/4)ψ(5)
ψ(1) = 1 + ψ(3)
ψ(2) = 1 + ψ(3).

We solve and find ψ(1) = 7.

(e) Let h(i) be the total amount of wood gnawed by the rat before its death if the rat
starts its rat-race in room i. We have

h(5) = 0
h(6) = 36 + (1/2)h(5) + (1/2)h(4)
h(4) = 16 + (1/2)h(6) + (1/2)h(3)
h(3) = 9 + (1/4)h(1) + (1/4)h(2) + (1/4)h(4) + (1/4)h(5)
h(1) = 1 + h(3)
h(2) = 4 + h(3).

We solve and find

321 275 177 263 259
h(5) = 0, h(6) = , h(2) = , h(4) = , h(1) = , h(3) = .
4 4 2 4 4

(f) The chain is irreducible with a finite number of states. Therefore all states are
recurrent. This means that the rat will return to Room 1 with probability 1. We find,
from π, that the mean recurrence time (i.e. the expected time to return) for the room
1 is 1/π(1)=12.

6.
Two players, A and B, play the game of matching pennies: at each time n, each player
has a penny and must secretly turn the penny to heads or tails. The players then
reveal their choices simultaneously. If the pennies match (both heads or both tails),
Player A wins the penny. If the pennies do not match (one heads and one tails), Player
B wins the penny. Suppose the players have between them a total of 5 pennies. If at
any time one player has all of the pennies, to keep the game going, he gives one back
to the other player and the game will continue.
(a) Show that this game can be formulated as a Markov chain.
(b) Is the chain irreducible? Is it aperiodic?
(c) If Player A starts with 3 pennies and Player B with 2, what is the probability that
A will lose his pennies first?
Solution (a) The problem is easy: The probability that two pennies match is 1/2.
The probability they do not match is 1/2. Let x be the number of pennies that A has.
Then with probability 1/2 he will next have x + 1 pennies or with probability 1/2 he
will next have x − 1 pennies. The exception is when x = 0, in which case, he gets, for
free, a penny from B and he next has 1 penny. Also, if x = 5 he gives a penny to B
and he next has 4 pennies. Thus:

5
 1 1/2 1/2 1/2 1/2

0 1 2 3 4 5

1/2 1/2 1/2 1/2 1

(b) The chain is clearly irreducible. But the period is 2. Hence it is not regular.
(c) To do this, modify the chain and make it stop once one of the players loses his
pennies. After all, we are NOT interested in the behaviour of the chain after this time.
The modification is an absorbing chain:

1/2 1/2 1/2 1/2

0 1 2 3 4 5
1
1
1/2 1/2 1/2 1/2

We then want to compute the absorbing probability ϕ01 (3) where

ϕ01 (i) = Pi (hit 0 before 1).

Write ϕ(i) = ϕ01 (i), for brevity, and apply first-step analysis:

ϕ(0) = 1
1 1
ϕ(1) = ϕ(0) + ϕ(1)
2 2
1 1
ϕ(2) = ϕ(1) + ϕ(2)
2 2
1 1
ϕ(3) = ϕ(2) + ϕ(3)
2 2
1 1
ϕ(4) = ϕ(3) + ϕ(4)
2 2
ϕ(5) = 0.

Six equations with six unknowns. Solve and find: ϕ(3) = 2/5.
Alternatively, observe, from Thales’ theorem,1 that ϕ must be a straight line:

ϕ(x) = ax + b.

From ϕ(0) = 1, ϕ(5) = 0, we find a = −1/5, b = 1, i.e.

ϕ(i) ≡ 1 − (i/5),

which agrees with the above.

Thales’ theorem says (proved around

1 theyear 600 BCE) says that if the lines
L, L′ are parallel then DE AE AD
BC = AC = AB .