Scribd
Upload
40 views

Markov Chain - HW4

This document contains 8 practice problems related to stochastic modeling and Markov chains. The problems cover calculating conditional probabilities for Markov chains, determining probabilities of Markov chain state sequences, finding mean times to absorption for Markov chains, calculating probabilities of reaching absorbing states, analyzing properties of random walks, and analyzing branching processes.

Uploaded by

Diego Salazar
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
40 views

Markov Chain - HW4

This document contains 8 practice problems related to stochastic modeling and Markov chains. The problems cover calculating conditional probabilities for Markov chains, determining probabilities of Markov chain state sequences, finding mean times to absorption for Markov chains, calculating probabilities of reaching absorbing states, analyzing properties of random walks, and analyzing branching processes.

Uploaded by

Diego Salazar
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Math 543/653: Stochastic Modeling )

HW #4
Instructor: Songfeng (Andy) Zheng

Problem 1. A Markov Chain X0 , X1 , X2 , · · · has the transition probability matrix



0.3 0.2 0.5

P = 0.5 0.1 0.4


0.5 0.2 0.3

The Markov chain has state space {0, 1, 2}.


(a). Determine the conditional probability P (X3 = 1|X0 = 0) and P (X3 = 1|X1 = 0).
(b). The initial distribution is p0 = 0.5 and p1 = 0.5. Please find

P (X0 = 1, X1 = 1, X2 = 0)

and
P (X1 = 1, X2 = 1, X3 = 0)

(c). The initial distribution is p0 = 0.5 and p1 = 0.5. Please determine the probabilities
P (X2 = 0) and P (X3 = 0).

Problem 2. Consider a Markov chain with state space {0, 1, 2, 3}. Suppose the transition
probability matrix is
0.4 0.3 0.2 0.1

0.1 0.4 0.3 0.2
P = .

0.3 0.2 0.1 0.4

0.2 0.1 0.4 0.3

If the initial distribution is pi = 1/4, for i = 0, 1, 2, 3. Show that for all n, P (Xn = k) = 1/4,
for k = 0, 1, 2, 3. Can you generalize this result from this example? Prove your conjecture.

Problem 3. Consider a Markov chain with state space {0, 1, 2, 3}, and the transition prob-
ability matrix is
1 0 0 0

0.1 0.6 0.1 0.2
P = .

0.2 0.3 0.4 0.1

0 0 0 1

1
2

(a) Starting in state 1, determine the probability that the Markov chain ends in state 0.
(b) Determine the mean time to absorption.

Problem 4. Consider a Markov chain with state space {0, 1, 2, 3, 4}, and the transition
probability matrix is
q p 0 0 0

q 0 p 0 0



P = q 0 0 p 0

q 0 0 0 p


0 0 0 0 1

where p + q = 1. Determine the mean time to reach state 4 starting from state 0. That is,
find E[T |X0 = 0], where T = min{n ≥ 0; Xn = 4}.

Problem 5. Consider a Markov chain with state space {0, 1, 2}, and the transition proba-
bility matrix is
0.3 0.2 0.5

P = 0.5 0.1 0.4 .

0 0 1
The process starts with X0 = 0. Eventually the process will end up in state 2. What is the
probability that when the process moves into state 2, it does so from state 1?
Hint: let T = min{n ≥ 0; Xn = 2}, then the asked case happens if XT −1 = 1, considering
the initial states, we want to find P (XT −1 = 1|X0 = 0). Let zi = P (XT −1 = 1|X0 = i), use
first step analysis to find the desired result.

Problem 6. Consider a general one dimensional random walk on {0, 1, 2, · · · , N } with


transition probability matrix


1 0 0 0 ···
0
q1 r 1 p 1 0 ··· 0



P = 0 q2 r 2 p 2 ···
0
.. .. .. .. .. ..

. . . . .
.
0 0 0 0 ··· 1

where pk > 0, qk > 0, and rk ≥ 0 for k = 1, 2, · · · , N − 1. Let T = min{n ≥ 0; Xn =


0 or Xn = N }. Consider the probability of gambler’s ruin
ui = P (XT = 0|X0 = i).
Use first step analysis to prove that, for i = 1, · · · , N − 1:
ρi + · · · + ρN −1
ui =
1 + ρ1 + ρ2 + · · · + ρN −1
3

where
q1 q2 · · · qk
ρk = .
p1 p2 · · · pk

Problem 7. The number of offspring of an individual in a population is 0, 1, or 2 with


probabilities a > 0, b > 0, and c > 0, a + b + c = 1. Express the mean and variance of the
population in step n.

Problem 8. Let {Xn } be a branching process with E(ξ) = µ. Show that Zn = Xn /µn is a
nonnegative martingale.

You might also like