Chapter 4

Markov Processes

4.1 Markov Processes and Markov Chains

Recall the following example from Section 3.1.
Two competing Broadband companies, A and B, each currently have 50% of the market share.
Suppose that over each year, A captures 10% of B’s share of the market, and B captures 20%
of A’s share.
This situation can be modelled as follows. Let an and bn denote the proportion of the market
held by A and N respectively at the end of the nth year. We have a0 = b0 = 0.5 (beginning of
Year 1 = end of Year 0).
Now an+1 and bn+1 depend on an and bn according to

an+1 = 0.8an + 0.1bn

bn+1 = 0.2an + 0.9bn

We can write this in matrix form as follows

    
an+1 0.8 0.1 an
 =  .
bn+1 0.2 0.9 bn
 
0.8 0.1
We define M =  . Note that every entry in M is non-negative and that the sum
0.2 0.9
of the entries in each column is 1. This is no accident since the entries in the first column of M
are the respective proportions of A’s market share that are retained and lost respectively by A
from one year to the next. Column 2 contains similar data for B.

62
Definition 4.1.1 A stochastic matrix is a square matrix with the following properties :

(i) All entries are non-negative.

(ii) The sum of the entries in each column is 1.

So the matrix M of our example is stochastic.  

an
Returning to the example, if we let vn denote the vector   describing the position
bn
at the end of year n, we have
 
0.5
v0 =   , v1 = M v0 , vn+1 = M vn .
0.5

Note that the sum of the entries in each vi is 1.

Definition 4.1.2 A column vector with non-negative entries whose sum is 1 is called a prob-
ability vector.

It is not difficult to see that if v is a probability vector and A is a stochastic matrix, then Av
is a probability vector. In our example, the sequence v0 , v1 , v2 , . . . of probability vectors is an
example of a Markov Chain. In algebraic terms a Markov chain is determined by a probability
vector v and a stochastic matrix A (called the transition matrix of the process or chain). The
chain itself is the sequence
v0 , v1 = Av0 , v2 = Av3 , . . .

More generally a Markov process is a process in which the probability of observing a partic-
ular state at a given observation period depends only on the state observed at the preceding
observation period.
Remark: Suppose that A is a stochastic matrix. Then from Item 5 in Section 3.4 it follows that
1 is an eigenvalue of A (all the columns of A sum to 1). The transition matrix in our example
is  
0.8 0.1
M = .
0.2 0.9
 
x
Eigenvectors of M corresponding to the eigenvalue 1 are non-zero vectors   for which
y
    
0.8 0.1 x x
  = 
0.2 0.9 y y

63
Thus 
0.8x + 0.1y = x 
=⇒ y = 2x.
0.2x + 0.9y = y 
 
x
So any non-zero vector of the form   is an eigenvector of M corresponding to the
2x
eigenvalue 1. Amongst all these vectors exactlyone isa probability vector, namely the one
1/3
with x + 2x = 1, i.e. x = 31 . This eigenvector is  
2/3
 
0.5
The Markov process in our example is v0 , v1 , v2 , . . . , where v0 =   and vi+1 = M vi .
0.5
We can observe  
0.3613
v5 = M 5 v0 ≈  
0.6887

 
0.3380
v10 = M 10 v0 ≈  
0.6620

 
0.3335
v20 = M 20 v0 ≈  
0.6665

 
0.3333
v30 = M 30 v0 ≈  
0.6667
 
1/3
So it appears that the vectors in the Markov chain approach the eigenvector   of M
2/3
as the process develops. This vector is called the steady state of the process.
This example is indicative of a general principle.

Definition 4.1.3 A stochastic n × n matrix M is called regular if M itself or some power of

M has all entries positive (i.e. no zero entries).

Example
 
0.8 0.1
• M =  is a regular stochastic matrix.
0.2 0.9

64
  
0 1
• A=  is a stochastic matrix but it is not regular :
1 0
   
1 0 0 1
A2 =   , A3 =   = A, etc
0 1 1 0

The positive powers of A just alternate between I2 and A itself. So no positive integer
power of A is without zero entries.

Theorem 4.1.4 Suppose that A is a regular stochastic n × n matrix. Then

• There is a unique probability vector v for which Av = v.

• If u0 is any probability vector then the Markov chain u0 , u1 , . . . defined for i ≥ 1 by

ui = Aui−1 converges to v.
(This means that for 1 ≤ i ≤ n, the sequence of the ith entries of u0 , u1 , u2 , . . . converges
to the ith entry of v).

Notes

1. Theorem 4.1.4 says that if a Markov process has a regular transition matrix, the process
will converge to the steady state v regardless of the initial position.

2. Theorem
 4.1.4does not apply  when
 the transition matrix is not regular. For example if
0 1 a
A=  and u0 =   (a 6= b) is a probability vector, consider the Markov
1 0 b
chain with initial state u0 that has A as a transition matrix.
         
0 1 a b 0 1 b a
u1 =   =  , u2 =   = .
1 0 b a 1 0 a b
   
a b
This Markov chain will switch between   and   and not converge to a steady
b a
state.

Example 4.1.5 (Summer 2004 Q4) An airline has planes based in Knock, Cork and Shannon.
1 1
Each week 4 of the planes originally based in Galway end up in Knock and 3 end up in Shannon
- the rest return to Galway.

65
 1 1
Of the planes starting the week in Knock, 5 end up in Galway and 10 in Shannon. The rest
return to Knock.
1 1
Finally, of the planes starting the week in Shannon, 5 end up in Galway and 5 , the rest
returning to Shannon.
Find the steady state of this Markov process.

Solution: The Markov process is a sequence v1 , v2 , . . . of column vectors of length 3. The entries
of the vector vi are the proportions of the airline’s fleet that are located at Galway, Knock and
Shannon at the end of Week i. They are related by

vi+1 = M vi ,

where M is the transition matrix of the process.

Step 1: Write down the transition matrix. If we let gi , ki , si denote the proportion of the
airline’s fleet at Galway, Knock and Shannon after Week i, we have

5 1 1
gi+1 = 12 gi + 5 ki + 5 si

1 7 1
ki+1 = 4 gi + 10 ki + 5 si

1 1 3
si+1 = 3 gi + 10 ki + 5 si

Thus  
5 1 1
   12 5 5  
gi+1 gi
 
 
    
vi+1 =  ki+1

=
  1 7 1   ki  = M v i .
 
   4 10 5  
si+1 si
 
 
 
1 1 3
3 10 5

M is the transition matrix of the process.

Note: If the rows and columns of M are labelled G, K, S for Galway, Knock and Shannon,
then the entry in the (i, j) position is the proportion of those planes that start the week in the
airport labelling Column j which finish the week in the airport labelling Row i. Note that M
is a regular stochastic matrix.

Step 2: The steady state of the process is the unique eigenvector of m with eigenvalue 1 that is a
probability vector. To calculate this we need to solve the system of equations whose coefficient

66
matrix is M − 1I3 (and which has zeroes on the right). The coefficient matrix is
 
7 1 1
− 12 5 5 

 
 
 
M − I3 =  1
− 3 1 

4 10 5 


 
 
 
1 1 2
3 10 − 5

Remark: If A is a stochastic matrix (transition matrix), then the sum of the entries in each
column of A is 1. It follows that the sum of the entries in each column of A − I is 0, since
A − I is obtained from A by subtracting 1 from exactly one entry of each column. So the sum
of the rows of A − I is the row full of zeroes. This means that in reducing A − I to reduced
row echelon form, we can begin by simply eliminating one of the rows (by adding the sum of
the remaining rows to it).
We proceed as follows with elementary row operations on the matrix M − I.
   
7 1 1 1 1
− 12 5 5  3 10 − 52
  
1 3 1  1 3 1
− 10 R1 ↔ R3 − 10
  
4 5  4 5
   
  
1 1
3 10 − 52 −→ 7
− 12 1
5
1
5

   
1 1
3 10 − 52 R1 × 3 1 3
10 − 56
   
1 3 1  1 − 12 4
R3 → R3 + (R1 + R2) − 10 −→
   
4 5 10 5
  
   
−→ 0 0 0 R2 × 4 0 0 0

   
3
1 10 − 65 1 3
10 − 65
R2 → R2 − R1   R2 × (−2/3)  
 0 − 15 2   0 1 − 43
   
10

−→   −→  
0 0 0 0 0 0

 
1 0 − 45
R1 → R1 − (3/10)R2  
 0 1 − 43
 

−→  
0 0 0
 
x
 
Thus any vector  y  satisfying x = 54 z and y = 34 z is an eigenvector of M corresponding
 
 
z
to the eigenvalue λ = 1. We need the unique such eigenvector in which the sum of the entries

67
is 1, i.e.
4 4 47
z + z + z = 1 =⇒ z = 1.
5 3 15
15
Thus z = , and the steady state vector is
47
 
12
 47 
 
 
 
 20 .

47

 
 
 
 
15
47

68