Scribd
Upload
6 views

IE 450 Part 4 Markov Chains-State Classification

Uploaded by

berker.apl08
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views

IE 450 Part 4 Markov Chains-State Classification

Uploaded by

berker.apl08
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 12

IE 450

Stochastic Models in Operations Research

Part 4 Markov Chains: Classification of States


Classification of States

• Our aim is a better understanding the characteristics of a process


• Classification of states is important for understanding the long run
behavior of a process
• Transient states
• Recurrent states
• Irreducibility
Transient and Recurrent States

• A state i is absorbing, if the process can not leave this state once it
is entered
• A state i is absorbing if pii=1
• As an example, consider GRP: states 0 and N are absorbing states
• In GRP, all states except 0 and N are transient states: there is a
time for which any state in {1,2,…,N-1} will be visited a last time,
and the chain will be absorbed by either state 0 or state N.
Transient and Recurrent States

• A state i is recurrent if:


starting from state i, it will be visited over and over again
without a last visit time. The probability of revisit is equal to
one.
• An absorbing state is a trivial recurrent state
• But, in order a state to be recurrent it doesn’t need to be an
absorbing state.
• A state is transient if:
starting from state i, there is a last visit time. The probability of
revisit is less than one.
Transient and Recurrent States

Consider the following state transition diagram with


states {1,2,3,4,5,6,7,8}. An arrow indicates a positive
probability.

States 1,2,3 are recurrent

States 6 and 7 are recurrent

State 8 is absorbing (and recurrent)

States 4 and 5 are transient.


An example: IE 450 Equations

• Consider a Markov Chain with the following transition matrix

1 2 3 4 5
0 1
1 1/2 0 1/2 0 0
2B
B 0 1/4 0 3/4 0 C
C
P = 3B
B 0 0 1/3 0 2/3 C
C
4 1/4 1/2 0 1/4 0 A
@
5 1/3 0 1/3 0 1/3

Refik Güllü (Boğaziçi University) IISE 2019, Orlando 30 / 1


Communicating classes

• State j is accessible from state i if there is some n≥0 for which


pij(n) > 0 (starting from i, the chain will eventually be able to reach j)

• States i and j are said to be communicating, if


• state j is accessible from state i and,
• state i is accessible from state j
• A Markov Chain is irreducible if all states are communicating
IE 450 Equations

Communicating classes

• It is possible to partition the states of a Markov Chain with respect


to communicating classes. That is,

S = C1 [ C2 [ · · · [ Cm

where all the states in Ck, for any k=1,2,…,m are communicating.
Communicating classes

Is the Markov Chain


irreducible?
Communicating classes

• A set of states C are said to be closed, if pij=0 for i ∈ C and j ∉ C.


• An absorbing state is trivially closed.
• In a finite state space Markov Chain the states in a closed set C
are recurrent.
• The states in the classes that are not closed are transient.
• In a finite state space Markov Chain all the states can not be
transient.
Communicating classes

C1, C3 and C4 are closed

C2 is not closed

You might also like