Scribd
Upload
5 views

session-9

Uploaded by

Irfan Khilji
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
5 views

session-9

Uploaded by

Irfan Khilji
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

MARKOV CHAIN AND MARKOV

DECISION PROCESS– SESSION 9


Sumanta Basu

Operations Management Group


Indian Institute of Management Calcutta
PROPERTIES AND CLASSES OF MARKOV
CHAIN
¢ Accessibility
¢ Communication

¢ Class definition

¢ Irreducible

¢ Transience

¢ Recurrence

¢ Absorbing

¢ Periodicity
MARKOV CHAIN CLASSIFICATION:
EXAMPLE

0 1 2 3 4
0 0 0.2 0.5 0.3 0
1 0 0 0 1 0
2 0 0.2 0 0.1 0.7
3 0 1 0 0 0
4 0.8 0.1 0 0.1 0

¢ Identify the classes in the Markov chain.


¢ Determine the properties (transient or recurrent)
of each class in the Markov chain.
¢ Calculate the periodicity of each class identified.
MARKOV CHAIN CLASSIFICATION:
EXAMPLE
P(1) P(2)
0 1 2 3 4 0 1 2 3 4
0 0 0.2 0.5 0.3 0 0 0 0.4 0 0.25 0.35
1 0 0 0 1 0 1 0 1 0 0 0
2 0 0.2 0 0.1 0.7 2 0.56 0.17 0 0.27 0
3 0 1 0 0 0 3 0 0 0 1 0
4 0.8 0.1 0 0.1 0 4 0 0.26 0.4 0.34 0

P(3)
0 1 2 3 4
0 0.28 0.29 0 0.43 0
1 0 0 0 1 0
2 0 0.38 0.28 0.34 0
3 0 1 0 0 0
4 0 0.42 0 0.3 0.28
EXAMPLE: STEADY STATE PROBABILITY
A telecom company offers two prepaid plans:
PR666 and PR719 plans, offering different data
packs. It also offers three different postpaid plans:
PO399, PO599, and PO799. Company tracked each
customer to understand the yearly transition and
the following TPM summarizes the data:
PR666 PR719 PO399 PO599 PO799
PR666 2/3 1/3 0 0 0
PR719 2/3 1/3 0 0 0
PO399 0 0 ¼ ½ ¼
PO599 0 0 ½ ½ 0
PO799 0 0 0 ½ ½
Consider a total customer base of 20 Lakhs customers with 30% of the
customers opting for Prepaid.
EXAMPLE: STEADY STATE PROBABILITY

0.080 0.184 0.368 0.368 0.286 0.285 0.264 0.166


0.632 0.368 0 0 0.286 0.285 0.264 0.166
0.264 0.368 0.368 0 0.286 0.285 0.264 0.166
0.080 0.184 0.368 0.368 0.286 0.285 0.264 0.166

1-step TPM 8-step TPM

Existence of non-zero steady state probability is ensured if:


- The Markov chain is irreducible
- The Markov Chain is aperiodic
LONG RUN PROPERTIES OF MARKOV
CHAIN: QUIZ PROBLEM
¢ Remember the transition probability matrix:

0 1
0 0.3 0.7
1 0.6 0.4

¢ Steady state equations for this problem will be:


— 𝜋! = 𝜋! 𝑝!! + 𝜋" 𝑝"!
— 𝜋" = 𝜋! 𝑝!" + 𝜋" 𝑝""
— 1 = 𝜋! + 𝜋"
LONG RUN PROPERTIES OF MARKOV
CHAIN: INVENTORY EXAMPLE
¢ Transition probability matrix for the inventory
example:
0.080 0.184 0.368 0.368
0.632 0.368 0 0
0.264 0.368 0.368 0
0.080 0.184 0.368 0.368
¢ The steady state equations will be as follows:
𝜋! = 𝜋!𝑝!! + 𝜋"𝑝"! + 𝜋#𝑝#! + 𝜋$𝑝$!
𝜋" = 𝜋!𝑝!" + 𝜋"𝑝"" + 𝜋#𝑝#" + 𝜋$𝑝$"
𝜋# = 𝜋!𝑝!# + 𝜋"𝑝"# + 𝜋#𝑝## + 𝜋$𝑝$#
𝜋$ = 𝜋!𝑝!$ + 𝜋"𝑝"$ + 𝜋#𝑝#$ + 𝜋$𝑝$$
LONG RUN PROPERTIES OF MARKOV
CHAIN: INVENTORY EXAMPLE
¢ Solving these simultaneous equations will
provide following steady state probabilities :
— 𝜋! = 0.286, 𝜋" = 0.285, 𝜋# = 0.263, 𝜋$ = 0.166

¢ Revisit the transition probability matrix after 8


time periods:

0.286 0.285 0.264 0.166


0.286 0.285 0.264 0.166
0.286 0.285 0.264 0.166
0.286 0.285 0.264 0.166
EXPECTED AVERAGE COST: INVENTORY
EXAMPLE
¢ The average cost structure is as follows:
— If xt = 0, C(xt) = 0
— If xt = 1, C(xt) = 2
— If xt = 2, C(xt) = 8
— If xt = 3, C(xt) = 18

¢ What will the expected average cost per week if


the above cost structure is followed?
— 0.286(0) + 0.285(2) + 0.263(8) + 0.166(18) = 5.662
EXPECTED AVERAGE COST: INVENTORY
EXAMPLE
¢ In the inventory example, we consider three
types of cost components: ordering cost,
procurement cost and penalty cost for unsatisfied
demand. Inventory carrying cost is ignored.
— Ordering cost: Rs. 10 per order
— Procurement cost: Rs. 25 per camera
— Penalty cost: Rs. 50 per unit of unsatisfied demand

— With this cost structure, what is the expected


average cost on this inventory system if the demand
has poisson distribution with mean of 1. Assume
maximum demand of 6 cameras in each week.
EXPECTED AVERAGE COST: INVENTORY
EXAMPLE
¢ K(0) = 10 + 75 + 50[1.PD(4) + 2.PD(5) + 3PD(6)]
¢ K(1) = 50[1.PD(2) + 2.PD(3) + 3PD(4) + 4.PD(5) +
5.PD(6) ]
¢ K(2) = 50[1.PD(3) + 2.PD(4) + 3PD(5) + 4.PD(6)]
¢ K(3) = 50[1.PD(4) + 2.PD(5) + 3PD(6)]

PD(1) PD(2) PD(3) PD(4) PD(5) PD(6)

0.368 0.184 0.06 0.015 0.003 0.0005

¢ K(0) = 86.15
¢ K(1) = 18.37
¢ K(2) = 5.16
¢ K(3) = 1.15
EXPECTED AVERAGE COST: INVENTORY
EXAMPLE
States (i) Expected Cost [K(i)] Steady State Prob.
0 86.15 0.286
1 18.37 0.285
2 5.16 0.263
3 1.15 0.166

Total Expected Average Cost


= (0.286)*86.15 + (0.285)*18.37 + (0.263)*5.16 + (0.166)*1.15
= 31.42
EXAMPLE: STEADY STATE PROBABILITY
A recent data collection reveals that some fraction of
PO599 postpaid customers is shifting to the PR666
prepaid plan. The company decides to attract the
prepaid customers to shift to the PO399 postpaid plan
by offering free data packs. As a result, the company
sees shift of some customers from the PR719 prepaid
plan to PO799 postpaid plan. The revised TPM reflects
the recent shift trends:

PR666 PR719 PO399 PO599 PO799


PR666 2/3 1/3 0 0 0
PR719 2/3 1/6 0 0 1/6
PO399 0 0 ¼ ½ ¼
PO599 ¼ 0 ½ ¼ 0
PO799 0 0 0 ½ ½

Consider a total customer base of 20 Lakhs customers.

You might also like