Markov Chains and Simulation

Techniques
- Dr. Sachin Vyavhare
(Associate Professor)
Markov Chains Analysis
• Introduction : Markov Chain Concepts
• To find the probabilities for the Kth Period
• Steady state probabilities
 Concepts- Markov Chain
(A)Markov chain: A sequence of events
(or outcomes) in which , an event depends
upon the immediate preceding event (only) ,but
not on the other prior events is called as
Markov Chain or Markov Process.
E.g. The market shares for a product during a
month or year , condition of machines to be
used for production each week etc.
 (B)States
 Each of these systems consist of several possible states
,E.g. the various brands of the product represent the states
for the market share problem, similarly working ,fairly
working and non-working conditions represent the states
of the machines; Attendance- Absent and Present etc.
 The states are assumed to be finite in number and are
collectively exhaustive and mutually exclusive .
 A state is called as an absorbing state if there is no
tendency to leave that state , otherwise it is an non-
absorbing state. E.g. If a customer once uses a brand
,does not change it at all, then it is an absorbing state.
 Markovchain analysis is used to study the
probabilities corresponding to the states at any given time
period, considering the movements from one state to
another.
(C)Transition Probabilities: The probabilities of
the system to change from a state (i) to the
state (j) is called as a transition probability and
it represents the likelihood of the system to
change the states, from one period to the next.
These transition probabilities are assumed to
remain constant over a period of time.
E.g. Probabilities that customers change their
brand of a product from A to other say B and C
etc.
(D)Transition Matrix(P) : A matrix representing
the states in one period (rows) and the state in the
next period (columns) ,along with the transition
probabilities between them is called as a transition
matrix(P).
e.g. If over a time, it is found that 70% customers
using brand A continue to use it next year while
20% shift to brand B and 10% to brand C. Similarly,
60% customers using brand B continue to use it next
year while 25% shift to brand A and 15% to brand
C. And for brand C 75% customers are retained
while 20% lost to brand A and 5% to brand B then
we have:
 Transition Matrix

(Note: Sum of the probabilities along each row is equal to 1.)

Transition diagram
Probability Tree Diagram
Uses of Markov Chains
Markov Analysis is useful for studying various
business problems such as:
(i) Marketing: To study customer loyalty,
switching tendencies, market share analysis
etc.
(ii) Finance: Accounts Receivables, Depositors
Analysis ete.
(iii) Production: Maintenance problems,
Inventory management, etc.
(iv) Personnel: Manpower planning, etc..
 Case I- To find the probabilities for kth
period of time
Q.1) The present market share of three brands
of soft drinks A , B, C are 60%,30%,10%. The
transition matrix on the basis of shifting pattern
for a year is

Find the probabilities (market share) for 2 years from now?

Solution:
Let a1,a2,a3 be the market shares of three brands of soft drink A,
B,C for the initial period/present year (n=0)
Ro= [ a1 a2 a3]
Ro= [0.60 0.30 .0.10]
The state probabilities for the next period (n=1 year)
R1=Ro*P
R1= [ 0.60 0.30 0.10] x

R1= [(0.42+0.06+0.01) (0.12+0.18+0.01) (0.06+0.06+0.08)]

R1=[0.49 0.31 0.20]
The market share of three brands of soft drinks at the end of
1 year is 49%,31%,20%
The state probabilities for the next period (n=2 year)
R2=R1*P
R2=[0.49 0.31 0.20] x

R2= [(0.343+0.062+0.02) (0.098+0.186+0.02)

(0.049+0.062+0.16)]
R2=[0.425 0.304 0.271]
The market share of three brands of soft drink at the end of 2
year is 42.5%,30.4%,27.10% respectively
 OR
R2=R1*P
=Ro*P*P
=Ro*P2
R2=[ 0.60 0.30 0.10] x x
R2= [0.425 0.304 0.271]
Q.2) The mobility of the population in a state to village ,
town and city is given in the following percentage.

From To
Village Town City
Village 50% 30% 20%
Town 10% 70% 20%
City 10% 40% 50%
What will be proportion of population in village , town
and city after two years given that present population is
0.7, 0.2 , 0.1 respectively?

Ans- [0.252 0.479 0.269]

Q.3) Three brands of product P, Q and R having market
share as 30%, 30% and 40% respectively. Customers
shift their brands. Brand switching matrix every quarter
is given below.
From To
P Q R
P 50% 30% 20%
Q 20% 70% 10%
R 20% 20% 60%

Apply concept of Markov Chain to find market share at

the end of First & Second quarter.
 Case II- steady state probabilities
Steady state probability: If the transition from one state to
other continue indefinitely, System becomes stable and
state probabilities tends to remain constant .This is steady
state (equilibrium) condition
Symbolically Rk = Rk-1
Rk = P x Rk-1
Rk = P x Rk
If SA and SB are steady state probabilities then
[SA SB] = [SA SB] X P
Also SA + SB =1
Q.4) A student tries to be punctual
for the classes. If (S)he is late on a
day,(S)he is 90 % sure to be on time
next day . Similarly , if (S)he is on
time then , there is 30 % chance
that (S)he will be late on next day.
How often in long run, (S)he is
expected to be late for the class?
Let a and b be the probabilities of being “on time “ and “ Late”

[ a b ] = [ a b ]* P

[ a b ] = [ a b ]*

[ a b ] = [0.70a+0.90b 0.30a+0.10b]
a= 0.70a+0.90b --------eq.1 and
b= 0.30a+0.10b --------eq.2
a= 0.70a+0.90b --------eq.1
0.30a=0.90b
a=3b
We have a+b=1 3b+b=1 4b=1
b=1/4=0.25
a=1-1/4=3/4=0.75
In the long run , the student is expected to be
late 25% of time.
Q.5) Market survey is made on two brands of breakfast
foods A & B. Every time a customer purchases, he may
buy the same brand or switch to another brand. The
transition matrix is given below:

From To
A B
A 0.8 0.2
B 0.6 0.4

At present 60% of people buy brand A and 40% buy

brand B. Determine market shares of brand A & B in
the steady state.
Q.5) Find the steady state probabilities for three
newspaper whose transition probabilities whose
transition probability matrix is given below.

0.8 0.125 0.075

0.09 0.9 0.01
0.117 0.017 0.866

Ans-[0.3307 0.4506 0.2187]

Simulation Technique
 Concept-Simulation
• Simulation involves development of model of
real system and performing experiment with a
view of predicting the behavior of the system
over a time.
 Process of simulation
• Defining the problem

• Development of model

• Experimentation

• Analysis and interpretation.

 Elements of a Simulation Model
1) System Input-
2) Random Number Generation-
3) Work Data Sheet-
 Monte Carlo Simulation Model
Step1: Identify the input variables, collect the data
and write the probability distribution-Simulation
Model.
Step 2: Determine cumulative probability distribution.
Step 3: Identify the random numbers corresponding to
cumulative probability.(two decimal random
numbers-(00-99 & three decimal random numbers-
(000-999)
Step 4. prepare a table of random number to simulate
the model and obtain the expected value of variables
Step 5- Interprete the results.
 Q1) A bakery keeps stock of a particular brand of
cake. Previous experience shows the daily demand
pattern for the cakes with associated probabilities
as given below. (May-June-2024)

Daily Demand 0 10 20 30 40 50
(No. of units)
Probability 0.01 0.20 0.15 0.50 0.12 0.02

Use the following sequence of random numbers to

simulate (estimate)the demand for next 10 days .
Find the average demand per day.
• Random Numbers: 25,39,65,76,12,05,73,89,19,49
Monte-Carlo simulation to simulate the data.
Daily Probability Cumulative Random Number
Demand Probability Interval
0 0.01
10 0.20
20 0.15
30 0.50
40 0.12
50 0.02
 Simulate the demand for next 10 days
Day Random Numbers Random Number Daily Demand
Intervals
1 25
2 39
3 65
4 76
5 12
6 05
7 73
8 89
9 19
10 49
Monte-Carlo simulation to simulate the data.
Daily Probability Cumulative Random Number
Demand Probability Interval
0 0.01 0.01 00
10 0.20 0.21 01-20
20 0.15 0.36 21-35
30 0.50 0.86 36-85
40 0.12 0.98 86-97
50 0.02 1.00 98-99
 Simulate the demand for next 10 days
Day Random Numbers Random Number Daily Demand
Intervals
1 25 21-35 20
2 39 36-85 30
3 65 36-85 30
4 76 36-85 30
5 12 01-20 10
6 05 01-20 10
7 73 36-85 30
8 89 86-97 40
9 19 01-20 10
10 49 36-85 30
Average Demand per day=(20+30+30+30+10+10+30+40+10+30)/10
= (240/10 )=24 Units
 Q2) At a bus depot every bus should leave with
driver. At the terminus , they should keep two
drivers as reserved if anyone on scheduled duty is
sick and could not come . Following is the
probability distribution that driver become sick.
No. of sick 0 1 2 3 4 5
drivers
Probability 0.30 0.20 0.15 0.10 0.13 0.12

Use the following sequence of random numbers to

simulate (estimate) the utilization of driver for
next 10 days . Find how many days and how many
buses can not run due to non-availability of bus
driver,
• Random Numbers: 30,54,34,72,20,02,76,74,48,22
Monte-Carlo simulation to simulate the data.
Number of Probability Cumulative Random Number
sick drivers Probability Interval
0 0.30
1 0.20
2 0.15
3 0.10
4 0.13
5 0.12
 Simulate the no. of sick drivers for next 10 days
Day Random Random Number of Utilization Shortage of
Numbers Number sick drivers of reserved driver
Intervals (simulated) drivers (out
of 2)
1 30
2 54
3 34
4 72
5 20
6 02
7 76
8 74
9 48
10 22
Monte-Carlo simulation to simulate the data.
Number of Probability Cumulative Random Number
sick drivers Probability Interval
0 0.30 0.30 00-29
1 0.20 0.50 30-49
2 0.15 0.65 50-64
3 0.10 0.75 65-74
4 0.13 0.88 75-87
5 0.12 1.00 88-99
Simulate the utilization of reserved drivers for next 10
days
Day Random Random Number of Utilization Shortage of
Numbers Number sick of reserved driver
Intervals drivers(sim drivers (out
ulated) of 2)
1 30 30-49 1 1 --
2 54 50-64 2 2 --
3 34 30-49 1 1 --
4 72 65-74 3 2 3-2=1
5 20 00-29 0 0 --
6 02 00-29 0 0 --
7 76 75-87 4 2 4-2=2
8 74 65-74 3 2 3-2=1
9 48 30-49 1 1 --
10 22 00-29 0 0 --
• Interpretation:
On 4th , 7th , 8th day –( 3 days)- 4 buses can not
run due to non- availability of drivers.
Q3) An electric gen-set manufacturing company
manufactures gen-sets which is a variable.
Following is the probability distribution for the
gen-set manufactured.
No. of gen-set 23 24 25 26 27 28
manufactured

Probability 0.05 0.10 0.30 0.25 0.20 0.10

According to company’s policy , the

manufactured sets are transported to the
godown on the same day. The truck carrying
gen-set is having a capacity of 25 sets.
• Simulate the system for next 10 days using
Random Numbers: 84,06,93,38,29,59,64,18,08,45.

• Find
• 1) Average number of sets manufactured per day.
• 2) Average number of vacant spaces in truck per
day.
• 3) Average number of sets remaining to be
transported per day.
Monte-Carlo simulation to simulate the data.
No. of gen-set Probability Cumulative Random Number
manufactured Probability Interval

23 0.05
24 0.10
25 0.30
26 0.25
27 0.20
28 0.10
Monte-Carlo simulation to simulate the data.
No. of gen-set Probability Cumulative Random Number
manufactured Probability Interval

23 0.05 0.05 00-04

24 0.10 0.15 05-14
25 0.30 0.45 15-44
26 0.25 0.70 45-69
27 0.20 0.90 70-89
28 0.10 1.00 90-99
 Simulate the no. of get-set manufactured for next 10 days
Day Random Random No. of gen- Balance Empty spaces
Numbers Number set with factory
Intervals manufactur
ed
(Simulated)
1 84
2 06
3 93
4 38
5 29
6 59
7 64
8 18
9 08
10 45
 Simulate the no. of get-set manufactured for next 10 days
Day Random Random No. of gen- Balance Empty spaces
Numbers Number set with factory
Intervals manufactur
ed
(Simulated)
1 84 70-89 27 2 --
2 06 05-14 24 2-1=1 --
3 93 90-99 28 1+3=4 --
4 38 15-44 25 4+0=4 --
5 29 15-44 25 4+0=4 --
6 59 45-69 26 4+1=5 --
7 64 45-69 26 5+1=6 --
8 18 15-44 25 6+0=6 --
9 08 05-14 24 6-1=5 --
10 45 45-69 26 5+1=6 --
1) Average number of gen-sets manufactured per
day.
=(27+24+28+25+25+26+26+25+24+26)/10
=25.6~ 26 gen-sets
2) Average number of vacant spaces in truck per
day.
=0/10=0
3) Average number of sets remaining to be
transported per day.
= 43/10
=4.3 Units
=~ 4 Units
Q 4) The rainfall distribution in mansoon season is
as follows:
Rain in Cm 0 1 2 3 4 5

Frequency 50 25 15 5 3 2

Simulate the rainfall for 10 days using following

random numbers : 67, 63, 39, 55, 29
78,70,06,78,76. Find the average rainfall.
 Home Exercise
The manager of book store has to decide the number of copies to be
ordered of a particular tax law book. The book costs Rs. 80. Since
some of the tax laws changes year after , any copies unsold while
the edition is current has a scrap value of Rs. 30. From the past
records, the distribution of demand for this book has been obtained
as follows.
Demand in no. 15 16 17 18 19 20 21 22
of copies
Probability 0.05 0.08 0.20 0.45 0.10 0.07 0.03 0.02

Using the following sequence of random numbers , generate the

demand for 20 time period. Calculate the average number of copies
in demand and place an order for the same. Find the average profit
under this number of copies to be ordered.
Random No.
14,02,93,99,18,71,37,30,12,10,88,13,00,57,69,32,18,08,92,73
Answer- Avg No. of copies=18; Net profit=6300; Average profit / time period=Rs. 315
 Hint
Copies demanded probability Cumulative Random number
probability interval

Time Period Random Numbers Random no. Simulated demand

interval of tax copies

Avg No. of copies=18

Time Order Simulated Qty. Qty Profit Loss on Net profit

period Quantity demand Sold Unsold on sale Unsold
1 18
2… 18..
6300
Profit= S.P.-C.P
Net Profit= profit on sale-loss on unsold