Mathematical

Modelling
Part 3: Probability Model
 Content of part 2

Chapter 1: Introduction to probability models

Chapter 2: Stochastic models

Chapter 3: Simulation of probability models

3
Outline

• Markov chain

• Markov process

• Time series

4
Markov chains
• A Markov chain is a discrete-time stochastic
model.
• Although the model is simple, the number and
diversity of applications are surprisingly large.

5
Example: Store problem
A pet store sells a limited number of 20–gallon
aquariums. At the end of each week, the store manager
takes inventory and places orders. Store policy is to
order three new 20–gallon aquariums at the end of the
week if all of the current inventory has been sold. If even
one of the 20–gallon aquarium remains in stock, no new
units are ordered. This policy is based on the
observation that the store only sells an average of one of
the 20–gallon aquariums per week. Is this policy
adequate to guard against potential lost sales of 20–
gallon aquariums due to a customer requesting one
when they are out of stock?

6
Step 1: Ask a question

We assume that the number of potential buyers in one week

will have a Poisson distribution with mean one.

7
Step 2: Select a modelling approach

• We will use a Markov chain model

• A Markow chain can best be described as a sequence
of random jumps (a finite discrete set of locations or
states). Suppose that the random variable 𝑋! take
values in a finite discrete set. There is no harm in
assuming that
𝑋! = 1, 2, … , 𝑚
• We say that the sequence 𝑋! is a Markov chain
provided that the probability that 𝑋!"# = 𝑗 depends only
on 𝑋! . If we define
𝑝$% = Pr 𝑋!"# = 𝑗 𝑋! = 𝑖}

8
Step 2: Select a modelling approach

• If we define
𝑝!" = Pr 𝑋#$% = 𝑗 𝑋# = 𝑖}
then the entire future history of the process
{𝑋# } is determined by the 𝑝!" and the probability
distribution of the initial 𝑋& . Of course, when we
say “determined”, we mean that the probabilities
Pr 𝑋# = 𝑖 are determined.
• The actual value of 𝑋# depends on random
factors.

9
Step 3: Formulate the model
• State space: One state contains all of the information
necessary in order to predict the (probabilistic) feature of
the process.
• We will take 𝑋! = 𝑆! , the number of 20-gallon aquariums
in stock at the beginning of our sales week, as the state
variable.
• The demand 𝐷! relates to the dynamics of the model
and will be used to construct the state transition matrix 𝑃.
• The state space is 𝑋! = {1, 2, 3}.
• We assume that 𝑋& = 3

10
Step 3: Formulate the model
• State transition diagram

11
Step 3: Formulate the model
• The distribution of the demand 𝐷# (Poisson
distribution with mean of 1) yields

12
Step 3: Formulate the model
• So that if 𝑋! = 3 then

• The state transition matrix

13
Step 4: Solve the model
• The analysis objective is to calculate the probability
Pr(𝐷! > 𝑆! ) that demand exceeds supply. It depends
on 𝑋! . For example, if 𝑋! = 3 then Pr 𝐷! > 𝑆! =
Pr 𝐷! > 3 = 0.019.
• Let denote 𝜋! 𝑖 = 𝑃𝑟 𝑋! = 𝑖 and 𝜋! denote vector
with entries 𝜋! 1 , 𝜋! 2 , … and 𝑃 denote the transition
matrix. So we can write 𝜋!"# = 𝜋! 𝑃.
• So, we obtain

14
Step 4: Solve the model
• We need to solve that system with the condition 𝜋# +
𝜋' + 𝜋( = 1 to obtain the steady-state distribution of
𝑋! .
• Solve that system, we get
𝜋 = 𝜋#, 𝜋', 𝜋( = 0.285, 0.263, 0.452 .
• For all large 𝑛 it is approximately true that

15
Step 4: Solve the model
• So,

• In the long run, demand will exceed supply

about 10% of the time.

16
Step 5: Answer the question
• The current inventory policy results in lost sale
about 10% of the time.

17
Outline

• Markov chain

• Markov process

• Time series

18
Markov process model
• A Markov process model is the continuous-time
analogue of the Markov chain model.
• It may also be considered as a stochastic
analogue to a continuous-time dynamical
system model.

19
Example: Forklift-truck maintenance
problems
A mechanic working for a heavy equipment repair
facility is responsible for the repair and maintenance
of forklift trucks. When forklifts break down, they are
taken to the repair facility and serviced in the order
of their arrival. There is space for 27 forklift trucks at
the facility, and last year the facility repaired 54
trucks. Average repair time for a single vehicle is
about three days. In the past few months certain
questions have been raised about the effectiveness
and the efficiency of this operation. The two central
issues are the time it takes to have a machine
repaired, and the percentage of time the mechanic
devotes to this part of his duties.

20
Step 1: Ask a question
• Forklift trucks arrive at the facility for repair at a rate of
)*
#'
= 4.5 per month.
• The maximum rate at which they can be repaired is
''
(
≈ 7.3 vehicles per months, due to an average of 22
working days per month.
• Let 𝑋+ denote the number of vehicles in the repair shop
at time 𝑡. We are interested in the average number in
service, 𝐸𝑋+ , and the proportion of time that the
mechanic is busy repairing machines, represented by
Pr 𝑋+ > 0

21
Step 2: Select a modelling approach

• We will model the repair facility using a Markov

process.
• Markov process is the continuous-time analogue of the
Markov chain.
• As before, we will assume that the state space is finite,
we will suppose 𝑋+ ∈ {1, 2, … , 𝑚}
• The stochastic process {𝑋+ } is a Markov process if the
current state 𝑋+ really represents the state of the
system. It totally determines the probabilistic future of
the process. This condition is formally written as
Pr 𝑋+", = 𝑗 𝑋- : 𝑢 < 𝑡} = Pr 𝑋+", = 𝑗 𝑋+ }

22
Step 1: Ask a question

23
Step 2: Select a modelling approach

• This condition is formally written as

Pr 𝑋+", = 𝑗 𝑋- : 𝑢 < 𝑡} = Pr 𝑋+", = 𝑗 𝑋+ }
• This property has two important implications:
• The first one: The time until the next transition does not
depend on how long the process has been in the
current state. In other word, this distribution of time
spent in a particular state has the memoryless
property.
• The second one: Markov process has to do with state
transition. The probability distribution that describes the
identity of the next state can depend only on the
current state.

24
Step 3: Formulate the model
• Denote 𝑋+ = the number of forklifts in repair at time 𝑡
months. Since the facility can handle only 27 forklifts,
we have 𝑋+ ∈ 0, 1, 2, … , 27 .
• The only allowed transitions are from 𝑋+ = 𝑖 to 𝑋+ = 𝑖 +
1 or 𝑖 − 1. The rates up and down are 𝜆 = 4.5 and 𝜇 =
7.3, respectively, except that we can not transition up
from state 27 or down from state 0.

25
Step 3: Formulate the model
• The steady-state equations can be obtained from the
rate diagram by using the principle:
𝑅𝑎𝑡𝑒 𝑜𝑢𝑡 = 𝑅𝑎𝑡𝑒 𝑖𝑛
equivalent to

26
Step 3: Formulate the model
• Solving that system with ∑𝑝! = 1 will yield the
steady-state Pr{𝑋' = 𝑖}.
• We are interested in Pr 𝑋' > 0 = 1 −
Pr 𝑋' = 0 = 1 − 𝑃& and in 𝐸𝑋' = ∑𝑖𝑃! .

27
Step 4: Solving the model
• We first solve for 𝑃% in terms of 𝑃& , then for 𝑃(
in terms of 𝑃% , and so forth, to obtain 𝑃# =
) ) #
*
𝑃#+% for all 𝑛 = 1, 2, … , 27. Then 𝑃# = *
𝑃&
for all such 𝑛. Since
(- (- #
𝜆
8 𝑃# = 𝑃& 8 =1
𝜇
#,& #,&
%+. )
• So, we must have 𝑃& = %+. !" , 𝛼 = *

28
Step 4: Solve the problem
• We substitute values for 𝜆, 𝜇 to get Pr{𝑋' >
0} = 1 − 𝑃& ≈ 0.616
• Next, we have

which yields 𝐸𝑋' = 1.607

29
Step 5: Answer the question
we consider a system where forklift trucks break down at
a rate of 4.5 per month and are taken to a repair facility
with the capacity to service up to 7.3 per month. Since
the rate at which vehicles arrive for repair is only about
60% of the potential service rate, the mechanic is busy
with this activity only about 60% of the time. However,
since breakdowns occur essentially at random, there will
be times when there is more than one vehicle in the
shop at one time, through no fault of the mechanic. In
fact, on an average day we would expect to see 1.6
vehicles in the repair facility. By this, we mean that if we
kept track on a daily basis of the number of vehicles in
the repair facility, then at the end of the year this number
would average about 1.6.

30
Step 5: Answer the question
Histogram showing the distribution of the number of
forklifts in service at any given time

31
Outline

• Markov chain

• Markov process

• Time series

32
Exercises

33
Time series
• Self-study

34