Notes for Math 450

Continuous-time Markov chains and

Stochastic Simulation
Renato Feres

These notes are intended to serve as a guide to chapter 2 of Norris’s textbook.

We also list a few programs for use in the simulation assignments. As always,
we fix the probability space (Ω, F, P ). All random variables should be regarded
as F-measurable functions on Ω. Let S be a countable (or finite) state set,
typically a subset of Z. A continuous-time random process (Xt )t≥0 is a family
of random variables Xt : Ω → S parametrized by t ≥ 0.

1 Q-matrices - Text sec 2.1

The basic data specifying a continuous-time Markov chain is contained in a
matrix Q = (qij ), i, j ∈ S, which we will sometimes refer to as the infinitesimal
generator, or as in Norris’s textbook, the Q-matrix of the process, where S is
the state set. This is defined by the following properties:
1. qii ≤ 0 for all i ∈ S;
2. qij ≥ 0 for all i, j ∈ S such that i 6= j;
P
3. j∈S qij = 0 for all i ∈ S.

The motivation for introducing Q-matrices is based on the following obser-

vation (see theorems 2.1.1 and 2.1.2 in Norris’s text), which applies to a finite
state set S: the matrices P (t) = etQ , for t ≥ 0, defined by the convergent
matrix-valued Taylor series

tQ t2 Q2 t3 Q3
P (t) = I + + + + ···
1! 2! 3!
constitute a family of stochastic matrices. P (t) = (pij (t)) will be seen to be the
transition probability matrix at time t for the Markov chain (Xt ) associated to
Q. The chain (Xt ) will be defined later not directly in terms of the transition
probabilities but from two discrete-time processes (the holding times and jump
chains) associated to Q, to be defined later. Only after that will we derive the
interpretation
pij (t) = Pi (Xt = j) = P (Xt = j|X0 = i).

1
In fact, we will prove later that a continuous-time Markov chain (Xt )t≥0 derived
from a Q-matrix satisfies:

P (Xtn+1 = j|Xtn = i, Xtn−1 = in−1 , . . . , Xt0 = i0 ) = pij (tn+1 − tn )

for all times 0 ≤ t0 ≤ t1 ≤ · · · ≤ tn+1 and all states j, i0 , . . . , in−1 , i. We take

this property for granted for the time being and examine a few examples.

2 A few examples
The information contained in a Q-matrix is conveniently encoded in a transition
diagram, where the label attached to the edge connecting state i to state j is
the entry qij . We disregard all self-loops.
Example 1. We begin by examining the process defined by the diagram of
figure 1.

λ
1 2

Figure 1: After an exponential random time with parameter λ, the process

switches from state 1 to state 2, and then remains at 2.

The Q-matrix associated to this diagram has entries −q11 = q12 = λ and
q21 = q22 = 0. To obtain the stochastic matrix P (t) we use Theorem 2.1.1,
which shows that P (t) satisfies the equation P 0 (t) = QP (t) with initial condi-
tion P (0) = I. It is immediate that the entry p11 (t) of P (t) must satisfy the
differential equation y 0 = −λy with the initial condition y(0) = 1. This gives
y(t) = e−λt , so that  −λt
1 − e−λt

e
P (t) = ,
0 1
The key remark is that the time of transition from 1 to 2 has exponential
distribution with parameter λ. (See Lecture Notes 3, section 5.) We will recall
later some of the salient features of exponentially distributed random variables.
Example 2. The next example refers to the diagram of figure 2.
The Q-matrix for this example is
 
−µ λ1 ... λN
 0 0 ... 0 
Q=
 
.. .. .. .. 
 . . . . 
0 0 ... 0

where µ = λ1 + · · · + λN . Once again, we use the equation P 0 (t) = QP (t) with

initial condition P (0) = I to obtain the transition probabilities. It is clear from

2
 j

2
λj λ2
λΝ λ1
Ν 0 1

Figure 2: After an exponential random time with parameter λ = λ1 + · · · + λN ,

the process switches from state 0 to one of the states 1, . . . , N , and then remains
there.

the equation
N
X
p0ij (t) = qik pkj (t)
k=0

that, whenever the present state is i 6= 0, the transition to j 6= i has probability

pij (t) = 0. For i = 0 we have p000 (t) = −µp00 (t), p00 (0) = 1, and for j 6= 0,
p00j (t) = −µp0j (t) + λj , p0j (0) = 0. The solution is easily seen to be
(
e−µt for j = 0
p0j (t) = λj −µt
µ (1 − e ) 6 0.
for j =

The solution can be interpreted as follows: assuming that the process is initially
at state zero, the transition to a state j 6= 0 happens at an exponentially dis-
tributed random time with parameter µ = λ1 + · · · + λN . At that jump time,
the new state j is chosen with probability
λj
gj = .
λ1 + · · · + λN
Example 3. We study now the process defined by the diagram of figure 3.
The transition probabilities pij (t) can be obtained as in the previous examples.

λ1 λ2 λ3
1 2 3

Figure 3: A continuous-time birth process.

The Q-matrix for this example has entries


−λi
 if j = i
qij = λi if j = i + 1

0 if j 6= i, i + 1.


3
 It can be shown in this case that at each state i the process waits a random
time, exponentially distributed with parameter λi , then jumps to the next state
i + 1. The mean holding (waiting) time at state i is 1/λi . (See properties of
exponential distribution in Lecture Notes 3.) Denote by Sn the holding time
before the n-th transition (to state n) Note that the expected value of the sum
ζ = S1 + S2 + . . . is finite if
∞
X 1
< ∞.
λ
i=1 i

In this case, J must be finite with probability 1. (If a random variable assumes
the value ∞ with positive probability, its expected value is infinite. This is clear
since the weighted average of a set of numbers that includes ∞ with positive
weight is necessarily equal to ∞.) The random variable ζ is called the first
explosion time of the process. If the process has finite explosion time, it will
run through an infinite number of transitions in finite time. We will have more
to say about this phenomenon later.
We consider in more detail the special case of the last example having con-
stant λi = λ. The transition probabilities in this case can be calculated (see
example 2.1.4 in text) to be

(λt)j−i
pij (t) = e−λt .
(j − i)!

In particular, the transition from i = 0 to j in time t has the Poisson dis-

tribution with parameter λt. (See Lecture Notes 3.) Therefore, the process
(Xt )t≥0 of example 3 for constant λ, and starting at X0 = 0, has the following
characterization: for each t, Xt is a Poisson random variable with parameter λt.

3 Jump times and holding times - Text sec. 2.2

Since the set of states is discrete and the time parameter is continuous, it is
clearly not possible for the sample paths Xt (ω) to be continuous functions of t.
At random times J0 = 0, J1 , J2 , . . . , called the jump times (or transition times)
the process will chance to a new state, and the sequence of states constitute a
discrete-time process Y0 , Y1 , Y2 , . . . .
It is convenient to assume that sample paths are right-continuous. This
means that for all ω ∈ Ω, there is a positive  such that Xs (ω) = Xt (ω) for s, t
such that t ≤ s ≤ t + . In particular, Xt = Yn for Jn ≤ t < Jn+1 .
More formally, we define the jump times of the process (Xt )t≥0 inductively
as follows: J0 = 0 and, having obtained Jn we define Jn+1 as

Jn+1 = inf{t ≥ Jn |Xt 6= XJn }.

The infimum, or inf, of a set A of real numbers is the unique number a (not
necessarily in A) such that every element of A is greater than or equal to a (i.e.,
a is a lower bound for A) any no other lower bound for A is greater than a.

4
Thus Jn+1 is the least random time greater than Jn at which the process takes
a new value Xt 6= XJn . In the definition of Jn+1 the infimum is evaluated for
each sample path. A more explicit statement is that for each ω ∈ Ω, Jn+1 (ω)
is the infimum of the set of times t ≥ tn = Jn (ω) such that Xt (ω) is different
from Xtn (ω). It could happen that the process gets stuck at an absorbing state
and no further transitions occur. In this case Jn+1 (ω) = ∞. In this case we
define XJn = X∞ (the final value of Xt ). If all Jn are finite, the final value of
the process is not defined.
We also define the holding times Sn , n = 1, 2, . . . , as the random variables
(
Jn − Jn−1 if Jn−1 < ∞
Sn =
∞ otherwise.

The right-continuity condition implies that the holding times Sn are positive
for all n, that is, there cannot be two state transitions happening at the same
time. It is, nevertheless, possible in principle for a sequence of jump times to
accumulate at a finite time. In other words, the random variable
∞
X
ζ= Sn
n=1

may be finite. The random variable ζ is called the first explosion time. As we
saw in the birth process in the previous section, it is possible that the holding
times of a sequence of state transitions become shorter and shorter, so that the
chain undergoes an infinite number of transitions in a finite amount of time.
This is called an explosion. We will describe later simple conditions for the
process to be non-explosive.
The analysis of a continuous-time Markov chain (Xt )t≥0 can be approached
by studying the two associated processes: the holding times Sn and the jump
process Yn , n = 0, 1, 2, . . . . This is explained in the next section.

4 The jump matrix Π - Text sec. 2.6

For a given Q-matrix Q = (qij ) we associate a stochastic matrix Π = (πij ),
called the jump matrix, asPfollows. Write qi = −qii for all i ∈ S. Note that qi
is non-negative and qi = j6=i qij . Now define Π as follows: for each i ∈ S, if
qi > 0 set the diagonal entry of the i-th row of Π to zero and the other entries
to πij = qij /qi . If qi = 0, set πii = 1 and the other entries in row i to 0. In
other words, define:


qij /qi if qi 6= 0 and j 6= i

0 if qi 6= 0 and j = i
πij =


0 if qi = 0 and j 6= i
1 if qi = 0 and j = i.


5
 As an example, consider the process specified by the Q-matrix:
 
−2 1 0 1 0 0
 0 0 0 0 0 0 
 
 0 1 −4 0 0 3 
Q= 
 2 0
 0 −4 2 0 

 0 3 1 0 −5 1 
0 0 0 0 0 0

The associated transition diagram is given in figure 4.

1 1
1 2 3

1 2 3 1 3

4 5 6
2 1

Figure 4: Transition diagram for a continuous-time Markov chain. The corre-

sponding Π-matrix is given in the text.

The Π-matrix for this process is:

0 12 1
 
0 2 0 0
 0 1 0 0 0 0 

 0 1 3 

4 0 0 0 4 
Π=  1 0 1
 2 3 0 0 2 0 

 0 1 1 
5 5 0 0 5
0 0 0 0 0 1

We can now present the general description of a continuous-time Markov

chain as consisting of two independent discrete-time processes: the holding times
Sn and the jump process Yn associated to the Π-matrix. We call this the hold-
and-jump process. It won’t be immediately apparent why the hold-and-jump
process has the transition probabilities matrix P (t) = etQ . We will return to
this point a little later and show the different but equivalent ways in which the
continuous-time Markov chain can be represented.
Here is the main definition. First suppose that the process is non-explosive.
For a given Q-matrix Q and initial probability distribution λ on S, let (Yn )n≥0
be a discrete-time Markov chain Markov(λ, Π), where Π is the Π-matrix asso-
ciated to Q. Having the Yn , let now S1 , S2 , . . . be a sequence of independent
exponential random variables with parameters qY0 , qY1 , . . . , respectively. For
each n ≥ 1, define the n-th jump time by Jn = S1 + · · · + Sn , and J0 = 0. Fi-
nally, define a hold-and-jump process with initial probability distribution λ and

6
generator matrix Q, written Markov(λ, Q), as the (right-continuous) process
(Xt )t≥0 given by
Xt = Yn if Jn ≤ t < Jn+1 .
This definition can be modified to include explosive chains. This amounts to
adding to S an extra state, denoted ∞, which is attained after explosion, and
defining Xt = ∞ if t does not lie in any of the intervals [Jn , Jn+1 ).
Representing a continuous-time Markov chain as a hold-and-jump process is
particularly useful as it suggests a method of stochastic simulation. We pursue
this in the next section.

5 Simulation of the hold-and-jump process

We begin by restating the description of the hold-and-jump process, making use
of some of the basic properties of exponential random variables. See Lecture
Notes 3 for a discussion of exponential random variables and their simulation.
In particular, we should keep in mind the following: (i) if T is an exponential
random variable of parameter 1, then αT is an exponential random variable of
parameter 1/α; and (ii) if M (j) are independent exponential random variables
(j)
of parameters λP j , then M = inf j M is an exponential random variable of
(j)
parameter λ = j M . Here, M can be interpreted as the time of the first
occurring event among events with exponential times M (j) .
To obtain a sample chain of holding times (Sn ) and states (Yn ), we do the
following: First choose X0 = Y0 with probability distribution λ. Then choose
(j)
an array (Tn : n ≥ 1, j ∈ S) of independent exponential random variables of
parameter 1. Inductively, for n = 0, 1, 2, . . . , if Yn = i set:
(j) (j)
Sn+1 = Tn+1 /qij for j 6= i,
(j)
Sn+1 = inf Sn+1
j6=i

Now choose the new state Yn+1 according to the rule:

(
(j)
j if Sn+1 = Sn+1 < ∞;
Yn+1 =
i if Sn+1 = ∞.
(j)
Conditional on Yn = i, the Sn+1 thus obtained are independent exponential
random variables of parameter qij , for all j 6= i, and Sn+1 is exponential of
parameter qi ; Yn+1 has distribution (πij : j ∈ S), and Sn+1 and Yn+1 are
independent and (conditional on Yn = i) independent of Y0 , Y1 , . . . , Yn and
S1 , S 2 , . . . , S n .
Before turning this into an algorithm for simulating the Markov chain, we
briefly mention another useful interpretation of the process which has some in-
tuitive appeal. We imagine the transitions diagram of the process as depicting a
system of chambers (vertices) with a gate for each directed edge from chamber i
to chamber j. The gates are directed, so that gate (i, j) controls the transit from

7
chamber i to j, while flow from j to i is controlled by gate (j, i). Each gate opens
independently of all others at random times for very brief moments, and when-
ever it does, anyone waiting to pass will immediately take the opportunity to do
so. Over a period [0, t], the gate (i, j) will open at times according to a Poisson
process with parameter qij . In other words, The number of times, Nij (t), that
the gate opens during [0, t] is a Poisson random variable with parameter qij t,
and these events are distributed over [0, t] uniformly. Now, someone moving in
this maze, presently waiting in chamber i, will move next to the chamber whose
gate from i opens first. The chain then corresponds to the sequence of chambers
the person visits and the waiting times in each of them.
We now describe an algorithm that implements the hold-and-jump chain.
The following assumes a finite state space S:
1. Initialize the process at t = 0 with initial state i drawn from the distribu-
tion λ;
2. Call the current state i; simulate the time of the next event, t0 , as an
Exp(qi ) random variable;
3. Set the new value of t as t ← t + t0 ;
4. Simulate the new state j: if qi = 0, set j = i and stop. If qi 6= 0, simulate
a discrete random variable with probability distribution given by the i-th
row of the Π-matrix, i.e., qij /qi , j 6= i;
5. If t is less than a pre-assigned maximum time Tmax , return to step 2.
The following program implements this algorithm in Matlab.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function [t y]=ctmc(n,pi,Q)
%Obtain a sample path with n events for a
%continuous-times Markov chain with initial
%distribution pi and generator matrix Q.
%The output consists of two row vectors:
%the event times t and the vector of states y.
%Vectors t and y may be shorter than n if
%an absorbing state is found before event n.
%Uses samplefromp(pi,n).
t=[0];
y=[samplefromp(pi,1)]; %initial state
for k=1:n-1
i=y(k);
q=-Q(i,i);
if q==0
break
else
s=-log(rand)/(-Q(i,i)); %exponential holding time

8
 t=[t t(k)+s];
p=Q(i,:);
p(i)=0;
p=p/sum(p);
y=[y samplefromp(p,1)];
end
end
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
As an example to illustrate the use of the program, consider the birth-and-
death chain given by the diagram of figure 5.

1 1 1 1 1
1
1 2 3 4 5 6 7

2 2 2 2 2

Figure 5: Diagram of a birth-and-death chain with an absorbing state.

The first 50 events of a sample path of the chain of figure 5 are shown in
figure 6.

5
states

1
0 10 20 30 40 50 60 70 80
time

Figure 6: A sample path for the continuous-time Markov chain of figure 5. The
initial state is 1. The chains stopped at after 157 events, at the absorbing state
7.

To obtain the sample path of figure 6, we used the following commands:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Q=zeros(7,7);
Q(1,[1 2])=[-1 1];

9
Q(2,[1 2 3])=[2 -3 1];
Q(3,[2 3 4])=[2 -3 1];
Q(4,[3 4 5])=[2 -3 1];
Q(5,[4 5 6])=[2 -3 1];
Q(6,[5 6 7])=[2 -3 1];
pi=[1 0 0 0 0 0 0];
[t y]=ctmc(50,p,Q);
stairs(t,y)
grid
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
It is worth to mention the following modification of the algorithm given above
for simulating a continuous-time Markov chain. It is based on the property that
if we have k independent exponential random variables T1 , . . . , Tk of parameters
λ1 , . . . , λk , respectively, then T = mini {Ti : i = 1, . . . , k} is also an exponential
random variable of parameter λ1 +· · ·+λk . Although it may not be immediately
obvious, the following algorithm produces another version of exactly the same
process as the first algorithm:
1. Initialize the process at t = 0 with initial state i drawn from the distribu-
tion λ;
2. Call the current state i. For each potential next state l (k 6= i), simulate
a time tl with the exponential distribution of parameter qil . Let j be the
state for which tj is minimum among the tl ;
3. Set the new value of t as t ← t + tj ;
4. Let the new state be j;
5. If t is less than a pre-assigned maximum time Tmax , return to step 2.
Although equivalent, this algorithm is less efficient since it requires the sim-
ulation of more random variables.

6 Remarks about explosion times - Text sec. 2.7

Recall that it is possible for a process to go through an infinite sequence of
transitions in a finite amount of time, a phenomenon called explosion. If the
holding times are S1 , S2 , . . . , then the explosion time is defined by
ζ = S1 + S2 + . . .
The process is called explosive if ζ is there is a positive probability that zeta is
finite.
Explosive processes can be extended beyond the first explosion time ζ. Sec-
tion 2.9 has more on the topic . We will not discuss this issue in any depth, but
only mention that any of the following conditions is enough to ensure that the
process is non-explosive (see Theorem 2.7.1):

10
 1. The state set S is finite;
2. There exists a finite number M such that qi ≤ M for all i ∈ S;
3. The initial state, X0 = i, is recurrent for the jump chain having transition
probabilities matrix Π.

7 Kolmogorov’s equations - Text sec. 2.8, 2.4

We have at this point two different description of the continuous-time Markov
chain: first as a hold-and-jump process, specified by a discrete time Markov
chain with transition probabilities given by a Π-matrix, and a sequence of ran-
dom times Sn defining the waiting times between two transition events. Second,
we have the process with a transition probabilities matrix P (t) and generator
Q so that P (t) = etQ . We need to show that these are different aspects of the
same process. We will follow the textbook from this point on.

References
[Nor] J.R. Norris. Markov Chains, Cambridge Series in Statistical and Proba-
bilistic Mathematics, Cambridge U. Press, 1997.
[Wil] Darren J. Wilkinson. Stochastic Modelling for Systems Biology, Chap-
man & Hall/CRC, 2006.

11