Scribd
Upload
185 views

Section 12 Continuous Time Markov Chains

The document discusses continuous-time Markov chains including their definition, transition probability functions, Chapman-Kolmogorov equation, embedded discrete-time processes, transient state behavior using Kolmogorov's equations, and steady state behavior. It also provides examples of Poisson counting processes and birth-and-death processes.

Uploaded by

Dylan Ler
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
185 views

Section 12 Continuous Time Markov Chains

The document discusses continuous-time Markov chains including their definition, transition probability functions, Chapman-Kolmogorov equation, embedded discrete-time processes, transient state behavior using Kolmogorov's equations, and steady state behavior. It also provides examples of Poisson counting processes and birth-and-death processes.

Uploaded by

Dylan Ler
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

Continuous-Time

Markov Chains
Professor Izhak Rubin
Electrical Engineering Department
UCLA
[email protected]

2014-2015 by Izhak Rubin

Continuous-Time Markov Chains:


Definition
Definition: The stochastic process X X t , t 0 whose
states assume values in a countable state space S,
where X t denotes the state of the process at time t 0, is
said to be a continuous-time (or continuous parameter)
Markov Chain (CTMC) if it satisfies the Markov Property (MP):
Markov Property:
P X t s y X u , u t P X t s y X t , y S , t , s 0 .

Prof. Izhak Rubin

Transition Probability Function


Its Transition Probability Function (TPF) is defined as
Ps ,t x, y

P X t y X s x , x,y S , t s 0

Consider only time-homogeneous Markov chains, for which:


Ps ,t x, y P0,t s x, y Pt s x, y , x,y S , t s 0.
The Transition Probability Function (TPF) satisfies the
following properties:
1. Pt x, y 0, x, y S , t 0
2.

P x, y 1, t 0, x S
yS

3. Pt s x, y Pt x, z Ps z , y , x, y S , s,t 0.
zS

(Chapman Kolmogorov Equation: CKE)


Prof. Izhak Rubin

Chapman-Kolmogorov Equation
and Poisson Counting Process
Proof of Chapman Kolmogorov Equation:
Pt s x, y P X t s y X 0 x
P X t s y, X t z X 0 x
zS

P X t s y X t z, X 0 x P X t z X 0 x
zS

P X t s y X t z P X t z X 0 x Pt x, z Ps z , y
zS

zS

s,t 0, x,y S.
Example: Let N N t , t 0 be the Poisson Counting Process with intensity ,
over the state space of non-negative integers S = {0,1,2,...}.
Then, we readily show that:
1. N satisfies the Markov Property
2. Pt x, y e

t I y x 0 ,

y x !
yx

t 0, x,y S.

Prof. Izhak Rubin

Probabilistic Structure
We consider a CTMC X with a standard TPF for which
lim Pt i, j ij , i, j S
t 0

We define Vt to designate the residual time period occupied by the state


assumed by the process at time t.
Theorem : P Vt x X t i e qi x , x 0, i S , t 0, for some qi 0
If qi , we set e qi x 0.
A state i is said to be absorbing if qi 0.
A state i is said to be stable if 0 qi .
A state i is said to be instantaneous if qi .
A Markov chain with a finite state space contains no instantaneous states.
Prof. Izhak Rubin

Probabilistic Structure: Embedded DT Processes


Normally (when the embedded point process is honest), a realization of a
Markov chain X, also known as a stochastic jump process, can be described
in terms of two discrete-time embedded processes:
1. Embedded Point Process representing the times at which jumps occur,
A An , n 0 , where A n = time at which the n-th jump occurs
2. Embedded State Sequence representing the successive states into which
the process jumps; , Y Yn , n 0 , Yn X A = state following the n-th jump
n

Xt

Y3

Y1
Y0

A0 A1

Y4
Y2
A2

A3

A4

Prof. Izhak Rubin

t
6

Probabilistic Structure: Embedded


Processes
Y is a DT Markov Chain; its TPF is denoted as R = {R(i, j ), i, j S }:
P Yn 1 j | Y0 , Y1 ,..., Yn i P Yn 1 j | Yn i R i, j
where R i, j 0, R i, i 0,

R i, j 1.
jS

Example :
For Poisson counting process N N t , t 0 with intensity ,
P An 1 An t | Yn i e t , t 0, qi 0, each i S.
For the embedded state sequence Y , the TPF is given as
R i, j j ,i 1 , i,j S; where j ,i 1 is the Kronecker Delta function, so that

j ,i 1 =1 if j = i+1 and = 0, otherwise.

Prof. Izhak Rubin

Transient State Behavior


Let X be a regular continuous time Markov chain.
Given {qi }, {R(i. j )} and the TPF {Pt (i, j )}.
For each t 0, i, j S , Pt (i, j ) is differentiable and its derivative is continuous.
At t 0, the derivative is given by:
i j
qi ,
d
qij lim Pt i, j
t 0 dt
qi R i, j , i j
Infinitesimal Generator of the Markov Chain: Q qij , i, j S .
Also, for i,j S:
qij
, i j , qi qii 0

R i, j qi
0, i j, q 0
i

Prof. Izhak Rubin

Transition
Intensities
We define:
qij

d
1
Pt i, j lim [ Ph i, j P0 i, j ]; P0 i, j ij
t 0 dt
h 0 h

lim

so that
Ph (i, j ) qij h o(h); for i j; where lim [o(h)/h] = 0
h 0

1 Ph (i, i ) qi h o(h); where lim [o(h)/h] = 0


h 0

Noting that qi qii

and that

q
jS

ij

0.

Thus :
1
1
For i j: qij lim [ Ph i, j ] lim [ Avg.# transitions i j over (0, h)]
h 0 h
h 0 h
transition rate from state i to state j;
1
1
qi lim [1 Ph i, i ] lim [ Avg.# transitions out of state i over (0, h)]
h0 h
h0 h
transition rate out of state i.
Prof. Izhak Rubin

Transient State Behavior


Kolmogorov's Backward Equations
d
Pt i, j qik Pt k , j , i, j S
dt
kS
Kolmogorov's Forward Equations
d
Pt i, j Pt i, k qkj , i, j S ,
dt
kS
Proof of KFE:
Pt h i, j Pt i, k Ph k , j
k

1
1
Pt h i, j Pt i, j Pt i, k Ph k , j kj
h
h
k
1
d
lim Pt h i, j Pt i, j Pt i, j Pt i, k qkj
h 0 h
dt
k
Prof. Izhak Rubin

10

Steady State Behavior


Consider a CTMC X X t , t 0 with an irreducible recurrent
Markov chain Y . The steady state distribution for X is:
P j lim P X t j ,
t

jS

Using KFE,
d
lim Pt i, j 0
t dt
P k qkj 0, j S
kS

Since qii q j , we obtain the following set of balance equations:


P j q j P k qkj ,

jS

(2.1)

k j

P j 1.

(2.2)

jS

Prof. Izhak Rubin

11

CT Birth-and-Death Process
A continuous time process X X t , t 0 , S = {0,1,2,...}, is defined to be a
Birth and Death process if it is a CTMC with the following generator:
j i 1, i 0
i ,
,
j i 1, i 1
i
qij
i i , j i, i 0
0,
o.w.
where qi i i , 0 0; i 0, i 0; i 0, i 1.
The TPF for the embedded DTMC Y is thus given by:
i
, j i 1

qij i i
R i, j

qi i
, j i 1
i i
for qi 0 and i 1; R 0,1 1.
Prof. Izhak Rubin

12

Birth-and-Death Process: Steady State


The limiting (steady state) probabilities P P j , j S , for j 0, j 0;

j 0 for j 0, are calculated as follows :


For j 0 :
For j 1:

P 0 0 P 1 1.

P j j j P j 1 j 1 P j 1 j 1.

P 0 0 P 1 1 0

P j j P j 1 j 1 P j 1 j 1 P j j 0
0

P
1

P 0

j 1
i
P j P 0

i 0 i 1
j 1

Let a j
i 0

i
for j 1, and set a0 1 ; then :
i 1

P j P 0 a j , j 0 .
Prof. Izhak Rubin

13

CT Birth-and-Death Process: Steady


State Distribution

Case 1:

a
j 0

P 0 1

; then, the steady-state distribution exists and is given by

a
j 0

, P i ai

a
j 0

, i 0.

Case 2: a j ; then, steady-state distribution does not exist, and we have:


j 0

lim Pi X t j 0,
t

j 0.

Prof. Izhak Rubin

14

Finite-State Birth and Death


Process
A continuous time process X X t , t 0 , S = {0,1,2,...,N}, N < is defined
to be a finite-state Birth and Death process if it is a CTMC with the following generator:
j i 1, N>i 0
i ,
,
j i 1, N i 1

qij i
i i , j i, N i 0
0,
o.w.
where qi i i , 0 0, N 0, and i 0, i 0, otherwise.
The TPF for the embedded DTMC Y is thus given by:
i
, j i 1
qij i i
R i, j

qi i
, j i 1
i i
for qi 0 and i 1, j N ; R 0,1 1, R N , N 1 1.
Prof. Izhak Rubin

15

Finite-State Birth and Death Process:


Steady State Distribution
X = CTMC birth-and-death with a finite state space: S {0,1,..., N }
The derivation of the steady state probabilities proceeds as performed for the
infinite state space case.
N

Since

a
j 0

always exists; the steady-state distribution always exists

(assuming positive birth and rate transition intensities, except on the boundaries)
and is given by
P i P 0 ai ai
noting that

P 0 1

a
j 0

, i0

a
j 0

.
Prof. Izhak Rubin

16

You might also like