Modern Applied Science www.ccsenet.

org/mas

Application of Markov Chains to Analyze and Predict the Time Series

Tie Liu (Corresponding author)
Department of mathematics Ankang University AnKang, ShanXi, 725000, China
E-mail: [email protected]

Specialized Scientific Research Program on Scientific Research of high-level talents in Ankang University
(Program No. AYQDZR200705)

Abstract
Markov chains are usually used in modeling many practical problems. They are also effective in modeling time
series. In this paper, we apply the Markov chains model to analyze and predict the time series. Some series can
be expressed by a first-order discrete-time Markov chain and others must be expressed by a higher-order Markov
chain model. Numerical examples are given. The results show that the performance and effectiveness of the
Markov chain model to predict the time series is very well.
Keywords: Markov chains, Time series analysis, Mathematical Modeling
1. Introduction
Markov chains are useful tools in modeling many practical systems such as queuing systems (Ching, 2001.and
Sharma, 1995.), manufacturing systems (Buzacott & Shanthikumar, 1993.) and inventory systems (Ching, Fung
& Ng, 2003,pp.291–298 and Nahmias, 1997). Applications of Markov chains in modeling categorical data
sequences can also be found in (Ching, Fung & Ng, 2002, pp.87–199 and MacDonald & Zucchini, 1997). Time
series occur frequently in many real world applications. If one can model the time series accurately, then one can
make good predictions and also optimal planning in a decision process (Ching, Ng & Fung, 2008, pp.492–507).
In this paper, we apply the Markov chains model to analysis and predict the time series. Some series can be
expressed by a first-order discrete-time Markov chain and others must be expressed by a higher-order Markov
chain model. Numerical examples are given. The results show that the performance and effectiveness of the
Markov chain model to predict the time series is very well.
2. Markov chain model
2.1 The first-order Markov chain model
We consider modeling a time series xt by a first-order Markov chains having k states E={1,2……k}. A
first-order discrete-time Markov chain having k states satisfies the following relationship:
P ( xt +1 = it +1 | x0 = i0 , x1 = i1 ,L , xt = it ) = P ( xt +1 = it +1 | xt = it ) ,
where xt is the state of a time series at time t and ij ∈E. The conditional probabilities
P ( xt +1 = it +1 | xt = it )

are called the one-step transition probabilities of the Markov chain. These probabilities can be written as pij =
P(xt+1 = i| xt = j) for i and j in E. The matrix P=(pij)k×k is called the one-step transition probability matrix. We note
that the elements of the matrix P satisfy the following two properties:
k
0≤ pij ≤1 ∀i, j ∈ E and = 1 ,∀ j ∈ E
∑p
i =1
ij

A first-order Markov chain model

xt+1 = Pxt (1)
is then constructed for the observed time series.
We have the following well-known proposition for a transition matrix P. The proof can be found in (Horn &
Johnson, 1985, pp. 508–511) and therefore omitted here.
Proposition 1. The matrix P has an eigenvalue equal to one and all the eigenvalues of P must have modulus less
than or equal to one.
Generally one has the following proposition for a non-negative matrix, see for instance (Horn & Johnson, 1985,
pp. 508–511).

162
 Modern Applied Science Vol. 4, No. 5; May 2010

Proposition 2 (Perron–Frobenius theorem). Let A be a non-negative and irreducible square matrix of order m.
Then
(i) A has a positive real eigenvalue, λ, equal to its spectral radius, i.e., λ = max | λk ( A) | where λk ( A) denotes the
λ
kth eigenvalue of A.
(ii) To λ there corresponds an eigenvector x of its entries being real and positive, such that A x = λx.
(iii) λ is a simple eigenvalue of A.
By using the above two propositions, one can see that there exists a positive vector
x = [x1, x2,..., xm]T
such that P x = x if P is irreducible. The vector x in normalized form is called the stationary probability vector of
P. Moreover xi is the stationary probability that the system is in state i ( Ching, Ng & Fung, 2008, pp.492–507).
2.2 The higher-order Markov chain model
Higher-order (nth-order) Markov chain models have been proposed by Raftery (1985, p.528–539) and Ching et
al. (2004, p.557–574) for modeling categorical data sequences.
Ching, Ng & Fung (2008, pp.492–507) have suggested a series of modeling methods based on the Markov chain
(including the higher-order Markov chain model). We note that a time series {xt } of k states can be represented
by a series of vectors (probability distribution)
{x0, x1, x2, . . . }
called the canonical form representation. If the system is in state j ∈ E at time (t + i) then the state probability
distribution vector is given by
x t +i = (0,L , 0, {
1 , 0,L , 0)T .
jth entry

In addition, We assumes that the state probability distribution at time t = m + 1 depends on the state probability
distribution of the sequence at times t =m, m−1, . . . , m−n+1.
The model is given as follows:
n
x m+1 = ∑ λi Pi x n −i +1 , i = m − 1, m,L (2)
i =1

with initials x0, x1, . . . , xn−1. Here the weights λi are non-negative real numbers such that
n

∑λ i = 1. (3)
i =1

Here xm is the state probability distribution at time m, Pi is the i-step transition matrix and λi are the non-negative
weights. The total number of parameters is of O (nk2) (Ching, Ng & Fung, 2008, pp.492–507).
We estimate the transition probability matrix Pi by the following method. Given the data series, we count the
transition frequency from the states in the sequence at time t = m −i+ 1 to the states in the jth sequence at time t =
m + 1 for 1≤i≤n. Hence one can construct the transition frequency matrix for the data sequences. After making
normalization, the estimates of the transition probability matrices Pˆi can also be obtained.
Besides the estimates of Pi, we need to estimate the parameters λi. As a consequence of Proposition 1 and
Proposition 2, the nth order Markov chain has a stationary vector X. The vector xj can be estimated from the
sequences by computing the proportion of the occurrence of each state in the series.
As a consequence of Proposition 1 and Proposition 2,
n
Qx ≡ x, where Q = λ P .
∑ ii i =1

one would expect that

Qˆ xˆ ≈ xˆ . (4)
From (4), one possible way to estimate the parameters λi is given as follows. One may consider solving the
following minimization problem:

163
Modern Applied Science www.ccsenet.org/mas

min Qˆ xˆ − xˆ
λ
(5)
⎧ n
⎪ ∑ λi = 1
s.t. ⎨ i =1
⎪λ ≥ 0 i = 1,L , n
⎩ i
Here ||·|| is certain vector norm. If ||·|| is chosen to be the ||·||∞ norm then the above optimization problem becomes
n
min max | [∑ λi Pˆ xˆ −x]
ˆ i|
λ i
i =1 (6)
⎧ n
⎪ ∑ λi = 1
s.t. ⎨ i =1
⎪λ ≥ 0 i = 1,L , n
⎩ i
where [·]i denote the ith entry of the vector. Problem (6) can be formulated as s linear programming problems as
follows:
min ε
λ

⎧ ⎛ λ1 ⎞ ⎛ ε ⎞ ⎛ x1 ⎞
⎪ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟
λ ε
⎪− M ⎜ ⎟ − ⎜ ⎟ ≤ − ⎜ x2 ⎟
2
⎪ ⎜ M ⎟ ⎜M⎟ ⎜M ⎟
⎪ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ (7)
⎪ ⎝ λn ⎠ ⎝ ε ⎠ ⎝ xn ⎠
⎪ λ ε
⎪⎪ ⎛ 1⎞ ⎛ ⎞ ⎛ 1⎞ x
⎜ ⎟ ⎜ ⎟ ⎜ ⎟
⎜ λ2 ⎟ ⎜ ε ⎟ ⎜ x2 ⎟
s.t. ⎨
⎪ M ⎜ M ⎟−⎜ ⎟ ≤ ⎜ M ⎟
⎪ M
⎜ ⎟ ⎜ ⎟ ⎜ ⎟
⎪ ⎝ λn ⎠ ⎝ ε ⎠ ⎝ xn ⎠
⎪ n
⎪
⎪ ∑
i =1
λi = 1
⎪
⎪⎩ λi ≥ 0 i = 1,L , n
where M = [ Pˆ1xˆ Pˆ2 xˆ L Pˆn x]
ˆ .

We remark that other norms such as ||·||2 and ||·||1 can also be considered. The former will result in a quadratic
programming problem while the latter will still result in a linear programming problem Then we use the
higher-order Markov model to predict the next state of the sequence xˆt at time t which can be taken as the state
with the maximum probability, i.e.,
xˆt = j , if [xˆ t ]i ≤ [xˆ t ] j , ∀1 ≤ i ≤ k . (8)

To evaluate the performance and effectiveness of the higher-order Markov chain model, a prediction result is
measured by the prediction accuracy r defined as
N
1
s= × ∑ at × 100% ,
N − n t = n +1
where N is the length of the data sequence and
⎧1, if xˆt = xt .
at = ⎨
⎩0, otherwise
3. An application to analysis and predict the time series
In this section, we demonstrate the effectiveness of the Markov chain model and we apply it to the price and
sales volume for beef prediction problems in a supermarket. The time series of the price and sales volume for
beef in a supermarket in AnKang is given in appendix. The price of beef can be classified into five possible
states (1, 2, 3, 4, 5), see Appendix. The price series are expressed as 1 = very low (≤RMB 29/kg), 2 = low (RMB
29∼32/kg), 3 = middle (RMB 32∼35/kg), 4 = high (RMB 35∼38/kg), 5 = very high (≥RMB 38/kg).Similarly, the
sales volume of beef can also be classified into five possible states (1, 2, 3, 4, 5), see Appendix. The sales
volume series are expressed as 1 = very low (≤50kg), 2 = low (50kg ∼55kg), 3 = middle (55kg ∼60kg), 4 = high
(60kg ∼65kg), 5 = very high (≥65kg). Such expressions are useful from both marketing and production planning
points of view.

164
Modern Applied Science Vol. 4, No. 5; May 2010

On the one hand the supermarket sales demand for beef in order to minimize its inventory build-up, on the other
hand the customers would like to predict the price of beef to decide purchase strategy. Moreover, the
supermarket can understand the sales pattern of his customer and then develop a marketing policy to deal with
customers.
With the price series, today’s price mostly dependent on yesterday’s price. We choose the first-order Markov
chain model. We first estimate the one-step transition probability matrix P by using the method said above.
⎛ 0.2917 0.0370 0.1500 0.1429 0.1096 ⎞
⎜ ⎟
⎜ 0.1250 0.3704 0.1500 0.2500 0.3014 ⎟
P = ⎜ 0.2083 0.0741 0.2000 0.0714 0.0685 ⎟
⎜ ⎟
⎜ 0.1250 0.1296 0.2500 0.1429 0.1233 ⎟
⎜ 0.2500 0.3889 0.2500 0.3929 0.3973 ⎟⎠
⎝
And we also have the estimates of the stationary probability distributions of the price series
x̂ = [0.1200 0.2750 0.1000 0.1400 0.3650]T
The prediction accuracy of the proposed model is r1 =0.5362.
But the sales volume series are much more complicated. We choose the order arbitrarily to be five, i.e., n = 5.
We first estimate all the transition probability matrices Pi by using the method proposed above and we also have
the estimates of the stationary probability distributions of the product:
x̂ = [0.3350 0.1350 0.2150 0.0600 0.2550]T
By solving the corresponding linear programming problems in (7), we obtain the following higher-order Markov
chain model:
x m +1 = 0.7022P1x m +0.0768P4 x m−3 + 0.2210P5 x m− 4 ,
where
⎛ 0.4776 0.2222 0.0233 0.0909 0.5294 ⎞
⎜ 0.1045 0.2593 0.1163 0.0909 0.1373 ⎟⎟
⎜
P1 = ⎜ 0.0149 0.2963 0.6279 0.3636 0.0588 ⎟
⎜ ⎟
⎜ 0.0149 0 0.1395 0.4545 0 ⎟
⎜ 0.3881 0.2222 0.0930 0 0.2745 ⎟⎠
⎝
⎛ 0.3538 0.2593 0.1860 0.2000 0.4706 ⎞
⎜ ⎟
⎜ 0.1692 0.1481 0.2093 0.2000 0.0196 ⎟
P4 = ⎜ 0.1846 0.2593 0.3023 0.1000 0.1961 ⎟
⎜ ⎟
⎜ 0.0462 0.1111 0.0930 0 0.0392 ⎟
⎜ 0.2462 0.2222 0.2093 0.5000 0.2745 ⎟⎠
⎝
⎛ 0.4063 0.2593 0.2093 0.4000 0.3333 ⎞
⎜ ⎟
⎜ 0.0625 0.2222 0.2326 0 0.1373 ⎟
P5 = ⎜ 0.2656 0.2222 0.1860 0.1000 0.2157 ⎟
⎜ ⎟
⎜ 0.0469 0.1852 0.0465 0 0.0392 ⎟
⎜ 0.2188 0.1111 0.3256 0.5000 0.2745 ⎟⎠ .
⎝

We can also see that the prediction accuracy of the proposed model is r2=0.5588. All results show that the
effectiveness of the Markov model to analyze and predict the time series is very well.
4. Summary
In this paper, we applied the Markov model to analyze and predict the time series. Some series can be expressed
by a first-order discrete-time Markov chain and others must be expressed by a higher-order Markov chain model.
Numerical examples are given. We applied it to the price and sales volume for beef prediction problems in a
supermarket. The results show that the performance and effectiveness of the Markov chain model to predict the
time series is very well.
Appendix. Price series of beef in a supermarket
555545353342553111334151133251515555214111245514241342252255

165
Modern Applied Science www.ccsenet.org/mas

254442252255553225452455411122324555252552425525512343313143
545545525252235535254215252222554552252223444545155135551522
255552452252522242224555322525444533531142255255525535541551
55151555515552125255235555525
1 = very low (≤RMB 29/kg), 2 = low (RMB 29∼32/kg), 3 = middle (RMB 32∼35/kg), 4 = high (RMB 35∼38/kg),
5 = very high (≥RMB 38/kg).
Sales demand series of beef in a supermarket
511112233343225155123323322115233332312115225523434221551515
511533333343351551151551551551515233333335155151555515115333
333522511151511111111111125344441355155115111151211252112333
344322515111521111443333251551515112233433111211511151111111
2331143132111115515151111111
1 = very low (≤50kg), 2 = low (50kg ∼55kg), 3 = middle (55kg ∼60kg), 4 = high (60kg ∼65kg), 5 = very high
(≥65kg).
References
Buzacott, J. & Shanthikumar, J. (1993). Stochastic Models of Manufacturing Systems, New Jersey: International
ed. Prentice-Hall, (Chapter 7).
Ching, W. (2001). Iterative Methods for Queuing and Manufacturing Systems. London: Springer, (Chapter 2).
Ching, W., Fung, E. & Ng, M. (2002). A multivariate Markov chain model for categorical data sequences and its
applications in demand predictions. IMA J. Manage. Math. 13, pp. 87–199.
Ching, W., Fung, E. & Ng, M. (2004). Higher-order Markov chain models for categorical data sequences. Int. J.
Nav. Res. Logist. 51, pp. 557–574.
Ching, W., Ng, M. & Fung, E. (2008). Higher-order multivariate Markov chains and their applications. Linear
Algebra and its Applications 428, pp.492–507
Horn, R. & Johnson, C. (1985). Matrix Analysis, UK, Cambridge: Cambridge University Press, (Chapter 8).
MacDonald, I. & Zucchini, W. (1997). Hidden Markov and Other Models for Discrete-valued Time Series.
London: Chapman & Hall, (Chapter 5).
Nahmias, S. (1997). Production and Operation Analysis. Chicago: McGraw Hill International.
Sharma, O. (1995). Markovian Queues. New York: Ellis Horwood, (Chapter 4).

166