Markov Process and Application

Muhammad Farooq

Virtual University of Pakistan

[email protected]

Abstract A Markov chain is a type of Markov process that

has either a discrete state space or a discrete
Markov chain was named after Andrew Markov
index set (often representing time), but the
(Russian Mathematician). It is a mathematical
precise definition of a Markov chain varies. For
system, which moves from a particular form to
example, it is common to define a Markov chain
the other particular form and the property of
as a Markov process in either discrete or
memorylessness given that the subsequent
continuous time with a countable state space.
form relies on the present form, but not the
whole sequence involved. This is the Markov A Markov chain is a sequence of random
chain’s characteristic. Markov chain is variables {Xn; n =0,1,2,...}, defined on some
applicable in different real-world processes as probability space (Ω,F,P), taking its values in a
statistical models and derived from random set E which could be arbitrary, but which for us
transitional process. will be either finite or countable, and which
possesses the Markov property. Intuitively, a
Keywords:discrete-time Markov Chain
Markov chain has the property that, knowing
process,continuous-time Markov
the present state Xn, one can forget the past if
Chain,applications
one wants to predict the future. One way to
construct a Markov chain is as follows. Let {Yn;
n ≥1} be mutually independent
Introduction
F-valued random variables, which are globally
A random procedure or system having the independent of X0. Given a mapping f : N×E ×F
attributes of Markov is a Markov chain. The → E, we define{Xn;n ≥1} recursively by Xn =
chain in the Markov system is the sequence of a f(n,Xn−1,Yn). In a way, this is the simplest
stochastic process in which the next stage is model of non-mutually independent random
dependent on the current stage and not the variables. The next two chapters will present
whole sequence. Thus, it can be a useful tool in many applications of Markov chains. Note that
describing a system dependent on a linked we shall restrict our presentation to
chain of events. homogeneous Markov chains (in the above
In simpler terms, a Markov process is a process recurrence relation, f does not depend upon n,
for which one can make predictions for its and the Yn all have the same law), even though
future based solely on its present state just as non-homogeneous chains are necessary in
well as one could knowing the process's full many applications. Even in those cases,
history. In other words, conditional on the understanding the long time behavior of the
present state of the system, its future and past homogeneous chains is crucial.
states are independent. Different definitions are as follows.
Definition 1 In 1912 Henri Poincaré studied Markov chains
on finite groups with an aim to study card
Markov chain as having discrete time in either
shuffling.
countable or continuous state space (thus
regardless of the state space). Diffusion model, introduced by Paul and
Tatyana Ehrenfest in 1907. In 1931, Andrei
Definition 2
Kolmogorov wrote a paper on continuous-time
A stochastic process is a sequence of events in theory. He introduced and studied a particular
which the outcome at any stage depends on set of Markov processes known as diffusion
some probability. processes, where he derived a set of differential
equations describing the processes.
Definition 3
The differential equations are now called the
A Markov process is a stochastic process with Kolmogorov equations or the Kolmogorov–
the following properties: Chapman equations. Other mathematicians
(a.) The number of possible outcomes or states who contributed significantly to the foundations
is finite. of Markov processes include William Feller,
starting in 1930s, and then later Eugene Dynkin,
(b.) The outcome at any stage depends only on starting in the 1950s.
the outcome of the previous stage.
Explanation
(c.) The probabilities are constant over time.
Ifx0 is a vector which represents the initial state
History of a system, then there is a matrix M such that
In developing Markov process/chain many the state of the system after one iteration is
scientists and mathematician have contributed. given by the vector Mx0. Thus we get a chain of
state vectors: x0, Mx0, M2x0,... where the state
In 20th century, first paper is published in 1906 of the system after n iterations is given by
written by Andry Markov in the form of Poisson Mnx0. Such a chain is called a Markov chain and
process. he was interested in studying an the matrix M is called a transition matrix. The
extension of independent random sequences, state vectors can be of one of two types: an
motivated by a disagreement with Pavel absolute vector or a probability vector. An
Nekrasov who claimed independence was absolute vector is a vector whose entries give
necessary for the weak law of large numbers to the actual number of objects in a give state, as
hold. Markov showed that under certain in the first example. A probability vector is a
conditions the average outcomes of the Markov vector where the entries give the percentage
chain would converge to a fixed vector of (or probability) of objects in a given state. We
values, so proving a weak law of large numbers will take all of our state vectors to be
without the independence assumption which probability vectors from now on. Note that the
had been commonly regarded as a requirement entries of a probability vector add up to 1.
for such mathematical laws to hold. Markov
later used Markov chains to study the Discrete-time Markov Chain
distribution of vowels in Eugene Onegin, written A discrete-time Markov chain is a sequence of
by Alexander Pushkin, and proved a central random variables X1, X2, X3 …. With Markov
limit theorem for such chains. property, namely that the probability of moving
to the next state depends only on the present a wide range of topics such as physics,
state and not on the previous state chemistry, medicine, music, game theory and
sports.

Physics

Markovian systems appear extensively in

thermodynamics and statistical mechanics,
Continuous-time Markov chain
whenever probabilities are used to represent
A continuous-time Markov chain (X t)t ≥ 0 is unknown or unmodelled details of the system,
defined by a finite or countable state space S, a if it can be assumed that the dynamics are time-
transition rate matrix Q with dimensions equal invariant, and that no relevant history need be
to that of the state space and initial probability considered which is not already included in the
distribution defined on the state space. For i ≠ j, state description. For example, a
the elements qij are non-negative and describe thermodynamic state operates under a
the rate of the process transitions from state i probability distribution that is difficult or
to state j. The elements q ii are chosen such that expensive to acquire. Therefore, Markov Chain
each row of the transition rate matrix sums to Monte Carlo method can be used to draw
zero, while the row-sums of a probability samples randomly from a black-box to
transition matrix in a (discrete) Markov chain approximate the probability distribution of
are all equal to one. attributes over a range of objects. The paths, in
the path integral formulation of quantum
Markov Chain in Tabular Form mechanics, are Markov chains.[64] Markov
countable Continuous chains are used in lattice QCD simulations.
state space time Chemistry
Continuous
or general A reaction network is a chemical system
state space involving multiple reactions and chemical
Discrete-time (discrete- Markov species. The simplest stochastic models of such
time) Markov chain on a networks treat the system as a continuous time
chain on a measurable Markov chain with the state being the number
countable or state space of molecules of each species and with reactions
finite state (for example,
modeled as possible transitions of the chain.
space Harris chain)
[66] Markov chains and continuous-time
Continuous- Any
Markov processes are useful in chemistry when
Continuous- time Markov continuous
time process or stochastic physical systems closely.
Markov jump process with The classical model of enzyme activity,
process the Markov
Michaelis–Menten kinetics, can be viewed as a
property (for
Markov chain, where at each time step the
example, the
reaction proceeds in some direction. While
Wiener
process) Michaelis-Menten is fairly straightforward, far
more complicated reaction networks can also
be modeled with Markov chains.
Applications Research has reported the
application and usefulness of Markov chains in
An algorithm based on a Markov chain was also Solar irradiance variability
used to focus the fragment-based growth of
Solar irradiance variability assessments are
chemicals in silico towards a desired class of
useful for solar power applications. Solar
compounds such as drugs or natural products.
irradiance variability at any location over time is
As a molecule is grown, a fragment is selected
mainly a consequence of the deterministic
from the nascent molecule as the "current"
variability of the sun's path across the sky dome
state. It is not aware of its past (that is, it is not
and the variability in cloudiness. The variability
aware of what is already bonded to it). It then
of accessible solar irradiance on Earth's surface
transitions to the next state when a fragment is
has been modeled using Markov chains, also
attached to it. The transition probabilities are
including modeling the two states of clear and
trained on databases of authentic classes of
cloudiness as a two-state Markov chain.
compounds.
Speech recognition
Also, the growth (and composition) of
copolymers may be modeled using Markov Hidden Markov models are the basis for the
chains. Based on the reactivity ratios of the most modern automatic speech recognition
monomers that make up the growing polymer systems.
chain, the chain's composition may be
calculated (for example, whether monomers Information and Computer Science
tend to add in alternating fashion or in long Markov chains are used throughout information
runs of the same monomer). Due to steric processing. Claude Shannon's famous 1948
effects, second-order Markov effects may also paper A Mathematical Theory of
play a role in the growth of some polymer Communication, which in a single step created
chains. the field of information theory, opens by
Similarly, it has been suggested that the introducing the concept of entropy through
crystallization and growth of some epitaxial Markov modeling of the English language. Such
super lattice oxide materials can be accurately idealized models can capture many of the
described by Markov chains. statistical regularities of systems. Even without
describing the full structure of the system
Testing perfectly, such signal models can make possible
very effective data compression through
Several theorists have proposed the idea of the
entropy encoding techniques such as arithmetic
Markov chain statistical test (MCST), a method
coding. They also allow effective state
of conjoining Markov chains to form a "Markov
estimation and pattern recognition. Markov
blanket", arranging these chains in several
chains also play an important role in
recursive layers ("wafering") and producing
reinforcement learning.
more efficient test sets—samples—as a
replacement for exhaustive testing. MCSTs also Markov chains are also the basis for hidden
have uses in temporal state-based networks; Markov models, which are an important tool in
Chilukuri et al.'s paper entitled "Temporal such diverse fields as telephone networks
Uncertainty Reasoning Networks for Evidence (which use the Viterbi algorithm for error
Fusion with Applications to Object Detection correction), speech recognition and
and Tracking" (Science Direct) gives a bioinformatics (such as in rearrangements
background and case study for applying MCSTs detection).
to a wider range of applications.
The LZMA lossless data compression algorithm crashes. The first financial model to use a
combines Markov chains with Lempel-Ziv Markov chain was from Prasad et al. in
compression to achieve very high compression 1974.Another was the regime-switching model
ratios. of James D. Hamilton (1989), in which a Markov
chain is used to model switches between
periods high and low GDP growth (or
Queuing theory alternatively, economic expansions and
recessions). A more recent example is the
Markov chains are the basis for the analytical Markov switching multifractal model of Laurent
treatment of queues (queueing theory). Agner E. Calvet and Adlai J. Fisher, which builds upon
Krarup Erlang initiated the subject in 1917. This the convenience of earlier regime-switching
makes them critical for optimizing the models. It uses an arbitrarily large Markov chain
performance of telecommunications networks, to drive the level of volatility of asset returns.
where messages must often compete for
limited resources (such as bandwidth). Dynamic macroeconomics heavily uses Markov
chains. An example is using Markov chains to
Numerous queuing models use continuous-time exogenously model prices of equity (stock) in a
Markov chains. For example, an M/M/1 queue general equilibrium setting.
is a CTMC on the non-negative integers where
upward transitions from i to i + 1 occur at rate λ Credit rating agencies produce annual tables of
according to a Poisson process and describe job the transition probabilities for bonds of
arrivals, while transitions from i to i – 1 (for i > different credit rating.
1) occur at rate μ (job service times are
Social Sciences
exponentially distributed) and describe
completed services (departures) from the Markov chains are generally used in describing
queue. path-dependent arguments, where current
structural configurations condition future
statistics
outcomes. An example is the reformulation of
Markov chain methods have also become very the idea, originally due to Karl Marx's Das
important for generating sequences of random Kapital, tying economic development to the rise
numbers to accurately reflect very complicated of capitalism. In current research, it is common
desired probability distributions, via a process to use a Markov chain to model how once a
called Markov chain Monte Carlo (MCMC). In country reaches a specific level of economic
recent years this has revolutionized the development, the configuration of structural
practicability of Bayesian inference methods, factors, such as size of the middle class, the
allowing a wide range of posterior distributions ratio of urban to rural residence, the rate of
to be simulated and their parameters found political mobilization, etc., will generate a
numerically. higher probability of transitioning from
authoritarian to democratic regime.

Mathematical biology
Economics and finance
Markov chains also have many applications in
Markov chains are used in finance and biological modelling, particularly population
economics to model a variety of different processes, which are useful in modelling
phenomena, including asset prices and market processes that are (at least) analogous to
biological populations. The Leslie matrix, is one A second-order Markov chain can be introduced
such example used to describe the population by considering the current state and also the
dynamics of many species, though some of its previous state, as indicated in the second table.
entries are not probabilities (they may be Higher, nth-order chains tend to "group"
greater than 1). Another example is the particular notes together, while 'breaking off'
modeling of cell shape in dividing sheets of into other patterns and sequences occasionally.
epithelial cells. Yet another example is the state These higher-order chains tend to generate
of ion channels in cell membranes. results with a sense of phrasal structure, rather
than the 'aimless wandering' produced by a
Markov chains are also used in simulations of
first-order system. Markov chains can be used
brain function, such as the simulation of the
structurally, as in Xenakis's Analogique A and B.
mammalian neocortex. Markov chains have also
Markov chains are also used in systems which
been used to model viral infection of single cells
use a Markov model to react interactively to
Genetics music input.

Markov chains have been used in population Usually musical systems need to enforce
genetics in order to describe the change in gene specific control constraints on the finite-length
frequencies in small populations affected by sequences they generate, but control
genetic drift, for example in the diffusion constraints are not compatible with Markov
equation method described by Motoo Kimura. models, since they induce long-range
dependencies that violate the Markov
Games hypothesis of limited memory. In order to
Markov chains can be used to model many overcome this limitation, a new approach has
games of chance.[1] The children's games been proposed.
Snakes and Ladders and "Hi Ho! Cherry-O", for Baseball
example, are represented exactly by Markov
chains. At each turn, the player starts in a given Markov chain models have been used in
state (on a given square) and from there has advanced baseball analysis since 1960, although
fixed odds of moving to certain other states their use is still rare. Each half-inning of a
(squares) baseball game fits the Markov chain state when
the number of runners and outs are considered.
During any at-bat, there are 24 possible
Music combinations of number of outs and position of
the runners. Mark Pankin shows that Markov
Markov chains are employed in algorithmic chain models can be used to evaluate runs
music composition, particularly in software such created for both individual players as well as a
as Csound, Max, and Super Collider. In a first- team.[100] He also discusses various kinds of
order chain, the states of the system become strategies and play conditions: how Markov
note or pitch values, and a probability vector for chain models have been used to analyze
each note is constructed, completing a statistics for game situations such as bunting
transition probability matrix (see below). An and base stealing and differences when playing
algorithm is constructed to produce output on grass vs. AstroTurf.
note values based on the transition matrix
weightings, which could be MIDI note values, Markov text generators
frequency (Hz), or any other desirable metric.
Markov processes can also be used to generate
superficially real-looking text given a sample
document. Markov processes are used in a
variety of recreational "parody generator"
software

Bioinformatics

In the bioinformatics field, they can be used to

simulate DNA sequences

Probabilistic forecasting

Markov chains have been used for forecasting

in several areas: for example, price trends, wind
power, and solar irradiance. The Markov chain
forecasting models utilize a variety of settings,
from discretizing the time series, to hidden
Markov models combined with wavelets, and
the Markov chain mixture distribution model
(MCM).