Abstract

Markov chain was named after Andrew Markov (Russian Mathematician). It is a mathematical
system, which moves from a particular form to the other particular form and the property of
memorylessness given that the subsequent form relies on the present form, but not the whole
sequence involved. This is the Markov chain’s characteristic. Markov chain is applicable in different
real-world processes as statistical models and derived from random transitional process.

Introduction

A random procedure or system having the attributes of Markov is a Markov chain. The chain in the
Markov system is the sequence of a stochastic process in which the next stage is dependent on the
current stage and not the whole sequence. Thus, it can be a useful tool in describing a system
dependent on a linked chain of events.

In simpler terms, a Markov process is a process for which one can make predictions for its future
based solely on its present state just as well as one could knowing the process's full history. In other
words, conditional on the present state of the system, its future and past states are independent.

A Markov chain is a type of Markov process that has either a discrete state space or a discrete index
set (often representing time), but the precise definition of a Markov chain varies. For example, it is
common to define a Markov chain as a Markov process in either discrete or continuous time with a
countable state space.

A Markov chain is a sequence of random variables {Xn; n =0,1,2,...}, defined on some probability
space (Ω,F,P), taking its values in a set E which could be arbitrary, but which for us will be either
finite or countable, and which possesses the Markov property. Intuitively, a Markov chain has the
property that, knowing the present state Xn, one can forget the past if one wants to predict the
future. One way to construct a Markov chain is as follows. Let {Yn; n ≥1} be mutually independent

F-valued random variables, which are globally independent of X0. Given a mapping f : N×E ×F → E,
we define{Xn;n ≥1} recursively by Xn = f(n,Xn−1,Yn). In a way, this is the simplest model of non-
mutually independent random variables. The next two chapters will present many applications of
Markov chains. Note that we shall restrict our presentation to homogeneous Markov chains (in the
above recurrence relation, f does not depend upon n, and the Yn all have the same law), even
though non-homogeneous chains are necessary in many applications. Even in those cases,
understanding the long time behavior of the homogeneous chains is crucial.

Different definitions are as follows.

Definition 1

Markov chain as having discrete time in either countable or continuous state space (thus regardless
of the state space).
Definition 2

A stochastic process is a sequence of events in which the outcome at any stage depends on some
probability.

Definition 3

A Markov process is a stochastic process with the following properties:

(a.) The number of possible outcomes or states is finite.

(b.) The outcome at any stage depends only on the outcome of the previous stage.

(c.) The probabilities are constant over time.

History

In developing Markov process/chain many scientists and mathematician have contributed.

In 20th century, first paper is published in 1906 written by Andry Markov in the form of Poisson
process. he was interested in studying an extension of independent random sequences, motivated
by a disagreement with Pavel Nekrasov who claimed independence was necessary for the weak law
of large numbers to hold. Markov showed that under certain conditions the average outcomes of
the Markov chain would converge to a fixed vector of values, so proving a weak law of large
numbers without the independence assumption which had been commonly regarded as a
requirement for such mathematical laws to hold. Markov later used Markov chains to study the
distribution of vowels in Eugene Onegin, written by Alexander Pushkin, and proved a central limit
theorem for such chains.

In 1912 Henri Poincaré studied Markov chains on finite groups with an aim to study card shuffling.

Diffusion model, introduced by Paul and Tatyana Ehrenfest in 1907. In 1931, Andrei Kolmogorov
wrote a paper on continuous-time theory. He introduced and studied a particular set of Markov
processes known as diffusion processes, where he derived a set of differential equations describing
the processes.

The differential equations are now called the Kolmogorov equations or the Kolmogorov–Chapman
equations. Other mathematicians who contributed significantly to the foundations of Markov
processes include William Feller, starting in 1930s, and then later Eugene Dynkin, starting in the
1950s.

Explanation

Ifx0 is a vector which represents the initial state of a system, then there is a matrix M such that the
state of the system after one iteration is given by the vector Mx0. Thus we get a chain of state
vectors: x0, Mx0, M2x0,... where the state of the system after n iterations is given by Mnx0. Such a
chain is called a Markov chain and the matrix M is called a transition matrix. The state vectors can be
of one of two types: an absolute vector or a probability vector. An absolute vector is a vector whose
entries give the actual number of objects in a give state, as in the first example. A probability vector
is a vector where the entries give the percentage (or probability) of objects in a given state. We will
take all of our state vectors to be probability vectors from now on. Note that the entries of a
probability vector add up to 1.

Discrete-time Markov Chain

A discrete-time Markov chain is a sequence of random variables X1, X2, X3 …. With Markov property,
namely that the probability of moving to the next state dependes only on the present state and not
on the previous state

Continuous-time Markov chain

A continuous-time Markov chain (Xt)t ≥ 0 is defined by a finite or countable state space S, a transition
rate matrix Q with dimensions equal to that of the state space and initial probability distribution
defined on the state space. For i ≠ j, the elements qij are non-negative and describe the rate of the
process transitions from state i to state j. The elements qii are chosen such that each row of the
transition rate matrix sums to zero, while the row-sums of a probability transition matrix in a
(discrete) Markov chain are all equal to one.

Markov Chain in Tabular Form

countable state space Continuous time

Continuous or general state
space
Discrete-time (discrete-time) Markov chain Markov chain on a measurable
on a countable or finite state state space (for example,
space Harris chain)
Continuous-time Markov Any continuous stochastic
Continuous-time process or Markov jump process with the Markov
process property (for example, the
Wiener process)

Applications Research has reported the application and usefulness of Markov chains in a wide range
of topics such as physics, chemistry, medicine, music, game theory and sports.

Physics

Markovian systems appear extensively in thermodynamics and statistical mechanics, whenever

probabilities are used to represent unknown or unmodelled details of the system, if it can be
assumed that the dynamics are time-invariant, and that no relevant history need be considered
which is not already included in the state description.[62][63] For example, a thermodynamic state
operates under a probability distribution that is difficult or expensive to acquire. Therefore, Markov
Chain Monte Carlo method can be used to draw samples randomly from a black-box to approximate
the probability distribution of attributes over a range of objects.[63] The paths, in the path integral
formulation of quantum mechanics, are Markov chains.[64] Markov chains are used in lattice QCD
simulations.[65]

Chemistry

A reaction network is a chemical system involving multiple reactions and chemical species. The
simplest stochastic models of such networks treat the system as a continuous time Markov chain
with the state being the number of molecules of each species and with reactions modeled as
possible transitions of the chain.[66] Markov chains and continuous-time Markov processes are
useful in chemistry when physical systems closely.

The classical model of enzyme activity, Michaelis–Menten kinetics, can be viewed as a Markov chain,
where at each time step the reaction proceeds in some direction. While Michaelis-Menten is fairly
straightforward, far more complicated reaction networks can also be modeled with Markov
chains.[67]

An algorithm based on a Markov chain was also used to focus the fragment-based growth of
chemicals in silico towards a desired class of compounds such as drugs or natural products.[68] As a
molecule is grown, a fragment is selected from the nascent molecule as the "current" state. It is not
aware of its past (that is, it is not aware of what is already bonded to it). It then transitions to the
next state when a fragment is attached to it. The transition probabilities are trained on databases of
authentic classes of compounds.[69]

Also, the growth (and composition) of copolymers may be modeled using Markov chains. Based on
the reactivity ratios of the monomers that make up the growing polymer chain, the chain's
composition may be calculated (for example, whether monomers tend to add in alternating fashion
or in long runs of the same monomer). Due to steric effects, second-order Markov effects may also
play a role in the growth of some polymer chains.

Similarly, it has been suggested that the crystallization and growth of some epitaxial superlattice
oxide materials can be accurately described by Markov chains.

Testing

Several theorists have proposed the idea of the Markov chain statistical test (MCST), a method of
conjoining Markov chains to form a "Markov blanket", arranging these chains in several recursive
layers ("wafering") and producing more efficient test sets—samples—as a replacement for
exhaustive testing. MCSTs also have uses in temporal state-based networks; Chilukuri et al.'s paper
entitled "Temporal Uncertainty Reasoning Networks for Evidence Fusion with Applications to Object
Detection and Tracking" (Science Direct) gives a background and case study for applying MCSTs to a
wider range of applications.

Solar irradiance variability

Solar irradiance variability assessments are useful for solar power applications. Solar irradiance
variability at any location over time is mainly a consequence of the deterministic variability of the
sun's path across the sky dome and the variability in cloudiness. The variability of accessible solar
irradiance on Earth's surface has been modeled using Markov chains, also including modeling the
two states of clear and cloudiness as a two-state Markov chain.

Speech recognition

Hidden Markov models are the basis for the most modern automatic speech recognition systems.

Information and Computer Science

Markov chains are used throughout information processing. Claude Shannon's famous 1948 paper A
Mathematical Theory of Communication, which in a single step created the field of information
theory, opens by introducing the concept of entropy through Markov modeling of the English
language. Such idealized models can capture many of the statistical regularities of systems. Even
without describing the full structure of the system perfectly, such signal models can make possible
very effective data compression through entropy encoding techniques such as arithmetic coding.
They also allow effective state estimation and pattern recognition. Markov chains also play an
important role in reinforcement learning.

Markov chains are also the basis for hidden Markov models, which are an important tool in such
diverse fields as telephone networks (which use the Viterbi algorithm for error correction), speech
recognition and bioinformatics (such as in rearrangements detection[77]).

The LZMA lossless data compression algorithm combines Markov chains with Lempel-Ziv
compression to achieve very high compression ratios.

Queuing theory

Markov chains are the basis for the analytical treatment of queues (queueing theory). Agner Krarup
Erlang initiated the subject in 1917.[78] This makes them critical for optimizing the performance of
telecommunications networks, where messages must often compete for limited resources (such as
bandwidth).[79]

Numerous queuing models use continuous-time Markov chains. For example, an M/M/1 queue is a
CTMC on the non-negative integers where upward transitions from i to i + 1 occur at rate λ according
to a Poisson process and describe job arrivals, while transitions from i to i – 1 (for i > 1) occur at rate
μ (job service times are exponentially distributed) and describe completed services (departures)
from the queue.

statistics

Markov chain methods have also become very important for generating sequences of random
numbers to accurately reflect very complicated desired probability distributions, via a process called
Markov chain Monte Carlo (MCMC). In recent years this has revolutionized the practicability of
Bayesian inference methods, allowing a wide range of posterior distributions to be simulated and
their parameters found numerically.
Economics and finance

Markov chains are used in finance and economics to model a variety of different phenomena,
including asset prices and market crashes. The first financial model to use a Markov chain was from
Prasad et al. in 1974.[84] Another was the regime-switching model of James D. Hamilton (1989), in
which a Markov chain is used to model switches between periods high and low GDP growth (or
alternatively, economic expansions and recessions).[85] A more recent example is the Markov
switching multifractal model of Laurent E. Calvet and Adlai J. Fisher, which builds upon the
convenience of earlier regime-switching models.[86][87] It uses an arbitrarily large Markov chain to
drive the level of volatility of asset returns.

Dynamic macroeconomics heavily uses Markov chains. An example is using Markov chains to
exogenously model prices of equity (stock) in a general equilibrium setting.[88]

Credit rating agencies produce annual tables of the transition probabilities for bonds of different
credit rating.

Social sciencs

Markov chains are generally used in describing path-dependent arguments, where current structural
configurations condition future outcomes. An example is the reformulation of the idea, originally
due to Karl Marx's Das Kapital, tying economic development to the rise of capitalism. In current
research, it is common to use a Markov chain to model how once a country reaches a specific level
of economic development, the configuration of structural factors, such as size of the middle class,
the ratio of urban to rural residence, the rate of political mobilization, etc., will generate a higher
probability of transitioning from authoritarian to democratic regime.

Mathematical biology

Markov chains also have many applications in biological modelling, particularly population
processes, which are useful in modelling processes that are (at least) analogous to biological
populations. The Leslie matrix, is one such example used to describe the population dynamics of
many species, though some of its entries are not probabilities (they may be greater than 1). Another
example is the modeling of cell shape in dividing sheets of epithelial cells.[91] Yet another example is
the state of ion channels in cell membranes.

Markov chains are also used in simulations of brain function, such as the simulation of the
mammalian neocortex.[92] Markov chains have also been used to model viral infection of single cells

Genetics

Markov chains have been used in population genetics in order to describe the change in gene
frequencies in small populations affected by genetic drift, for example in the diffusion equation
method described by Motoo Kimura.[94]

Games

Markov chains can be used to model many games of chance.[1] The children's games Snakes and
Ladders and "Hi Ho! Cherry-O", for example, are represented exactly by Markov chains. At each turn,
the player starts in a given state (on a given square) and from there has fixed odds of moving to
certain other states (squares)

Music

Markov chains are employed in algorithmic music composition, particularly in software such as
Csound, Max, and SuperCollider. In a first-order chain, the states of the system become note or pitch
values, and a probability vector for each note is constructed, completing a transition probability
matrix (see below). An algorithm is constructed to produce output note values based on the
transition matrix weightings, which could be MIDI note values, frequency (Hz), or any other desirable
metric.[95]

A second-order Markov chain can be introduced by considering the current state and also the
previous state, as indicated in the second table. Higher, nth-order chains tend to "group" particular
notes together, while 'breaking off' into other patterns and sequences occasionally. These higher-
order chains tend to generate results with a sense of phrasal structure, rather than the 'aimless
wandering' produced by a first-order system.[96] Markov chains can be used structurally, as in
Xenakis's Analogique A and B.[97] Markov chains are also used in systems which use a Markov
model to react interactively to music input.[98]

Usually musical systems need to enforce specific control constraints on the finite-length sequences
they generate, but control constraints are not compatible with Markov models, since they induce
long-range dependencies that violate the Markov hypothesis of limited memory. In order to
overcome this limitation, a new approach has been proposed.[99]

Baseball

Markov chain models have been used in advanced baseball analysis since 1960, although their use is
still rare. Each half-inning of a baseball game fits the Markov chain state when the number of
runners and outs are considered. During any at-bat, there are 24 possible combinations of number
of outs and position of the runners. Mark Pankin shows that Markov chain models can be used to
evaluate runs created for both individual players as well as a team.[100] He also discusses various
kinds of strategies and play conditions: how Markov chain models have been used to analyze
statistics for game situations such as bunting and base stealing and differences when playing on
grass vs. AstroTurf.[101]

Markov text generators

Markov processes can also be used to generate superficially real-looking text given a sample
document. Markov processes are used in a variety of recreational "parody generator" softwar

Bioinformatics

In the bioinformatics field, they can be used to simulate DNA sequences

Probabilistic forecasting

Markov chains have been used for forecasting in several areas: for example, price trends,[107] wind
power,[108] and solar irradiance.[109] The Markov chain forecasting models utilize a variety of
settings, from discretizing the time series,[108] to hidden Markov models combined with
wavelets,[107] and the Markov chain mixture distribution model (MCM