Markov Chain Example using Python

Markov chain is used to model a series of events. Each sequence is usually composed of various events and the order and the length of the sequence can vary drastically from sample to sample.
Such chain of events are usually very difficult to describe with deterministic statistics.
For instance, we can think of a monthly subscription service where a user can choose whether they want to subscribe or not each month.
Suppose we obtain a yearly subscription pattern of an user.

User 1
Month 1: Subscribed
Month 2: Subscribed
Month 3: Subscribed
Month 4: Unsubscribed
Month 5: Unsubscribed
Month 6: Unsubscribed
Month 7: Subscribed
Month 8: Unsubscribed
Month 9: Subscribed
Month 10: Subscribed
Month 11: Unsubscribed
Month 12: Unsubscribed

In this example, an user has two types of events: Subscribed and Unsubscribed.
Let's try to model such subscription behavior using Markov chain.

State of the Markov Chain Model

First, we should define the states. In this example, we can model a Markov chain consisting of two behaviors: Subscribed(S) and Unsubscribed(U).
Keep in mind that state should be defined as a function of time.
Hence, the state $X_n$ could be defined as the subscription behavior of an user at the begining of month $n$.
The state space $S$ would be $\{S, U\}$. Hence, the number of states is $n=2$. Having defined the states, we should check whether Markov chain is a valid approach for this problem. What
assumptions should we make?

Calculate the Transition Probabilities

Now let's use the above data to calculate the transition probabilities.
At each month, the user needs to make a decision of whether to move from one state to the other.
Let's transform the subscription sequence into transition pairs.
Month 1->2: S -> S
Month 2->3: S -> S
Month 3->4: S -> U
Month 4->5: U -> U
Month 5->6: U -> U
Month 6->7: U -> S
Month 7->8: S -> U
Month 8->9: U -> S
Month 9->10: S -> S
Month 10->11: S -> U
Month 11->12: U -> U

What can we observe from this table?

Out of 6 times the user started with a subscribed state, the user remained subscribed 3 times, and switched to unsubscribed state 3 times.
Hence, $P(S\rightarrow S) = 3/6 = 0.5$ and $P(S\rightarrow U) = 3/6 = 0.5$.
Similarly, $P(U\rightarrow S) = 2/5 = 0.4$ and $P(U\rightarrow U) = 3/5 = 0.6$.
Hence we can obtain the transition matrix as follows:
$P_{SS} = P(X_1=S | X_0 = S) = 0.5$
$P_{SU} = P(X_1=U | X_0 = S) = 0.5$
$P_{US} = P(X_1=S | X_0 = U) = 0.4$
$P_{UU} = P(X_1=U | X_0 = U) = 0.6$

Interpretation and Prediction

Now, we can use this Markov chain to make some predictions.
For a user who is currently subscribed, what is the probability of being subscribed in 1 month and 5 months?

In [45]: i m p o r t numpy a s np

#Number of states
n = 2

#Define the transition probability matrix

P = np. matrix(
[
[0.5, 0.5],[0.4, 0.6]
]
)

#Define initial probability distribution vector

s = np. matrix([1,0])

In [171… s * P

matrix([[0.5, 0.5]])
Out[171]:

In [172… s * (P* * 5)

matrix([[0.44445, 0.55555]])
Out[172]:

The probability of this user staying subscribed is 0.5 in 1 month and 0.4445 in 5 months.

Now, let's assume we obtain additional user subscription histories as follows.

 User 2
Month 1: Unsubscribed
Month 2: Subscribed
Month 3: Subscribed
Month 4: Subscribed
Month 5: Unsubscribed
Month 6: Unsubscribed
Month 7: Subscribed
Month 8: Subscribed
Month 9: Subscribed
Month 10: Subscribed
Month 11: Unsubscribed
Month 12: Unsubscribed

User 3
Month 1: Subscribed
Month 2: Subscribed
Month 3: Subscribed
Month 4: Subscribed
Month 5: Subscribed
Month 6: Unsubscribed
Month 7: Unsubscribed
Month 8: Unsubscribed
Month 9: Subscribed
Month 10: Subscribed
Month 11: Subscribed
Month 12: Subscribed

Now we can use all 3 users' subscription history data to recalculate the transition probabilities.
Even more, now we can try to compute the initial probability distribution as well.
Of course, all of these computations are based on the assumption that these 3 users' data are enough to obtain statistically signifciant results (which in reality is not, but let's assume for the sake of
this problem).

In [32]: s = np. matrix([2/ 3,1/ 3])

P = np. matrix(
[
[15/ 21, 6/ 21],[5/ 12, 7/ 12]
]
)

With the updated model, compute the probability of an user to be subscribed in 5, 10, 15, 20 months.

In [174… s * (P* * 5)

matrix([[0.59339184, 0.40660816]])
Out[174]:

In [175… s * (P* * 10)

matrix([[0.59322074, 0.40677926]])
Out[175]:

In [176… s * (P* * 15)

matrix([[0.59322034, 0.40677966]])
Out[176]:

In [177… s * (P* * 20)

matrix([[0.59322034, 0.40677966]])
Out[177]:

Do the probabilities seem to converge? Let's try to compute all of the probabilities up to the 20th month and visualize them as well.

In [27]: # Get the data

plot_data = []
f o r step i n range(20):
result = s * (P* * step)
plot_data. append(np. array(result). flatten()) #flatten(): Return a copy of the array collapsed into one dimension

#Convert the data format

plot_data = np. array(plot_data)

In [28]: i m p o r t matplotlib.pyplot a s plt

#Create the plot

f o r i i n range(n):
plt. plot(plot_data[:,i],'o',label = "S%i" % (i+ 1))

plt. legend()
plt. ylim(0,1)

(0.0, 1.0)
Out[28]:
 From the plots, we can see that the probabilities are converging.
Let's formally try to compute the steady-state probabilities.
Recall that we can compute stationary probabilities via: $\pi P = \pi$ and $\sum \pi_i = 1$.
$\pi P = \pi$ can be expressed as $(P^T-I)\pi = 0$ where $\pi$ is a column vector of the steady state probabilities.
Combining with $i \cdot \pi=1$ where $i$ is a unit row vetor, i.e., sum of the proabilities should be equal to 1, we obtain
$A\pi =b$ where $A = [(P^T-I);i]$ and $b$ is a vector of which all elements except the last is 0.
This equation can be solved using $A^T A\pi = A^T b$.
Below is the python code for solving this linear equation.

In [51]: i m p o r t pandas a s pd
f r o m random i m p o r t random

A = np. append(np. transpose(P)- np. identity(n),[[1,1]], axis= 0) #you should append a unit vector of size equal to n. If n=3, you should append [[1,1,1]] instead of [[1,1]]
#axis = the axis along which values are appended
b = np. transpose(np. array([[0,0,1]])) #you should input a vector with number of zeros equal to n. If n=3, you should input [0,0,0,1] instead of [0,0,1]
np. linalg. solve(np. transpose(A). dot(A), np. transpose(A). dot(b))

matrix([[0.44444444],
Out[51]:
[0.55555556]])

You can check that the steady state probabilites match with the converging numbers identified in the above figure.

Assignment
Formulate a problem setting where you can model the situation as a Markov chain.
You should keep in mind that the Markov property should hold true. You don't have to formally prove the Markov propery, but should be able to give reasons to why Markov property holds true
(qualitative reasoning would be enough).
You are free to find real world data or generate hypothetical data.
Dataset size should consist of at least 10 samples (in the above example, it consisted of 3 user history samples), and the sequence length of each sample should be at least 50.
State space should consist of at least 3 states (in the above example, state space consisted of 2 states $S = \{S, U\}$).
Using the data, you need to go through the same steps as the provided example:
Step 1. Explain the problem setting. Provide and explain the dataset.
Step 2. Define the state and check Markov chain assumptions.
Step 3. Calculate the transition probabilities.
Step 4. Predict the state in n-steps (ex. what is the probability to be in a certain state after 10 transitions?)
Step 5. Visualize the probability to be in each states upto 20th transition. Are the probabilities converging?
Step 6. If converging, check steady-state existence conditions and compute the steady state probabilites. If not converging, explain why.

In [ ]: