Universidad Nacional Abierta y a Distancia

Vicerrectoría Académica y de Investigación

Course: Decisions Theory
Code: 212066

Activity Guide and Evaluation Rubric – Task 3 Markovian decision processes

1. Activity Description

Type of activity: Collaborative

Evaluation moment: Intermediate Unit 3
Highest score of the activity: 110 points
The activity starts on: Sunday, October The activity ends on: Wednesday,
29, 2023 November 22, 2023
With this activity, you are expected to achieve the following learning
outcomes:
Learning Outcome 3: Interpret the characteristics of finite or infinite-stage Markov
processes for effective business decision making.
The activity consists of:

Below you will find 4 exercises, the first on matrix and transition diagram of a Markov
chain, the next two on Markov chains - Initial state multiplication, the fourth on
Markov chains – stady state, which you must develop and present in the discussion
forum of task 3, and submit collaboratively in the assessment environment. Finally,
you will find a complementary exercise corresponding to the internationalization of
the curriculum to be presented individually in the discussion forum of task 3.

In the Learning environment, consult the bibliographic reference of unit 3.

Optimization models with Markov Chains

• Adhwarjee, D. (2021). Theory and Applications of Operations Research (pp.182-188).

Laxmi Publications Pvt Ltd.

• Ibe, O. (2013). Markov Processes for Stochastic Modeling (pp. 59-65). Elsevier.
Exercise 1. Markov Chain and transition diagram

In a community there are 3 supermarkets (S1, S2, S3) there is mobility of a

customer from one to another. Each month S1 retains 90% of its customers and
loses 10% to S2. It was found that S2 retains only 5% and loses 85% to S2. 5% and

1
loses 85% which goes to S1 and the rest goes to S3, S3 retains only 40%, loses 50%
which goes to S1 and 10% goes to S2.

According to the data determine:

• The states of the chain.
• The transition matrix.
• The transition diagram.

Exercise 2. Markov Chain, transition diagram and initial state multiplication

Coffee consumers in the Andean region use three brands José Valdéz, Cascada and
Candelas. A survey was conducted in March 2022 in which 8450 coffee buyers were
interviewed. The results are presented in Table 1: Coffee consumers.

Table 1. Coffee consumers.

Brand José Brand Brand

Valdez Cascada Candelas
Brand José
338 845 507
Valdez
Brand
676 1928 776
Cascada
Brand
1690 745 945
Candelas

According to the information, determine:

• The transition matrix.
• The transition diagram.
• If purchases are made monthly, what will be the distribution of the coffee
market in the Andean region in the month of May?
• If purchases are made monthly, what will be the distribution of the coffee
market in the Andean region in the month of July?
• In September, what is the proportion of clients loyal to your coffee brands?

Exercise 3. Markov Chain and initial state multiplication.

A commercial agent performs his work in six cities A, B, C, D, E and F. To avoid

unnecessary travel, he stays all day in the same city and spends the night there,
moving to another city the next day if he does not have enough work. In the Table 2.
Transition probabilities, presents the probabilities that the agent is in a city and stays
working there or has to travel to another city.
2
Table 2. Transition probabilities

Cities A B C D E F
A 0,3 0,1 0,2 0,2 0,1
B 0,1 0,4 0,1 0,1 0,2
C 0,1 0,2 0,5 0,1 0,1
D 0,3 0,1 0,2 0,1 0,3
E 0,2 0,2 0,1 0,3 0,2
F 0,4 0,1 0,1 0,4

According to the information, determine:

• Complete transition matrix
• If today the traveler is at D, what is the probability that he will also have to
work at D in four days?
• If today the traveler is at D, what is the probability that he will also have to
work at A in seven days?
• If today the traveler is at D, what is the probability that he will also have to
work at C in nine days?

Exercise 4. Markov Chain – Stady state

One of the world's largest brewers (Guinness) has hired an operations research
analyst to analyze its market position. They are particularly concerned about their
major competitors (Heineken, Stella, Club, Reds). The analyst thinks that brand
switching can be modeled as a Markov chain including five states, the G, H, S, C, and
R states representing customers who drink beer produced by the aforementioned
breweries. The data is taken every month and the analyst has consolidated the
historical consumption data for each of the brands, which is shown in Table 3.
Historical data on beer consumption

Table 3. Historical data on beer consumption

Guinness Heineken Stella Club Reds

Guinness 890 695 507 840 1690
Heineken 680 873 798 335 398
Stella 1020 745 1180 530 983
Club 945 598 790 1120 764
Reds 845 1925 776 498 675

3
According to the data:
• Use the stable initial Markov method.
• Find the transition matrix with the participation of each event according to its
total.
• Find the values that satisfy the equations of state given the conditions of the
problem, via Excel Solver Add-in.
• Analyze the situations and interprets the problem like a chain Markov.

Exercise 5. Complementary exercise - internationalization of the curriculum.

Participation in academic events is aimed at transferring and updating knowledge on

a topic of interest, allowing the student to enrich his or her integral formation.
Therefore, as part of the internationalization component of the curriculum, and its
articulation with the courses, the student is required to actively participate in an
academic event in the area of decision sciences or basic sciences of UNAD in which
there is the presence of at least one international speaker. As a result of their
participation, they must submit in the discussion forum of task 3, a message in
English containing the following elements:

• Event information
• Indicate the topics of the event that most interested you.
• Describe how the topics of the event would contribute to your professional
training.
• Attach to the message a certificate or evidence of participation.
For the development of the activity consider that:

In the Initial Information Environment, you must:

• Constantly review the course agenda (verify the start and end dates of the
activity).
• Check the synchronous attention schedules via skype.
• Consult the classroom news forum of the course.
• Review the course presentation.
• Consult the Synchronous Meetings page via web conference.

In the Learning Environment, you must: Consult the reference proposed in the
required educational resources of unit 3. Optimization models with Markov Chains.
Submit in a timely manner in the discussion forum of the course, the contributions
and advances in the development and solution of the problems.

4
In the Evaluation Environment, you must: Deliver the group report according to its
specifications and availability scheduled in the course agenda.
Evidence of individual work:
The individual evidence to be submitted is:
Present in the Discussion forum - Task 3 (learning environment) in an individual and
timely manner, a spreadsheet document (excel) with the development of:

Exercise 1. Markov Chain and transition diagram

• The states of the chain.
• The transition matrix.
• The transition diagram.

Exercise 2. Markov Chain, transition diagram and initial state multiplication

• The transition matrix.
• The transition diagram.
• If purchases are made monthly, what will be the distribution of the coffee
market in the Andean region in the month of May?
• If purchases are made monthly, what will be the distribution of the coffee
market in the Andean region in the month of July?
• In September, what is the proportion of clients loyal to your coffee brands?

Exercise 3. Markov Chain and initial state multiplication

• Complete transition matrix
• If today the traveler is at D, what is the probability that he will also have to
work at D in four days?
• If today the traveler is at D, what is the probability that he will also have to
work at A in seven days?
• If today the traveler is at D, what is the probability that he will also have to
work at C in nine days?

Exercise 4. Markov Chain – Stady state

• Use the stable initial Markov method.
• Find the transition matrix with the participation of each event according to its
total.
• Find the values that satisfy the equations of state given the conditions of the
problem, via Excel Solver Add-in.
• Analyze the situations and interprets the problem like a chain Markov.

5
Exercise 5. Complementary exercise - internationalization of the curriculum.
You must submit in the discussion forum of task 3, a message in English containing
the following elements:
• Event information
• Indicate the topics of the event that most interested you.
• Describe how the topics of the event would contribute to your professional
training.
• Attach to the message a certificate or evidence of participation.
Evidence of collaborative work:
The collaborative evidence to be submitted is:
Collaboratively, the working group must select exercise 1, exercise 2, exercise 3, and
exercise 4 within the exercises presented individually to review the procedure and the
application of the models and algorithms. Construct a spreadsheet document (Excel)
consolidated by the members of the group that participated in the forum, entitled
Resolution of the Markovian decision processes, present it in the Discussion Forum -
Task 3 (learning environment) and deliver it in a collaborative in the evaluation
environment, in an which must contain the following information and presentation:

Cover: Identifying the activity, the title, the authors, the tutor, the course, the
institution, the school, the program and the year.

Exercise 1. Markov Chain and transition diagram

• The states of the chain.
• The transition matrix.
• The transition diagram.

Exercise 2. Markov Chain, transition diagram and initial state multiplication

• The transition matrix.
• The transition diagram.
• If purchases are made monthly, what will be the distribution of the coffee
market in the Andean region in the month of May?
• If purchases are made monthly, what will be the distribution of the coffee
market in the Andean region in the month of July?
• In September, what is the proportion of clients loyal to your coffee brands?

Exercise 3. Markov Chain and initial state multiplication

• Complete transition matrix
• If today the traveler is at D, what is the probability that he will also have to
work at D in four days?

6
 • If today the traveler is at D, what is the probability that he will also have to
work at A in seven days?
• If today the traveler is at D, what is the probability that he will also have to
work at C in nine days?

Exercise 4. Markov Chain – Stady state

• Use the stable initial Markov method.
• Find the transition matrix with the participation of each event according to its
total.
• Find the values that satisfy the equations of state given the conditions of the
problem, via Excel Solver Add-in.
• Analyze the situations and interprets the problem like a chain Markov.

Bibliography (with APA standard).

File Extension: .xls (excel)
Identification: Task 3_Number of group

2. General Guidelines for the Development of Evidences to Submit

For Collaborative evidences, consider the following:

• All members of the group must participate with their contributions in the
development of the activity.

• In each group a single member will be chosen to submit the requested product in
the environment indicated by the teacher.

• Before submitting the requested product, students should check that it meets all
the requirements mentioned in this activity guide.

• Only the members of the group that participated with contributions during the
time assigned for the activity should be included as authors of the submitted
product.

Please keep in mind that all individual or collaborative written products must comply
with the spelling rules and presentation conditions defined in this activity guide.
Regarding the use of references, consider that the product of this activity must
comply with APA style.

7
In any case, make sure you comply with the rules and avoid academic plagiarism.
You can review your written products using the Turnitin tool found in the virtual
campus.

Under the Academic Code of Conduct, the actions that infringe the academic order,
among others, are the following: paragraph e) Plagiarism is to present as your own
work all or part of a written report, task or document of invention carried out by
another person. It also implies the use of citations or lack of references, or it
includes citations where there is no match between these and the reference and
paragraph f) To reproduce, or copy for profit, educational resources or results of
research products, which have rights reserved for the University. (Agreement 029 -
December 13, 2013, article 99)

The academic penalties students will face are:

a) In case of academic fraud demonstrated in the academic work or evaluation, the
score obtained will be zero (0.0) without any disciplinary measures being derived.
b) In case of proven plagiarism in academic work of any nature, the score obtained
will be zero (0.0), without any disciplinary measures being derived.

3. Evaluation Rubric Template

Type of activity: Collaborative

Evaluation moment: Intermediate Unit 3
The highest score in this activity is 110 points
First evaluation High level: The student presents significant contributions in
criterion: the forum for the development of the proposed activities, in a
timely manner and contributes to the collective construction of
Use the discussion the final report.
forum, presenting If your work is at this level, you can get between 10
the development of points and 15 points
the exercises
proposed in the Average level: The student presents partial contributions in
guide the forum for the development of the proposed activities but
This criterion does not do so on time and/or does not contribute to the
represents 15 collective construction of the final report.
points of the total If your work is at this level, you can get between 2
of 110 points of points and 9 points
the activity.

8
 Low level: The student does not present contributions in the
forum for the development of the proposed activities, and/or
does so late.
If your work is at this level, you can get between 0
points and 1 point
Second evaluation High level: The student applies and analyzes the Markov chain
criterion: method and transition diagram to find the transition matrix to
Use the Markov chain the exercise 1 and interprets the results of the model for decision
method and making, correctly determining all the requested aspects.
transition diagram to If your work is at this level, you can get between 8 points
find the transition and 10 points
matrix to the
exercise 1, in order to Average level: The student applies and partially analyzes
make decisions in Markov chain method and transition diagram to find the
production, logistics transition matrix to the exercise 1, partially determining all the
and service requested aspects and interprets the results of the model for
systems. decision making.
This criterion If your work is at this level, you can get between 2 points
represents 10 and 7 points
points of the total
of 110 points of Low level: The student does not apply or analyze Markov chain
the activity. method and transition diagram to find the transition matrix to
the exercise 1.
If your work is at this level, you can get between 0 points
and 1 point.
Third evaluation High level: The student applies and analyzes the Markov
criterion: chain method and Initial state multiplication to find the
Use the Markov transition matrix and the state vector that gives the
chain method and distribution after a specified number of transitions to the
Initial state exercise 2, and interprets the results of the model for decision
multiplication to find making, correctly determining all the requested aspects.
the state vector that If your work is at this level, you can get between 10
gives the distribution points and 15 points
after a specified
number of Average level: The student applies and partially analyzes the
transitions to the Markov chain method and Initial state multiplication to find the
exercise 2, in order
transition matrix and the state vector that gives the
to make decisions in
distribution after a specified number of transitions to the
production, logistics exercise 2 and interprets the results of the model for decision
making, partially determining all the requested aspects.

9
and service If your work is at this level, you can get between 2
systems. points and 9 points
This criterion
represents 15 Low level: The student does not apply or analyze Markov
points of the total chain method and Initial state multiplication to find the state
of 110 points of vector that gives the distribution after a specified number of
the activity. transitions to the exercise 2.
If your work is at this level, you can get between 0
points and 1 point.
Fourth evaluation High level: The student applies and analyzes the Markov
criterion: chain method and Initial state multiplication to find the
Use the Markov transition matrix and the state vector that gives the
chain method and distribution after a specified number of transitions to the
Initial state exercise 3, and interprets the results of the model for decision
multiplication to find making, correctly determining all the requested aspects.
the state vector that If your work is at this level, you can get between 10
gives the distribution points and 15 points
after a specified
number of Average level: The student applies and partially analyzes the
transitions to the Markov chain method and Initial state multiplication to find the
exercise 3, in order
transition matrix and the state vector that gives the
to make decisions in
distribution after a specified number of transitions to the
production, logistics exercise 3 and interprets the results of the model for decision
and service making, partially determining all the requested aspects.
systems. If your work is at this level, you can get between 2
This criterion points and 9 points
represents 15
points of the total
Low level: The student does not apply or analyze Markov
of 110 points of
chain method and Initial state multiplication to find the state
the activity.
vector that gives the distribution after a specified number of
transitions to the exercise 3.
If your work is at this level, you can get between 0
points and 1 point.
Fifth evaluation High level: The student applies and analyzes the Markov
criterion: chain method and stady state to find the transition matrix and
the probabilities stationary to the exercise 4 and interprets the
Use the Markov results of the model for decision making, correctly determining
chain method and all the requested aspects.
stady state to find If your work is at this level, you can get between 18
the transition matrix points and 25 points

10
and the probabilities
stationary to the Average level: The student applies and partially the Markov
exercise 4, in order chain method and stady state to find the transition matrix and
to make decisions in the probabilities stationary to the exercise 4, partially
a production and determining all the requested aspects and interprets the results
service process. of the model for decision making.
If your work is at this level, you can get between 2
This criterion points and 17 points
represents 25 Low level: The student does not apply or analyze the
points of the total Markov chain method and stady state to find the transition
of 110 points of matrix and the probabilities stationary to the exercise 4.
the activity. If your work is at this level, you can get between 0
points and 1 point.
Sixth evaluation High level: The student participates in an international
criterion: academic event, presents a message in the forum of
Participation in an assignment 3 with the development of the items indicated and
international attaches a certificate or evidence of attendance.
academic event, as If your work is at this level, you can get between 16
part of the points and 20 points
internationalization
component of the Average level: The student participates in an international
curriculum, and its academic event, does not present message in the forum of
articulation with the assignment 3 with the development of the items indicated or
course. does not attaches a certificate or evidence of attendance.
This criterion If your work is at this level, you can get between 2
represents 20 points and 15 points
points of the total
of 110 points of Low level: The student does not participate in an
the activity. international academic event.
If your work is at this level, you can get between 0
points and 1 point.
Seventh
High level: The student contributes and presents in group the
evaluation
academic product with the approach, development, solution
criterion:
and interpretation of correct results of the Exercises of Task 3,
Group report on
with excellent writing, spelling, citations and bibliographic
decision methods
references according to the APA standard in its contents.
Markov chains.
If your work is at this level, you can get between 8
This criterion
points and 10 points
represents 10
points of the total

11
of 110 points of Average level: The student contributes and presents in group
the activity. the product academic with the approach, development, solution
and interpretation of the results of the Exercises of Task 3,
with regular writing, spelling, citations, and bibliographical
references according to the APA standard in its contents.
If your work is at this level, you can get between 2
points and 7 points

Low level: The student does not contribute and does not
submit the academic product of task 3 collaboratively.
If your work is at this level, you can get between 0
points and 1 point.

12