International Forum of Educational Technology & Society

How to Gamify Learning Systems? An Experience Report using the Design Sprint
Method and a Taxonomy for Gamification Elements in Education
Author(s): Armando M. Toda, Paula T. Palomino, Wilk Oliveira, Luiz Rodrigues, Ana C. T.
Klock, Isabela Gasparini, Alexandra I. Cristea and Seiji Isotani
Source: Journal of Educational Technology & Society , Vol. 22, No. 3 (July 2019), pp. 47-
60
Published by: International Forum of Educational Technology & Society

Stable URL: https://www.jstor.org/stable/10.2307/26896709

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

International Forum of Educational Technology & Society is collaborating with JSTOR to

digitize, preserve and extend access to Journal of Educational Technology & Society

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
Toda, A. M., Palomino, P. T., Oliveira, W., Rodrigues, L., Klock, A. C. T., Gasparini, I., Cristea, A. I., & Isotani, S. (2019). How
to Gamify Learning Systems? An Experience Report using the Design Sprint Method and a Taxonomy for Gamification Elements
in Education. Educational Technology & Society, 22 (3), 47–60.

How to Gamify Learning Systems? An Experience Report using the Design

Sprint Method and a Taxonomy for Gamification Elements in Education
Armando M. Toda1,4*, Paula T. Palomino1, Wilk Oliveira1, Luiz Rodrigues1, Ana C. T.
Klock2, Isabela Gasparini3, Alexandra I. Cristea4 and Seiji Isotani1
1
Institute of Mathematics and Computer Science, University of São Paulo, Brazil // 2Institute of Informatics, Federal
University of Rio Grande do Sul, Brazil // 3Department of Computer Science, Santa Catarina State University, Brazil
// 4Department of Computer Science, University of Durham, United Kingdom // [email protected] //
[email protected] // [email protected] // [email protected] // [email protected] //
[email protected] // [email protected] // [email protected]
*
Corresponding author

ABSTRACT: One of the main goals of gamification in educational settings is to increase student motivation and
engagement. To facilitate the design of gamified educational systems, in recent years, studies have proposed various
approaches (e.g., methodologies, frameworks and models). One of the main problems, however, is that most of these
approaches are theoretical, and do not provide a proof-of-concept. This paper advances the state of the art by
providing a practical way to help implement this kind of system. In this study, we present, for the first time, how one
can apply gamification elements in a learning system using the Design Sprint method, to guide designers and
developers on replicating this process. Additionally, as starting point, we use a taxonomy composed of 21 game
elements, proposed to be used within learning environments, organised into five game element categories, according
to their goal/usage. Our main contribution is to present how to systematically implement the gamification elements
focused on educational ends, which is of special value to practitioners, designers and developers.

Keywords: Gamification, Design Sprint, Taxonomy, Design, Education

1. Introduction
Gamified systems adoption has increased in the last decade, since the definition was coined (Thiebes, Lins, &
Basten, 2014). These systems aim at using game-like elements to provide a gameful experience to their users
(Landers, 2019; Thiebes et al., 2014). This caught the attention of education professionals, since the field of
education still struggles with motivating and engaging students (Borges, et al., 2014; Lee & Hammer, 2011; Martí-
Parreño, Seguí-Mas, & Seguí-Mas, 2016; Paula & Fávero, 2016; Sánchez-Mena & Martí-Parreño, 2016; Toda et al.,
2018a). In education, gamification consists of using game-like elements to achieve positive impacts in motivating,
engaging, persuading and improving the performance of students (Deterding, Sicart, Nacke, O’Hara, & Dixon, 2011;
Kapp, 2012; Seaborn & Fels, 2014). According to recent research, gamified systems impact on psychological
characteristics, and effective gamified systems lead to behavioural change (Landers, 2019). However, for a positive
impact, gamification needs to follow a well-thought design process; otherwise, it may lead to undesired behaviours,
or worsen performance, due to disengagement or other declining effects (De-Marcos, Domínguez, Saenz-de-
Navarrete, & Pagés, 2014; Dichev & Dicheva, 2017; Toda et al., 2018b; Zichermann & Cunningham, 2011).
Therefore, many authors proposed the use of frameworks and methodologies to support the gamification design
process (Mora, Riera, González, & Arnedo-Moreno, 2017).

However, these gamification frameworks and methods present some limitations, ranging from their purpose, to the
number of definitions of game elements used, which can confuse and drive away designers, developers, teachers, and
instructors who wish to gamify their learning activities (Koivisto & Hamari, 2019; Pedreira, García, Brisaboa, &
Piattini, 2015). Moreover, some frameworks are too generic and do not encompass learning objectives and other
properties derived from the education field, and others are too specific to a given niche (Mora et al., 2017); e.g., the
framework proposed by Kotini and Tzelepi (2015), focused on gamifying computational thinking activities. As for
the definitions, literature sees it as a considerable limitation on the field of gamification, since there are many
gamification frameworks (more than 40, up to date) and all of them use different types of game elements that may
not encompass all elements within a game. Additionally, recent studies report the lack of proof-of-concept in
gamification studies that may support the theories on which they are based (Kasurinen & Knutas, 2018).

ISSN 1436-4522 (online) and 1176-3647 (print). This article of the Journal of Educational Technology & Society is available under Creative Commons CC-BY-ND- 47
NC 3.0 license (https://creativecommons.org/licenses/by-nc-nd/3.0/). For further queries, please contact Journal Editors at [email protected].

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
To address this, this work presents and applies a recent taxonomy, created and evaluated to mitigate the issues
related to the game elements definitions (Toda et al., 2019a) and the lack of proof-of-concept for gamification
frameworks. The taxonomy is composed of 21 game elements to be used within learning environments, which were
grouped into five game element categories, according to their goal/usage (Toda et al., 2019b). Next, we present how
we can apply that taxonomy using the Design Sprint Method (Sumual, Batmetan, & Kambey, 2019), to guide
designers and developers on replicating this process. Hence, we propose the following research question: How can
we gamify learning environments using the Design Sprint method and existing game elements? Thus, our main
contribution is presenting how to systematically implement a recent, expert-validated gamification elements
taxonomy, that is focused on educational ends, which is of special value to practitioners (e.g., designers, developers,
teachers, and professors), who aim to use this taxonomy to implement gamification in their learning environments.

The rest of the paper is organised as follows: the next section introduces the research background, by reviewing
relevant gamification frameworks in education. The research model used in this research is then described, followed
by the description of the application of the taxonomy. Finally, the implications of the findings are discussed, and
conclusions are drawn.

1.1. Background and related works

Gamification in education is not a novelty, as many studies have focused on applying game elements in learning
environments, even before the concept was coined (Darejeh & Salim, 2016; Dichev & Dicheva, 2017; Vargas-
Enriquez, Garcia-Mundo, Genero, & Piattini, 2015). The purpose of using gamification in education is to motivate
and engage students, to improve their performance and training, and change undesired behaviours (Huotari &
Hamari, 2012; Kapp, 2012; Seaborn & Fels, 2014). However, the literature in this field reported mixed results on the
application of gamification, wherein most of the negative impacts were related to a poor design (Dichev & Dicheva,
2017; Toda et al., 2018b).

The factors that influence a poor design ranged from users’ demographic to behavioural profiles, context and
learning activities, and the way the gamified strategies were designed, or recommended (Klock, Gasparini, &
Pimenta, 2016; Toda et al., 2019c). Considering the latter, most of the existing frameworks were either conceptual or
lacked proper definitions of game elements. Moreover, recent conceptual frameworks lacked empirical evidence on
their use, which hindered their adoption by teachers and instructors (Pedreira et al., 2015; Sánchez-Mena & Martí-
Parreño, 2016). Furthermore, the lack of proper definitions may confuse designers and other education domain
specialists, since most frameworks used not only different names for the same concept, but also the same definition
for different concepts; e.g., in Gamify-SN (Toda et al., 2018a) the authors define “acknowledgements” as a type of
feedback given to the users when certain actions are performed, while in another framework (Wongso, Rosmansyah,
& Bandung, 2014) the authors define the same element as “medals” or “badges.”

Furthermore, considering frameworks in the field of education, a recent systematic review (Mora et al., 2017) found
6 frameworks. From this group, one is focused on serious games and five others on gamification.

Simões, Redondo and Vilas (2013) presented a framework for educational platforms divided into three groups. The
first group described game elements (N = 12) divided into game mechanics and dynamics. Following, the second
group presented guidelines for teachers, focusing on learning tasks, however, without linking these tasks to the game
elements. A third group connected focused these guidelines, aligning the objectives with the school identity. These
objectives aimed to help students overcome failure, achieve the flow state (Csikszentmihalyi, 1975), experience new
roles and enhance their skills. However, the framework did not present empirical evidence concerning its application
to learning environments.

Following, Wongso et al. (2014) proposed a framework for educational domains focused on linking gamification and
Web 2.0 social features with five steps: Analysis, Design, Development, Implementation and Evaluation. Game
elements and social features are defined in the Analysis step. The authors considered game elements as game
mechanics (N = 7), further linked to tasks developed in the Design phase. Nonetheless, the authors did not present an
empirical validation.

48

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
Kotini and Tzelepi (2015) designed a gamification framework focused on gamifying computational thinking courses.
The framework divided the game elements (N = 18) into three categories: Behaviour, Progression and Feedback.
These groups were tied to computational thinking skills, behaviours and definitions. Nonetheless, the framework did
also lack an empirical validation (e.g., any instance or proof-of-concept on its usage) and the elements were heavily
tied to the concepts of computational thinking, which may have hindered adoption by other fields.

Concerned with software development, Mora, Zaharias, González, and Arnedo-Moreno (2016) also proposed a
framework for education, called FRAGGLE. This framework used an Agile method from software engineering and
gamification features, aligned with learning objectives, to gamify learning systems. The framework was focused on
aiding developers and designers and consisted of 4 main steps: Declaration, Creation, Execution and Learning. They
considered the use of player profiles to select game mechanics. Again, no empirical evidence was provided, nor a
description on the game elements that can be used.

Finally, the most recent framework for gamified education was designed by Ana et al. (2016) where they developed
a user-centred gamification framework for the educational field. This framework was organised into 7 steps: Who?
What? Why? When? How? Where? How Much? The framework was applied and evaluated with 139 students
enrolled in an online course, providing empirical evidence on its use and efficiency on motivation, performance and
engagement. This is the only framework to have empirical evidence on its use. However, the framework presented
little on the use of game elements and it was focused on learning systems, while ours can be used with unplugged
gamification (i.e., the use of gamification without a computer or digital tool).

Considering the related works, we can observe that none of them presented any kind of validation to the game
elements that were used nor provided usage information of these elements, e.g., how these elements can be applied
within the context of the framework. Only one work presented empirical evidence on its use and another provided
partial evidence (e.g., presented how the system worked). As for the definitions and number of elements included,
most of the frameworks focused only on elements that acted as a kind of feedback (e.g., points, levels and badges)
not considering contextualising elements, such as Narrative and Storytelling. To the best of our knowledge, our study
is the first study to use a validated set of game elements to gamify learning systems. Table 1 presents a comparison
between ours and related works. Some of the studies appear as having provided partial empirical evidences, meaning
lack of methods to measure what was intended or focused upon. Also, none of the frameworks presented an explicit
way on how to use the game elements.

Table 1. Related works comparison

Work # of Game Provides empirical Provides validation of Provides a how-to-use
Elements evidence? the game elements? the game elements?
Simões et al. 12 Partially, used in a No No, presented the
(2012) digital learning system with the game
environment. elements.
Wongso et al. 7 No No No
(2014)
Kotini and Tzelepi 18 No No No
(2015)
Mora et al. (2015) Not explicit No No No
Klock et al. (2019) 17 Yes, used in a learning No No, presented the
environment. system with the game
elements.
Our study 21 Yes Yes, a validation Yes
based on experts’
opinion.

2. Research method
Our study uses the Design Sprint method, developed by GV (Google Ventures). Its focus is to answer critical
business questions through design, prototyping and testing ideas (as a proof-of-concept, i.e., the practical model that
can prove the theoretical concept established by research). It has been used to design new products, develop new
49

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
features, and define marketing strategies, with a good cost-benefit rating. Comparing it with other agile methods,
such as Scrum’s sprints, the Design Sprint is focused on learning about an idea, without having to build and launch
it, as shown in Figure 1. As such, one of the main advantages of this method is the possibility to shortcut debate
cycles and compress months of time into a single week (Sumual et al., 2019).

Figure 1. Design Sprint shortcut to learning, without building and launching (Knapp et al., 2019)

In gamification context, as stated in the previous sections of this paper, one of the main problems of its effectiveness
is the lack of guidelines and coherent methods to create the strategies and/or applications. Therefore, we chose to
work with a systematic and established method for creating and validating new ideas and products, in their
conceptual stages, to evaluate the empirical application of the gamification taxonomy for educational purposes and
allowing for the proof-of-concept for future building and launching of a digital product based on it. The first step to
use the Design Sprint is to set the stage, establishing the right challenge and the right team to deal with it. After that,
the sprint is split into five steps (ideally one for each weekday).

1. Understanding and discussions: The first day of structured discussions should organise the subsequent steps
for the rest of the week. Amongst the tasks included are: establishing a long-term goal and mapping the
challenge, picking a target to work (Knapp et al., 2019);
2. Focus on solutions: On the second day of the sprint, brainstorming is performed, reviewing existing ideas, in
order to remix and improve them. Then, we progress to the sketch phase, emphasising critical thinking (Knapp
et al., 2019);
3. Decision: here, the team chooses one solution to work, test, and validate, creating a step-by-step plan for the
prototype (Knapp et al., 2019);
4. Prototype: In the fourth day/step, the team creates the prototype, focusing on testing with customers (end-
users). Here, all planning is reviewed and organised for the final step of the sprint (Knapp et al., 2019);
5. Test and validate: This includes testing the prototype, interviewing customers and/or learning by watching
them reacting to the prototype. As a result, the team knows whether an idea is feasible or not, ending the sprint
(Knapp et al., 2019).

For our research, we used the Design Sprint method to validate the use of our taxonomy in the creation, prototyping
and testing of an educational gamified application. At this stage of the research, we were not concerned with the
application interface and, therefore, the user prototype. Instead, our focus was on the design of gamification

50

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
strategies (i.e., how can the 21 game elements be used to improve learning experience in a gamified educational
system).

The taxonomy used (Toda et al., 2019a) is composed of 21 gamification elements for the education field (Figure 2).
These game elements were collected from the literature and focused on creating a syllabus for gamification in
education. The authors defined a concept, alongside its synonyms, and a definition for each of the 21 elements. This
taxonomy was validated by 19 experts on the field of gamification and education (most of the experts were also
lecturers and professors), achieving an overall acceptance of its elements, concepts and definitions. Thus, in
summary, we chose this approach as it is an expert-validated, state-of-the-art alternative, specifically developed for
educational environments that suits our need.

Figure 2. Taxonomy design (Toda et al., 2019b)

According to Toda et al. (2019b), those elements have been hierarchically linked by classification into five
dimensions, related to performance/measurement, environment, social/personal interaction and student experience.
Importantly, all the 21 elements contain synonyms of alternate names found in the literature (Table 2), for domain
specialists to be able to use this taxonomy’s recommendations alongside other frameworks. In the next section, we
describe how this method was used in our context and the results arising from it.

51

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
 Table 2. Gamification elements and definitions (Toda et al., 2019a)
Concept Description Dimension
Acknowledgement Type of feedback that praises the players’ specific actions. Some examples and Performance
synonyms are badges, medals, trophies.
Chance Randomness and probability properties that increase or decrease the odds of Ecological
certain events; examples/synonyms: randomness, luck, fortune.
Competition When two or more players compete against each other towards a common Social
goal; examples/synonyms: Player vs Player, scoreboards, conflict.
Cooperation When two or more players collaborate to achieve a common goal; Social
examples/synonyms: teamwork, co-op missions.
Economy Transactions within the game, monetising game values and other elements; Ecological
examples/synonyms: markets, transaction, exchange.
Imposed Choice Decisions that the player is obliged to make in order to advance the game; Ecological
examples/synonyms: judgements, forced choices (different from Narrative).
Level Hierarchical game layers, providing a gradual way for players to obtain new Performance
advantages upon advancing; examples/synonyms: character levels, skill
level.
Narrative Order of events happening in a game; i.e., choices influenced by player Fiction
actions; examples/synonyms: strategies the player uses to go through a level
(stealth or action), also the good/bad actions influencing the ending, karma
system (different from Imposed Choice).
Novelty New, updated information presented to the player continuously; Personal
examples/synonyms: changes, surprises, updates.
Objectives Guide the players’ actions. Quantifiable or spatial, from short- to long-term; Personal
examples/synonyms are missions, quests, milestones.
Point Unit used to measure users’ performance; examples/synonyms: scores, number Performance
of kills, experience points.
Progression This allows players to locate themselves (and their progress) within a game; Performance
examples/synonyms: progress bars, maps, steps.
Puzzles Challenges within the game that should make a player think Personal
examples/synonyms: actual puzzles, cognitive tasks, mysteries.
Rarity Limited resources and collectables; examples/synonyms: limited items, rarity, Ecological
collection.
Renovation When players can redo/restart an action; examples/synonyms are extra life, Personal
boosts, renewal.
Reputation Player titles to accumulate in-game; examples/synonyms: titles, status, Social
classification.
Sensation Use of players’ senses to create new experiences; examples/synonyms: visual Personal
stimulation, sound stimulation.
Social Pressure Pressure through social interactions with another player (s) (playable and non- Social
playable); examples/synonyms: peer pressure, guilds.
Stats Visible information for the player, about their in-game outcomes; Personal
examples/synonyms: results, health bar, magic bar, HUD, indicators, data
from the game presented to the user.
Storytelling The way the story of the game is told (as a script) within the game, via text, Fiction
voice, or sensorial resources; examples/synonyms: stories told through
animated scenes, audio queues or in-game text queues.
Time Pressure Pressure through time in-game; examples/synonyms: countdowns, clock, Ecological
timer.

3. Application
In this section, we aim to describe how we used the Design Sprint method to propose the gamification design
instantiating the taxonomy proposed by Toda et al. (2019a). Our main idea was to use the Design Sprint method in
order to think, propose, prototype and evaluate the gamification design. The team is composed of five experts (each
52

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
with more than five years of experience) in Education, Computer Science, Gamification Design, and Human-
computer Interaction (HCI). This number is also a recommendation by Nielsen and Landauer (1993).

On the First Day (understanding and discussions), the team members set the long-term goal, mapped project
challenges, and set targets for the project execution. As a result, it was decided to propose a gamification design,
capable of being used in the implementation of different gamified systems and implementing the gamification
elements proposed in the taxonomy. In the challenge mapping stage, four challenges were defined to guide the
project management:

1. Definition of a general gamification design architecture (Day 1 and 2): At this stage, the main objective was
to define a general architecture of a gamified educational system and defining how the elements could be
organised (i.e., on which pages each element should appear); this was done through a brainstorming session.
2. Implementation of the elements according to the taxonomy (Day 3): At this state, the main objective was to
define how gamification elements should be organised and the internal relationships between elements and
activities in the system (e.g., when finishing an activity, which gamification elements should be changed as a
reward to the user); to achieve this step, we mapped the activities and events on the system and matched each
with the elements based on recommendations presented in previous works (Toda et al., 2018a; Toda et al.,
2019c).
3. Gamification design proposal (Day 3 and 4): At this stage, the main objective was to write the documentation
of the gamification design, condensing the results from the previous steps. In other words, this meant
formalising the gamified strategies (A gamified strategy in the scope of this work is an event that links a task
and a given gamification element, e.g., Perform a Login (Task) and receive a badge (Gamification element:
Acknowledgement)).
4. Gamification design instantiation (Day 5): In this step, the main objective was to apply the design in a
learning system.

On the Second Day (focus on solutions), team members reviewed what was defined in the previous day. Through
meetings and brainstorming sessions, the team re-analysed what was proposed and made any changes that could
impact onto the final system. Then, an outline of the proposal was defined, seeking to relate each gamification
element and discussing how the elements could be implemented. These annotations and definitions were made using
Trello (see https://www.trello.com), a system design to manage team projects.

Figure 3. Gamification design flowchart

53

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
On the Third Day (decision), team members detailed how each gamification element should be implemented in
educational systems, and how these elements relate to each other. At the end of the day, the team finished the
gamification design, seeking to define how the elements proposed in the taxonomy could be grouped, organised and
implemented within an educational system, through the gamified strategies (Figure 3). An example was to design the
Home page. In this page, the students would have visual access to certain elements, as the weekly leaderboard
(Competition and Time Pressure), Cooperation (Their groups), their progress within the system (Point, Level,
Progression, Acknowledgement and Reputation) as well as the missions that were assigned to them (Objectives).
These elements were combined and proposed based on the recommendations of the Taxonomy and other studies that
validated those combinations with students (Toda et al., 2019d).

On the Fourth Day (prototype), the gamification design was discussed among all team members and all team
members agreed with the proposal. Next, through another brainstorming session, the team defined the functions that
would be present in the prototype. On the Fifth Day (test and validate), we analysed the prototype and tested the
gamification elements interactions through decision tables testing, which is a common software development
technique for defining software restrictions and events. It is a systematic method to test input combinations and their
output (Jorgensen, 2013). An example of the Decision table testing can be seen in Table 3, where each “Event” row
is an action that can be performed in the prototype and the following columns are the gamification elements that are
affected by that action.

Table 3. Table testing example

Event Point Acknowledgement Competition Time Pressure Social Pressure Imposed Choice
A1: Student got Yes Partially Yes No Partially No
a question
right
A2: Student No No No No No Yes
chose their
avatar
A3: Student Yes Yes Yes Partially Yes No
achieved a
new rank on
the leader
board
A4: Student got Yes Yes Yes No Partially No
5 questions
right in a
sequence

Figure 4. Interaction diagram to exemplify the decision table test

54

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
An example on how to understand Table 3 is the first row, where A1 is the event “Student got a question right,”
triggered when the student is answering a question related to a certain content. When students answer correctly, they
gain a Point, which can be summed towards an Achievement (Acknowledgement), and towards the Leader Board
(Competition). The update on the Leader Board can influence the Social Pressure in the system, since the student
can increase their rank or decrease it, affecting other students. Figure 4 demonstrates an interaction diagram of the
events A1 and A4, to exemplify the approach.

The system consists of three main pages: Home, where students can track their progress and evolution in the system;
Learn, where students will have educational activities; and Profile, where students will have access to all their
information. The Home page has a sub-page called Store, where students can buy special items. The Learn page has
three task settings (missions), where students will have lessons. The Profile page has a sub-page called Friends,
where students can view other members and follow them. These pages were designed using scenarios and evaluated
using the persona technique, which is based on creating goal-directed, role-based and fictional users that will interact
with the system (De Borba, Gasparini, & Lichtnow, 2017; Nielsen, 2013; Preece, Rogers, & Sharp, 2015). For
example, Gareth, the undergraduate student and avid competitor, who tends to get questions wrong, and may not
notice functionality of the system due to his impulsivity. Or Cynthia who likes to buy clothes in real-life, and is
attracted by the store icon in the system, etc. This technique is used for testing prototypes and can aid designers to
visualise the behaviours within the system. This technique has also been used to evaluate other gamified educational
systems, as seen in Palomino et al. (2019a).

An example for the gamification design is the implementation of the Point element is as follows: “The Point element
will be displayed on all pages (in the fixed header) and will be represented by experience points (XP). The student
will earn seven (or a specific number according to the system specificities) points for each activity done and two
extra points if the activity is done correctly (hit the answer). Points will be updated each time the student completes
an activity group. The total points will also be highlighted on the Profile page.” Table 4 synthesises the proposed
gamification design.

Table 4. Proposed gamification design

Concept Design description
Acknowledgement The Acknowledgment element should be displayed on all pages and will be represented
through the achievement feedbacks. Thus, the user will receive immediate notifications of all
achievements in the system.
Chance The Chance element should appear on the Learn page and will consist of a random option
offered to the user to increase their prize.
Competition The Competition Element should be featured on the homepage and represented by weekly
leaderboards with up to 10 students.
Cooperation The Cooperation element should be featured on the Homepage and represented by the
formation of random teams of up to 5 students.
Economy The Economy element should appear on every page and represented by coins that can be used
to make in-game purchases.
Imposed Choice The Imposed Choice element should be displayed on the Profile page. When viewing their
profile for the first time, users should be able to choose an avatar to represent them in the
system. This avatar will evolve when using the system. In addition, the student should have
to make different types of choices during use (e.g., choosing between chests with coins).
Level The level element should be displayed on the Homepage, represented by phase (bronze, silver,
gold, and diamond, respectively).
Narrative The narrative element should be presented on the Learn page, represented by the user's ability
to do extra activities. At times, when the user completes a quest, they may choose to visit
other system pages or immediately do a new quest, for earning extra coins.
Novelty The Novelty element should appear on the Learn page and the Store page. On the Learn page, it
should be represented by hints that will be appear when the user misses a sequence of three
questions in a row. On the Store page, it should be represented by selling special objects.
Objectives The Objectives element should be displayed on the Homepage and should be represented by a
quest tree. This tree can show the entire sequence of missions the student has in the system.
Point The Point element should be displayed on all pages (in the fixed header) and should be
represented by experience points (XP). The student should earn points for each activity done
55

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
 and extra points if the activity is done correctly (hit the answer).
Progression The Progression element should be displayed on the Home and Learn pages and should be
represented as a progress bar. The Homepage should be represented by a circular progress
bar in the activity tree, indicating how much of each activity group has been completed and
how much remains to be completed. In the Learn page it should be represented by a progress
bar, showing how much has been completed and how much is left to complete each activity
group.
Puzzles The Puzzles element should be presented on the Learn page and should be represented by
proposing (non-mandatory) surprise challenges, related to the subject being studied.
Rarity The Rarity element should appear on the Store page and should be represented as a series of
shields. Rare items should be available in the system store with purchase values that can be
purchased through coins.
Renovation The Renovation element should be displayed on the Learn page and should be represented
through the possibility for students to perform activities in which they err. There should be
no cost for students to redo the activities they have missed.
Reputation The Reputation element should be displayed on the Homepage and should be represented by
the student's title/patent display. Different hierarchical levels can be achieved through special
sequences in the system.
Sensation The Sensation element should be displayed on the Learn page and should be represented by
immediate feedback (visual and audible) regarding each action/response from users,
indicating whether they have hit or missed each question.
Social Pressure The Social Pressure element should be displayed on the Profile page and should be represented
by an alert message, whenever the user drops in the ranking (is exceeded by a colleague).
Stats The Stats element should be present on the Profile page and should be represented by all user
progress information, which by default will already be displayed on the Profile page.
Storytelling The Storytelling element should be present on the Profile page, represented by the evolution of
the avatar (and its story) of a student (thus associated with the Imposed Choice element).
Time Pressure The Time Pressure element should be present on the Home page and should be represented by a
weekly countdown (thus associated with the Competition element).

4. Discussions and limitations

Before starting the discussions related to our study, it is important to highlight that our study generated some
limitations inherent to this type of study. Initially, because it is a critical and creative process, it is not possible to
systematise all design decisions (e.g., document all discussions). To mitigate this limitation, we used a process
known in the literature and used in similar studies (Design Sprint method). In the last step (Fifth day), it was not
possible to perform an evaluation with users; however, we used the persona technique, which is a valid HCI
technique.

Additionally, there is the limitation of implementing content elements from the Taxonomy by Toda et al. (2019a). By
content elements we especially refer to Storytelling and Narrative (Kapp, 2012; Palomino et al., 2019b) that,
although already mentioned in the literature, e.g., Marczewski’s Periodic Table of Gamification Elements (Tondello
et al., 2016) and Klock’s gamification conceptual model (Klock et al., 2019), lack systematically validated
procedures (e.g., frameworks or processes) guiding designers on how to implement them. For instance, Armstrong
and Landers (2017) investigated the impact of transforming regular texts into scripted texts, thereby making users
interact with texts telling a story, which fits in the Storytelling game element (Toda et al., 2019b). Another example
is Champagnat, Delmas and Augeraud (2010) research which dealt directly with the Storytelling concept applied to
learning. They presented a variation of Campbell’s Hero Journey (Campbell, 2008), specifically, for interactive
storytelling, and detailed how this model could be used in an educational context.

In these cases, authors often rely on some specific or self-developed framework/process for implementing those
elements. Whereas there exist options for developing stories, which might be used for Storytelling (e.g., that used by
Landers et al. (2017)), the literature still lacks a systematic process for adding the Narrative game element to GES,
although research towards this direction has recently emerged (e.g., Marczewski, 2015; Palomino et al., 2019c).
Thus, future studies maturing the field in terms of how to implement content game elements would benefit designers
56

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
and, as using these elements along with other common game elements (e.g., Cooperation, Objectives and Puzzles) is
of users’ interest (Palomino et al., 2019b), their experiences would be benefited as well. Therefore, we call for
further research on this topic.

Furthermore, defining which set of elements to use together was another challenging process. This happens due the
lack of studies that provide clear guidelines and justifications on the combination of game elements, which has been
pointed as an important aspect in the gamification design (Dichev & Dicheva, 2017; Toda et al., 2018b). On one
hand, each element from the taxonomy used in this work has a specific goal and, therefore, is likely to be used in
different occasions. On the other hand, there are some elements that have similar goals, as can be seen by their
grouping shown in Figure 2. However, selecting which game elements to use together, by simply following their
grouping, might not be the best option as, for instance, one might be seeking to create a gamification design (game
elements set) that involves showing users’ performance (one group) based on their social interactions (another
group). To define those sets, there are two high-level approaches that have been explored: theory- and data-driven
insights.

While theory-driven approaches explore theories such as the self-determination theory (Deci & Ryan, 1985) to
define which game elements to use, data-driven ones rely on, for instance, usage data to select the gamification
design (Meder, Plumbaum, & Albayrak, 2017). On the other hand, the data-driven approaches have recently
emerged, and scholars have defended their benefits over theory-driven ones, in the context of gamification (Meder et
al., 2017) . Given this context, studies on how to define gamification designs based on data have started to appear
(Toda et al., 2019c). Nevertheless, as this is a recent field study, it is yet to mature and further research is required to
both improve the understanding on how to create those data-driven designs, as well as to identify whether those are
more effective than theory-driven designs in affecting users’ behaviour – or perhaps combined approaches are
required.

Another recent, relevant issue of gamification designs that was not addressed by the design we presented in this
study is personalisation. That is, providing gamification design tailored to different user types aiming at improving
their experiences (Oliveira & Bittencourt, 2019). As gamified systems are a specific type of information systems, the
personalisation dimension is an important aspect to be tackled (Klock, Ogawa, Gasparini, & Pimenta, 2018; Liu &
Stacey, 2015). Personalisation emerges as an approach to accommodate different users within the same gamified
systems (Seaborn & Fels, 2014), which is a necessary step, as users have different behaviours, interpretations,
preferences, and experiences (Lavoué, Monterrat, Desmarais, & George, 2019; Orji, Tondello, & Nacke, 2018),
thereby, the same gamification design is unlikely to work for all of them.

Thus, we highlight two closely related research veins that should be tackled. Future studies should further investigate
whether the use of personalisation approaches can improve gamification’s effectiveness, compared to generic design.
The other is that personalisation approaches focusing not only on the users, but also on the task they are performing,
should be performed, to provide gamified design aiming to satisfy users’ preferences, as well as the task at hand.
Consequently, creating guidelines on how to deploy it, which will then support practitioners deciding on how and
whether to personalise the gamification designs of their systems.

Regarding the use of the Design Sprint method, it was noted that this method allowed team members to propose a
solution rapidly and through a critical-creative approach, where team members were able to share opinions on each
step of the solution proposal, and at the same time, criticise colleagues’ proposals and self-criticise their propositions.
Thus, it is possible to conduct further studies using this method and perform evaluations that can measure the
effectiveness of the method in the gamification design process.

5. Conclusions and future work

This work presented a method on how we can use gamification elements to gamify learning environments. We
compared our taxonomy with other works concerned with gamifying learning activities that were found through an
existing systematic mapping. We also used an agile process alongside the given taxonomy. Through this work, we
present a new way on how to gamify learning systems using methods different from other frameworks. We also
believe that this taxonomy can be used within most existing frameworks in the education field, since its definitions
cover most of the elements that exist in previous frameworks.
57

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
For future work, we are focusing on designing an experiment to research if this taxonomy can be used alongside
data-driven gamified recommendations based on the elements that compose it. We intend to conduct a deeper
analysis on the scenarios provided in Table 3, by using other types of evaluation besides the persona technique (e.g.,
students’ evaluations of the system). We also intend to design a gamified educational system based on the
gamification design proposed in this article and to conduct a longitudinal study assessing the students’ experience in
that system. We are especially interested to investigate the gamification influence in the students’ flow experience
and learning outcomes.

Acknowledgement
The authors would like to thank the funding provided by FAPESP (Projects 2016/02765-2; 2018/11180-3;
2018/15917-0; 2018/07688-1), FAPESC (public call FAPESC/CNPq No. 06/2016 support the infrastructure of CTI
for young researchers, project T.O. No.: 2017TR1755 - Ambientes Inteligentes Educacionais com Integração de
Técnicas de Learning Analytics e de Gamificação) and CNPq. This study was financed in part by the Coordenação
de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) Finance Code 001.

References
Armstrong, M. B., & Landers, R. N. (2017). An Evaluation of gamified training: Using narrative to improve reactions and
learning. Simulation & Gaming, 48(4), 513–538. doi:10.1177/1046878117703749
Borges, S. de S., Durelli, V. H. S., Reis, H. M., & Isotani, S. (2014). A Systematic mapping on gamification applied to education.
In Proceedings of the 29th Annual ACM Symposium on Applied Computing - SAC ’14 (pp. 216–222).
doi:10.1145/2554850.2554956
Campbell, J. (2008). The Hero with a thousand faces. Novato, CA: New World Library.
Champagnat, R., Delmas, G., & Augeraud, M. (2010). A Storytelling model for educational games: Hero’s interactive journey.
International Journal of Technology Enhanced Learning, 2(1–2), 4–20. doi:10.1504/IJTEL.2010.031257
Csikszentmihalyi, M. (1975). Flow: The Psychology of optimal experience. Harper & Row. Retrieved from
http://www.amazon.com/Flow-The-Psychology-Optimal-Experience/dp/0061339202
Darejeh, A., & Salim, S. S. (2016). Gamification solutions to enhance software user engagement – A Systematic review.
International Journal of Human-Computer Interaction, 32(8), 613-642. doi:10.1080/10447318.2016.1183330
De-Marcos, L., Domínguez, A., Saenz-de-Navarrete, J., & Pagés, C. (2014). An Empirical study comparing gamification and
social networking on e-learning. Computers & Education, 75, 82–91. doi:10.1016/j.compedu.2014.01.012
De Borba, E. J., Gasparini, I., & Lichtnow, D. (2017). The Use of time dimension in recommender systems for learning. In
Proceedings of the 19th International Conference on Enterprise Information Systems (ICEIS 2017) (vol. 2, pp. 600–609).
doi:10.5220/0006312606000609
Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. Boston, MA: Springer US.
doi:10.1007/978-1-4899-2271-7
Deterding, S., Sicart, M., Nacke, L., O’Hara, K., & Dixon, D. (2011). From game design elements to gamefulness: Defining
“Gamification.” In Proceedings of the 2011 Annual Conference Extended Abstracts on Human Factors in Computing Systems -
CHI EA ’11 (pp. 2425-2428). doi:10.1145/1979742.1979575
Dichev, C., & Dicheva, D. (2017). Gamifying education: What is known, what is believed and what remains uncertain: a critical
review. International Journal of Educational Technology in Higher Education, 14(1), 9. doi:10.1186/s41239-017-0042-5
Huotari, K., & Hamari, J. (2012). Defining gamification: A Service marketing perspective. In Proceeding of the 16th
International Academic MindTrek Conference (pp. 17–22). doi:10.1145/2393132.2393137
Jorgensen, P. C. (2013). Software testing: A Craftsman’s approach, third edition. Software Testing: A Craftsman’s approach (3rd
ed.). New York, NY: Auerbach Publications. doi:10.1201/9781439889503
Kapp, K. M. (2012). The Gamification of learning and instruction: Game-based methods and strategies for training and
education. San Francisco, CA: John Wiley & Sons.
58

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
Kasurinen, J., & Knutas, A. (2018). Publication trends in gamification: A Systematic mapping study. Computer Science Review,
27, 33–44. doi:10.1016/j.cosrev.2017.10.003
Klock, A. C. T., Gasparini, I., & Pimenta, M. S. (2016). 5W2H framework. In Proceedings of the 15th Brazilian Symposium on
Human Factors in Computer Systems - IHC ’16 (pp. 1–10). New York, NY: ACM Press. doi:10.1145/3033701.3033715
Klock, A. C. T., Ogawa, A. N., Gasparini, I., & Pimenta, M. S. (2018). Does gamification matter? A Systematic mapping about
the evaluation of gamification in educational environments. In Proceedings of the 33rd Annual ACM Symposium on Applied
Computing (pp. 2006–2012). New York, NY: ACM Press. doi:10.1145/3167132.3167347
Klock, Ana Carolina Tomé, Gasparini, I., & Pimenta, M. S. (2019). User-centered gamification for e-learning systems: A
Quantitative and qualitative analysis of its application. Interacting with Computers, 31(5), 425-445. doi:10.1093/iwc/iwz028
Koivisto, J., & Hamari, J. (2019). The Rise of motivational information systems: A Review of gamification research.
International Journal of Information Management, 45, 191-210. doi:10.1016/j.ijinfomgt.2018.10.013
Kotini, I., & Tzelepi, S. (2015). A Gamification-based framework for developing learning activities of computational thinking. In
Gamification in Education and Business (pp. 219–252). doi:10.1007/978-3-319-10208-5_12
Landers, R. N. (2019). Gamification misunderstood: How badly executed and rhetorical gamification obscures its transformative
potential. Journal of Management Inquiry, 28(2), 137–140. doi:10.1177/1056492618790913
Lavoué, É., Monterrat, B., Desmarais, M., & George, S. (2019). Adaptive gamification for learning environments. IEEE
Transactions on Learning Technologies, 12(1), 16–28. doi:10.1109/TLT.2018.2823710
Lee, J. J., & Hammer, J. (2011). Gamification in Education: What, how, why bother? Academic Exchange Quarterly, 15, 1–5.
Liu, L., & Stacey, P. (2015). Development process of intrinsic gamification in a learning difficulty context. In UK Academy for
Information Systems Conference Proceedings 2015, 4. Retrieved from http://aisel.aisnet.org/ukais2015/4
Marczewski, A. (2015). Even ninja monkeys like to play: Gamification, game thinking and motivational design. London, United
Kingdom: Blurb.
Martí-Parreño, J., Seguí-Mas, D., & Seguí-Mas, E. (2016). Teachers’ attitude towards and actual use of gamification. Procedia -
Social and Behavioral Sciences, 228, 682–688. doi:10.1016/j.sbspro.2016.07.104
Meder, M., Plumbaum, T., & Albayrak, S. (2017). A Primer on data-driven gamification design. In Proceedings of the Data-
Driven Gamification Design Workshop (pp. 12-17). Retrieved from
https://pdfs.semanticscholar.org/c8dd/744530be00bc3b12046b60facb4b1bd47137.pdf
Mora, A., Riera, D., González, C., & Arnedo-Moreno, J. (2017). Gamification: A Systematic review of design frameworks.
Journal of Computing in Higher Education, 29(3), 516-548. doi:10.1007/s12528-017-9150-4
Mora, A., Zaharias, P., González, C., & Arnedo-Moreno, J. (2016). FRAGGLE: A Framework for agile gamification of learning
experiences. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes
in Bioinformatics) (Vol. 9599, pp. 530–539). doi:10.1007/978-3-319-40216-1_57
Nielsen, J., & Landauer, T. K. (1993). A Mathematical model of the finding of usability problems. In Proceedings of the SIGCHI
conference on Human factors in computing systems - CHI ’93 (pp. 206–213). New York, NY: ACM Press.
doi:10.1145/169059.169166
Nielsen, L. (2014). Personas. In Encyclopedia of Human-Computer Interaction (pp. 1-37). Retrieved from
https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/personas
Oliveira, W., & Bittencourt, I. I. (2019). Tailored gamification to educational technologies. Singapore: Springer Singapore.
doi:10.1007/978-981-32-9812-5
Orji, R., Tondello, G. F., & Nacke, L. E. (2018). Personalizing persuasive strategies in gameful systems to gamification user
types. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1-14). New York, NY: ACM.
Palomino, P. T., Toda, A. M., dos Santos, W. O., Cristea, A. I., & Isotani, S. (2019a). Narrative for gamification in education:
why should you care? In Proceedings of the 19th IEEE International Conference on Advanced Learning Technologies (pp. 97-
99). Maceió, Brazil: IEEE Computer Society.
Palomino, P. T., Toda, A. M., Oliveira, W., Rodrigues, L., & Isotani, S. (2019b). Gamification journey: A Novel approach for
classifying gamer types for gamified educational systems. In Proceedings of SBGames 2019 (pp. 165-173). Rio de Janeiro, Brazil:
Sociedade Brasileira de Computação.

59

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms
Palomino, P. T., Toda, A., Oliveira, W., Rodrigues, L., Cristea, A. I., & Isotani, S. (2019c). Exploring content game elements to
support gamification design in educational systems: Narrative and storytelling. In Proceedings of the SBIE 2019 (pp. 773–782).
doi:10.5753/cbie.sbie.2019.773
Paula, F. R. De, & Fávero, R. da P. (2016). A Gamificação da educação na compreensão dos profissionais da educação [The
Gamification of education in the perspective of education professionals]. In SBC - Proceedings of SBGames 2016 (pp. 1459–
1465). São Paulo, Brazil: Sociedade Brasileira de Computação.
Pedreira, O., García, F., Brisaboa, N., & Piattini, M. (2015). Gamification in software engineering - A Systematic mapping.
Information and Software Technology, 57(1), 157–168. doi:10.1016/j.infsof.2014.08.007
Preece, J., Rogers, Y., & Sharp, H. (2015). Interaction design: Beyond human-computer interaction. Edinburgh Gate Harlow,
United Kingdom: John Wiley & Sons.
Sánchez-Mena, A., & Martí-Parreño, J. (2016). Gamification in higher education: teachers’ drivers and barriers. In Proceedings of
the International Conference of The Future of Education (pp. 180-184). Florence, Italy: Libreriauniversitaria.it.
Seaborn, K., & Fels, D. I. (2014). Gamification in theory and action: A Survey. International Journal of Human-Computer
Studies, 74, 14–31. doi:10.1016/j.ijhcs.2014.09.006
Simões, J., Redondo, R. R. D., & Vilas, A. A. F. (2013). A Social gamification framework for a K-6 learning platform. Computers
in Human Behavior, 29(2), 345–353. doi:10.1016/j.chb.2012.06.007
Sumual, H., Batmetan, J. R., & Kambey, M. (2019). Design sprint methods for developing mobile learning application. KnE
Social Sciences, 3(12), 394–407. doi:10.18502/kss.v3i12.4106
Thiebes, S., Lins, S., & Basten, D. (2014). Gamifying information systems: A Synthesis of gamification mechanics and dynamics.
In Proceedings of the Twenty Second European Conference on Information Systems (pp. 1–17). Retrieved from
http://aisel.aisnet.org/ecis2014/proceedings/track01/4/
Toda, A. M., do Carmo, R. M. C., da Silva, A. P., Bittencourt, I. I., & Isotani, S. (2018a). An Approach for planning and
deploying gamification concepts with social networks within educational contexts. International Journal of Information
Management, 46, 294-303. doi:10.1016/J.IJINFOMGT.2018.10.001
Toda, A. M., Valle, P. H. D. D., & Isotani, S. (2018b). The Dark side of gamification: An Overview of negative effects of
gamification in education. In Communications in Computer and Information Science (vol. 832, pp. 143–156). doi:10.1007/978-3-
319-97934-2_9
Toda, A. M., Oliveira, W., Klock, A. C., Palomino, P. T., Pimenta, M., Gasparini, I., Shi, L., Bittencourt, I. I., Isotani, S., &
Cristea, A. I. (2019a). A Taxonomy of game elements for gamification in educational contexts: Proposal and evaluation. In 2019
IEEE 19th International Conference on Advanced Learning Technologies (ICALT) (pp. 84-88). doi:10.1109/ICALT.2019.00028
Toda, A. M., Klock, A. C. T., Oliveira, W., Palomino, P. T., Rodrigues, L., Shi, L., Gasparini, I., Bittencourt, I., I., Isotani, S., &
Cristea, A. I. (2019b). Analysing gamification elements in educational environments using an existing Gamification taxonomy.
Smart Learning Environments, 6(1), 16. doi:10.1186/s40561-019-0106-1
Toda, A. M., Oliveira, W., Shi, L., Bittencourt, I., Isotani, S., & Cristea, A. (2019c). Planning gamification strategies based on
user characteristics and DM: A Gender-based case study. In Proceedings of the Educational Data Mining 2019 conference (pp.
438–443). Retrieved from http://arxiv.org/abs/1905.09146
Toda, A., Palomino, P., Rodrigues, L., Oliveira, W., Shi, L., Isotani, S., & Cristea, A. (2019d). Validating the effectiveness of
data-driven gamification recommendations: An Exploratory study. In Anais do XXX Simpósio Brasileiro de Informática na
Educação (SBIE 2019) (vol. 30, pp. 763-772). doi:10.5753/cbie.sbie.2019.763
Tondello, G. F., Wehbe, R. R., Diamond, L., Busch, M., Marczewski, A., & Nacke, L. E. (2016). The Gamification user types
Hexad scale. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play - CHI PLAY ’16 (pp. 229–
243). doi:10.1145/2967934.2968082
Vargas-Enriquez, J., Garcia-Mundo, L., Genero, M., & Piattini, M. (2015). A Systematic mapping study on gamified software
quality. In Proceeding of the 7th International Conference on Games and Virtual Worlds for Serious Applications (VS-Games)
(pp. 1–8). doi:10.1109/VS-GAMES.2015.7295763
Wongso, O., Rosmansyah, Y., & Bandung, Y. (2014). Gamification framework model, based on social engagement in e-learning
2.0. In Proceedings of the 2nd International Conference on Technology, Informatics, Management, Engineering, and
Environment (TIME-E) (pp. 10–14). doi:10.1109/TIME-E.2014.7011583
Zichermann, G., & Cunningham, C. (2011). Gamification by design: Implementing game mechanics in web and mobile apps (1st
ed.). Sebastopol, CA: O’Reilly Media.
60

This content downloaded from

148.252.133.35 on Thu, 24 Sep 2020 14:13:57 UTC
All use subject to https://about.jstor.org/terms