2013 3rd International Conference on Computer Science and Network Technology

An Empirical Study of Gamification Impact on

E-Learning Environment

Afifa Amriani Alham F. Aji Andika Y. Utomo Kasiyah M. Junus

Faculty of Computer Science Faculty of Computer Science Faculty of Computer Science Faculty of Computer Science
Universitas Indonesia Universitas Indonesia Universitas Indonesia Universitas Indonesia
Depok, Indonesia Depok, Indonesia Depok, Indonesia Depok, Indonesia
[email protected] [email protected] [email protected] [email protected]

Abstract—Effective e-Learning requiresstudents to be active, by conducting it with online learning simulation, gathering the
thus, creating an environment that engages the students is one of daily logs, and analyze the data quantitatively.
the challenges. One of the ways to build an engaging e-Learning
environment is by applying gamification concept. It improves II. GAMIFIED E-LEARNING
user engagement by enrichingthe system withthe game design
elements. This research aims to analyze the impact of A. Gamification Concept on Education
gamification environment on e-Learning system by looking at Gamification is one of the new evolutions on supporting
student’s quantitative interaction. 38 students from different high user interaction on a system. The practices of gamification are
schoolswere voluntarily joined to test out two different system. well used in business or marketing, but the use of gamification
This experiment focus on observing the impact of removing and itself has been varied to wider area and nudge in some fields of
adding gamification on the existing e-Learning system. The result studies. Thus, bringing up the issue of proposing gamification
shows that removing gamification causes significant decrement in education is one of a break through to support the learning
on student participation while adding gamification had no process.
significant impact.
Keywords—e-Learning; Gamification; User Engagement; Gamification denoted as “use of game design elements in
Human Computer Interaction. non-game contexts” (Deterding et al., 2011)[8]. It is parts of
gaming [4], which are represented as several interactions
I. INTRODUCTION between actors and also requires active involvement of the
In the recent years, e-Learning becomes the solution to players [9]. The avail of gamification in educational fields
surmount the barriers of time and place in learning. The tends to be new experiential way to trigger student presence to
researchers have been working for years to improve the e- make an achievement. With the gamification concept, the new
Learning effectiveness, making the e-Learning paradigm way of learning is expected to be more challenging to the
substantially changed, from teaching to learning [1][2]. It students and embroiled them to create an active learning.
points out that e-Learning is not simply focusing on the needs
B. E-Learning with Gamification
of educating but it is more to paying attention on learning.
These result to emphasize e-Learning as cognitive tool and E-learning emphasizes the metaphor of participation. It
emotional motivation tool which promote independent learning points the learning process should be developed through
[3]. E-Learning should encourage students to become active collaboration and communication [1]. Although e-Learning is
learners andsupport the real class-like environment to one of the new innovative solutions on education, it is still
encourage the participation. To accomplish this goal, the lacking on emotional interaction on the users. This concern
system should trigger more efficient and engage learning point out the feeling of engagement through the system is not
behavior [4]. well perceived. Gamification on the other hands had the
concept on creating users involvement and engagement
One of the ways to build the engaging system is by through the system. Its main goal is to raise the engagement of
applying the gamification concept. The main goal of applying users by using game-like techniques such as scoreboards and
this concept is to raise the user’s motivation by improving the personalized fast feedback (Flatla et al, 2011)[10]. Combining
system engagement [5] and give the personalized fast feedback gamification with e-learning will help creates learning
to user’s activities [6]. That goal is supported by the fact that environment which make the students feel more motivated,
students learn more effectively when they are engaged and engaged and triggered to complete the task given.
interested [7].
According to "self-determination theory" from Deci and
The purpose of this research is to discover the impression Ryan there are three needs of intrinsic motivation:
between e-Learning and gamification in case of raising user
engagement. This research will study the impact of adding or 1) Relatedness
removing gamification fromits existing system for each group Relatedness defined as things that connected a person with
their personal goals and their meaningful community of interest

978-1-4799-0561-4/13/$31.00 ©2013 IEEE 265 Dalian, China

 2) Competence construction of content. It has several features considered
Person interests on challenging task aiming to complete typical of an e-learning platform, such as course page,
his/her goals. discussion forum,calendar, file download, assignment
submission, etc. Recent version of Moodle alsosupport badges
3) Autonomy feature which is one of the gamification element.
Autonomy satisfies the need to control one’s own life by
taking part on voluntary play to fulfill their individual pursuit We created two similar e-Learning environments with exact
[11]. same design, system, material and facilitators. The only
difference is one of the environment will be implemented with
Nicholson [12] suggests that gamification can be used to gamification elements. Two different groups of students will
improve these intrinsic motivations by making the game design assigned into corresponding environment to see how they
elements used be meaningful for the users, this should be the perform.
students on our gamified e-Learning environment.
B. Gamification Design
The idea of using gamification on e-Learning is presented
as blending the game design elements on e-Learning system. The idea of implementing gamification concept on the e-
The designs of the game element used for e-Learning are based Learning system is depend on the game design element used.
on three design area [13]: This elements have the important role to create the effective e-
Learning as well as to improve student’s motivation throughout
1) Cognitive Area the process. There are five gamification elements chosen to be
The gamifications on cognitive area designed focus on implemented onour e-Learning system. Those elements are
making complex system where students do the task gradually leaderboard, score, title, badge, and completion track, as shown
based on rules stated and utilize the system to complete the task below.
given step by step to each level completion.
TABLE I. GAMIFICATION ELEMENTS IN THE E-LEARNING SYSTEM
2) Emotional Area
As for the emotional area, the gamification elements are Gamification elements Description
placed to create positive emotions on the students by including Each student will receive score for their
virtual reward for every completed task. These rewards raise Score assignment performance and their various
activities in the system
the positive emotion that will lead the students feel they were Students will be awarded with badges by
improving and performing well. Badge completing various action that related
with their activities
3) Social Area Top ranked students will be displayed in
As for the social area, gamification elements planted to give Leaderboard leaderboard based on their scores and
competitive mechanism, for example leaderboard or ranking. badges collected.
This competitive mechanism is expected to motivate the Each student will get title based on their
received score. The title presented in
students to improve themselves in order to gain social Title
different levelsand will be attached in
recognition. their accounts name.
Each student can see their own progress
Each of these game design elements has different effects to Completion track on the system, what task they finished,
the students.In this research, we want to see how far the game and what material they have viewed.
design elements will impact the students. In this research the
students are divided into two groups. Each one will experience C. Learning Curriculum
different environment for two weeks. There are two
TABLE II. CURRICULUM DESIGN
environments that will be tested in this research.
Hereinafter, we interested on finding how will the Day Material Task Quiz Forum
gamification effect on student behavior for each group. Is there Day 1 Review of Java ¥ - ¥
any significant influence on adding or removing gamification Day 2 For Loop and Conditional Branch ¥ ¥ ¥
on e-Learning environment? Day 3 Logic Foundation ¥ - ¥
III. E-LEARNING SYSTEM DESIGN Day 4 User Interface ¥ - ¥
Day 5 First Week Review - - ¥
A. E-Learning Environment
Day 6 Quiz - ¥ -
To simulate the e-Learning process, this research uses
Moodle version 2.5.1. Moodle(Modular Object-Oriented Day 7 Weekend Break - - ¥
Dynamic Learning Environment) is one of theopen source e- Day 8 Array ¥ - ¥
Learning systems that can be downloaded for free from its web Day 9 Array 2 ¥ - ¥
page. It was built in PHP and support relationaldatabases such Day 10 Debugging - ¥ ¥
as MySQL, PostgreSQL, or Microsoft SQL Server as well as
Object-Oriented database like Oracle. Day 11 Java Class ¥ - ¥
Day 12 Overall Review - - ¥
Moodle was developed primarily to help educators create
Day 13 Final Test - - -
online courses with focuson interaction and collaborative

266
 The curriculum was created by considering some aspects TABLE III. CLASS AND ENVIRONTMENT DISTRIBUTION
which included the time, material and the respondents. The
Time Class A Class B
learning process were held withintwo weekscontinuously. The
learning materials were basic Java programming language, and Week 1 Gamification Environment Non-gamification
the respondents were high school students.
Week 2 Non-gamification Gamification Environment
Table II shows the activities of the e-Learning simulation.
Every day the students learn new material and discuss related
topic in the forum. For practicing, the facilitators also prepare The measurement acquired by comparing quantitative
cases in the forum in order to deepen the student understanding aspects of the participants before and after the system was
about the material of the day. switched, as shown in Table III. The number of assignments,
and discussion forum posted was same for each week in order
IV. RESEARCH METHODS to keep the data balanced. These two weeks data of student
interaction will later used to analyze the impact of gamification
on students participation.
C. Learning Process

Fig. 1. Research Flow

Flow in Fig. 1 represents five main process of this research.
In the first step, we create e-Learning system using Moodle as
mentioned in previous chapter. Afterwards, the participants
were selected from various high schools. Then the selected Fig. 2. Work activities based on roles within the learning process
students perform the simulationin the e-Learning system for As shown in Fig. 2, there are four main activities in
two weeks.All of the respondent activities within the gamifiede-Learning environment where the facilitators and
simulation period were recorded in log filesandanalyzed by students as the actors. Every day, the facilitators create and
doing some text processing method. post learning materials into the system as well as the task of the
A. Research Population day. And every three days there is a quiz to be done by the
student. The facilitators facilitate forums for the students to
Thirty-eight high school students were voluntarily join to discuss about the topic given on that day, and also trigger them
test out the gamified e-Learning.The participant’s age ranged with some cases related to the topic.
from 15 to 18 years old, with 24 male students and 14 female
students. The participants were taken from the alumnus of Java V. RESULT
ProgrammingWorkshop for beginner and from senior high
This experiment concerns on quantitative values of student
schools that had passion in programming. Our assumption was
activities based on their logs in both groups.The log data
these participants level of understanding about the learning
consist of daily activities of each participant. We record each of
material are similar.
participant activities which consist of their course view,
The participants were divided into two classes to figure out assignment submission, discussion created, discussion reply
the effect of gamification in e-Learning. The class A consists of and discussion view.
18 students while class B consists of 20 students. The
participants were tested out with pre-test of basic programming ഥ ୈ ൌ  σሺୖభିୖమሻ 
   
୒
to measure their knowledge and capability. Before taking a step
to the real simulation, the research provide few days of trial use where:
of e-Learning system. This is done due to familiarize the ܺത஽ : Mean of differences
participants with the new system that will be used in this
experiment. R1 : Data record for the first week
R2 : Data record for the second week
B. Population Division
N : Number of students
The experiment of the e-Learning systems held for two
weeks. At the beginning of the experiment, the participants are To see the impact of adding or removing gamification on e-
divided into two classes and assigned into two different Learning system, the data are summarized for each week to see
corresponding systems. After a week of experiment, the the general differences. Mean of differences of each user
systems are switched for both groups. Therefore, every activity record is as in (1) also calculated to indicate the
participant experienced two different systems in two weeks. significance for each of them.

267
 TABLE IV. ACTIVITY RECORD OF CLASS A The value in table VI and Fig. 3 represents the value of t for
each recordedactivities in class A. Based on t-distribution table,
Total
critical value of 2.1098 is acquired with significance level of
Record Week 1 Week 2 (non- ܺത஽
0.05 and 17 degrees of freedom. The figure shows that by
(gamification) gamification)
appliying gamification on first week then remove it on the
Course view 889 554 18.6111 second week causes significant decrement on the number of
Assignment submitted 40 38 0.11111 participation in course view, discussion created, and discussion
Discussion created 16 7 0.5 view, as those t value is greater than the critical value.
Discussion reply 246 239 0.388889 The graph in Fig. 3provides the value of t for each record
Discussion view 1530 1210 17.7778 activity in class B. With 19 degrees of freedom and
significance level of 0.05, the critical value calculated as
TABLE V. ACTIVITY RECORD OF CLASS B 2.0930.The t value of each activity record should be greater
than 2.0930 to show it significantly decreased or it should be
Total smallerthan-2.0930 to be said as significantly increased.
Record Week 1 (non- Week 2 ܺത஽
gamification) (gamification)
Calculation of t value on each data record on class B show
that even though it seems there are improvement by applying
Course view 552 630 -5.4
gamification on e-Learning system, none of t value calculated
Assignment submitted 36 34 0.1 has exceed the critical value. Thus, it can not be claimed that it
Discussion created 2 14 -0.6 has differences to user participation by switching from non-
Discussion reply 59 95 -1.8 gamification environment to gamification environment. The
main reason is that even there are increasing student
Discussion view 439 683 -12.2
participation record, it only happened to a few students, hence
The student activity logs in both groups show that make it no real differences in the participation level of the
gamification gave impact on student participation in e-Learning whole user from class B.
system. As shown in Table IV, switching from gamification
environment to non-gamification in class A decreased the VI. DISCUSSION
number of participations and activities. On the other hand, class
B data record shows increasing number of participations and A. Outcome Analysis
activities after switching from non-gamification to gamification The behavior of the participants in the experiment shows
environment as pointed out in Table V. that there are different outcome for both group while
participating the learning process in the system.Both group
To clarify the differences, two-tailed dependent t-test were shows change of behavior after environment switching,
used to check the significancy of gamification. The therefore by adding or removing gamification to existed e-
significancy level of 0.05 was picked for this experiment. The Learning system causes change of participation level to some
paired value is the difference between each activity log in first participants. Based on this experiment, removing gamification
and second week of experiment. environment from existing e-Learning system causes more
impact compared to adding it.
TABLE VI. CALCULATED T-VALUE
Experiencing conventional e-Learning environment on the
T Value
Record first week points out to little participations. Thus, adding
Class A Class B gamification on the second week turns out only affecting a few
Course view 2.44498 -0.499226 participants, as they already saturated by the conventional e-
Assignment submitted 0.324946 0.270849 Learning system. On the other side, applying gamification on
the first week causes many participants became interested with
Discussion created 2.47386 -1.43105
the system and encouraged them to be more active. This cause
Discussion reply 0.469126 -1.12709 a lot of students enthusiast in the learning process on the first
Discussion view 3.02039 -1.07975 week. By removing gamification in the second week reduce
their motivation factors, therefore making them not as active as
in first week.
B. Obstacle and Challenge
As this research held, we face several issuesthat might
influence the results of this experiment.
1) Technical aspect
There were some technical issues that might affect the
result, for example some students had their PC broken and also
difficulties on accessing the internet.
2) Environmental aspect
Fig. 3. T-value chart of each activity record of both class

268
 The differences of learning material that delivered in first of the external factors that might affect student motivation. It
and second week could take effect on student participation. also noticed that this research scope is limited on high school
Some of students claim the learning materials were too hard students. Another experiment might needed on wider and more
which make them have difficulties on following the discussion. heterogeneous population as there might be different result
While there are also phase that the students felt the learning applying gamification in e-Learning in different population.
material is interesting due to it is a new topic for them. Some
students also claim that the user interface is needed to be fixed, REFERENCES
although they find it easy to use it is still need an improvement
on the design. [1] U. D. Ethlers, "Web 2.0, e-Learning 2.0, Quality for New Learning
Cultures," in Quality Assurance in Education, 3rd Ed. Bradford, United
3) Participants aspect Kingdom:Emerald Group Publishing, of Proquest, 2009.
Different personality of each participant may affect the [2] P. Henry, “E-Learning Technology, Content, and Services,” in
result, as it influences the discussion and interaction of each Educating + Traingin, vol. 43, MCB University Press, 2001,pp. 249-255
people. Other issue we face on participant aspect that should be [3] S.-m. Zhang, Q.-c. Zhan and H.-m. Du, "Research on the HUman
noted is some of them cannot participate actively in a few days Computer Interaction of E-Learning," IEEE, 2010.
because of affected by disease. The daily activity of each [4] S. Deterding, D. Dixon, R. Khaled and L. Nacke, "From Game Design
student is also different, hence results in different available Elements to Gamefulness : Defining Gamification," ACM, 2011.
online time. [5] Pavlus J., “The Game of Life,” Scientific American, 2010.
[6] C. I. Muntean, “Raising Engagement in e-Learning through
VII. CONCLUSION Gamification,” 6th International Conference on Virtual Learning, 2011.
[7] H. Lee and Y. Y. Doh, “A Study on the Relationship Between
As the result of the researchshowed above, it can be Educational Achievement and Emotional Engagement in a Gameful
concluded that adding gamification on a non-gamified Interface for Video Lecture System,” IEEE International Symposium on
environment did not affect significantly on the students Ubiquitous Virtual Reality, 2012.
participation in the system. While on the other hand, removing [8] S. Deterding, K. O'Hara, M. Sicart, D. Dixon and L. Nacke,
gamification from a gamified e-Learning decreased the "Gamification: Using Game Design Elements in Non-Gaming
student’s performance significantly. Although it give different Contexts," ACMProceeding of CHI extended abstract, Vancouver, 2011.
result for both classes, the gamification itself had created the [9] K. Huotari and J. Hamari, "Defining Gamification - A Service
Marketing Perspective," ACM, 2012.
active atmosphere on the students for both environments.
[10] Flatla, D., Gutwin, C., Nacke, L., Bateman, S., Mandryk, R.,
The result denoted that gamification did improve student “Calibration Games: Making Calibration Tasks Enjoyable by Adding
participation, but the role of facilitators cannot be eliminated to Motivating Game Elements,” UIST 2011, Santa Barbara, California
support this process runs effectively. Teacher or facilitator is [11] F. Groh, "Gamification: State of the Art Definition and Utilization,"
Human Computer Interaction, p. 41, 2012.
needed to create dynamic interaction between learner and the
teacher itself [1]. They have to accommodate and facilitate the [12] S. Nicholson, "A user-centered Theoretical Framework for meaningful
gamification," in Games Learning Society, Madison, 2012.
needs of the students by being involved in the learning process,
[13] Domínguez, A., Saenz-de-Navarrete, J., de-Marcos, L., Fernández-Sanz,
triggered them to be active, and give them some feedbacks. L., Pagés, C., Martínez-Herráiz, J.J. “Gamifying learning experiences:
Gamification role in e-Learning system is only as a booster for Practical implications and outcomes.” Computers & Education. 63. pp.
the students, therefore facilitator is still needed to keep the 380- 392.
learning process as real as class-like environment. [14] D. S. Moore, The Basic Practise of Statistics Second Edition, New York:
Michelle Russel Julet, 2000.
This research focused on quantitative aspect of
gamification impact on e-Learning, therefore future work is
needed to see the impact of gamification on qualitative
aspect. Further research could be elaborated to view the impact

269