Accepted Manuscript

Augmented reality for STEM learning: A systematic review

María-Blanca Ibáñez, Carlos Delgado-Kloos

PII: S0360-1315(18)30102-7
DOI: 10.1016/j.compedu.2018.05.002
Reference: CAE 3345

To appear in: Computers & Education

Received Date: 11 November 2017

Revised Date: 30 April 2018
Accepted Date: 3 May 2018

Please cite this article as: Ibáñez Marí.-Blanca. & Delgado-Kloos C., Augmented reality for STEM
learning: A systematic review, Computers & Education (2018), doi: 10.1016/j.compedu.2018.05.002.

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to
our customers we are providing this early version of the manuscript. The manuscript will undergo
copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please
note that during the production process errors may be discovered which could affect the content, and all
legal disclaimers that apply to the journal pertain.
 ACCEPTED MANUSCRIPT

Augmented reality for STEM learning: A systematic review

María-Blanca Ibáñez
(Universidad Carlos III de Madrid, Madrid, Spain
[email protected])

Carlos Delgado-Kloos
(Universidad Carlos III de Madrid, Madrid, Spain
[email protected])

PT
Abstract

This study presents a systematic review of the literature on the use of augmented reality
technology to support science, technology, engineering and mathematics (STEM) learning. It

RI
synthesizes a set of 28 publications from 2010 to 2017. A qualitative content analysis is used to
investigate the general characteristics of augmented reality applications in STEM education,
the instructional strategies and techniques deployed in the studies reviewed, and the

SC
evaluation approaches followed in the interventions. This review found that most augmented
reality applications for STEM learning offered exploration or simulation activities. The
applications reviewed offered a number of similar design features based on digital knowledge

U
discovery mechanisms to consume information through the interaction with digital elements.
However, few studies provided students with assistance in carrying out learning activities.
AN
Most of the studies reviewed evaluated the effects of augmented reality technology in
fostering students’ conceptual understanding, followed by those that investigated affective
learning outcomes. A number of suggestions for future research arose from this review.
M

Researchers need to design features that allow students to acquire basic competences related
with STEM disciplines, and future applications need to include metacognitive scaffolding and
experimental support for inquiry-based learning activities. Finally, it would be useful to explore
D

how augmented reality learning activities can be part of blended instructional strategies such
as the flipped classroom.
TE

Keywords

Augmented reality; Applications in STEM education; Interactive learning environments;

Systematic review.
EP

Research highlights

1. A review of the literature on AR technology to support STEM education is shown.

C

2. Most AR applications use trigger mechanisms to aid the consumption of information.

3. Some AR applications allow the exploration of digital assets to build knowledge.
AC

4. AR STEM education research should focus on science skill-based outcomes.

5. AR STEM applications should offer metacognitive and experimental support.
 ACCEPTED MANUSCRIPT

Title
Augmented reality for STEM learning: A systematic review

1. Introduction
Augmented reality (AR) is a 3D technology that enhances the user’s sensory perception of the
real world with a contextual layer of information (Azuma, 1997). AR has become a popular
topic in educational research in the last decade (Akçayır & Akçayır, 2017; Bacca, Baldiris,
Fabregat & Graf, 2014). The main determinants of AR acceptance in the educational arena

PT
include the availability of low-cost handheld devices with innovative features that allow the
deployment of AR-based applications, and the recognition of AR by the New Media
Consortium (www.NMC.org) and the Educause Learning Initiative (www.educause.edu) as one

RI
of the most promising technologies for both higher and K-12 education. AR’s media
characteristics, namely sensory immersion, navigation and manipulation, seem to work as
promoters of positive emotions while learning, and create more efficient and better learning

SC
outcomes (Cheng & Tsai, 2013; Wu, Lee, Chang & Liang, 2013). Some researchers have pointed
out that AR has potential educational affordances which are especially useful in the science,
technology, engineering, and mathematics (STEM) fields, including spatial ability, practical
skills, conceptual understanding, and scientific inquiry learning (Bujak, Radu, Catrambone,

U
Macintyre, Zheng & Golubski, 2013; Cheng & Tsai, 2012; Dunleavy, Dede & Mitchell, 2009; Wu,
Lee, Chang & Liang, 2013). This study focuses on scientific educational contexts.
AN
Among the most popular educational contexts of application of AR-technology are the
humanities and arts (Di Serio, Ibáñez & Delgado, 2013; Liu & Tsai, 2013), eHealth (Calle-Bustos,
M

Juan, García-García & Abad, 2017), engineering, manufacturing and construction (Henderson &
Feiner, 2011), and science (Kerawalla, Luckin, Seljeflot & Woolard, 2006; Martín-Gutiérrez,
Saorín, Contero, Alcañiz, Pérez-López & Ortega, 2010; Squire & Jan, 2007). Each study provides
D

valuable insight into the learning advantages that this technology might provide in its
application field, and some reviews have synthesized the results of previous studies to identify
TE

broader research trends (Akçayır & Akçayır, 2017; Bacca, Baldiris, Fabregat, & Graf, 2014;
Radu, 2014). However, there are few recent studies providing comprehensive explanations of
the educational effects and implications of AR in the STEM fields (Cheng, & Tsai, 2013). To fill
this gap, this study systematically reviews and synthesizes the relevant literature, and analyzes
EP

the main tendencies and uses of AR in educational STEM settings together with its advantages
and limitations in terms of providing insights for helpful AR design features and promising
instructional processes to be included in future AR STEM learning environments. Specifically,
the present study poses the following three research questions:
C

1. What are the general characteristics and specific design features of AR-based
AC

learning applications for STEM-AR studies?

2. What are the instructional processes followed by STEM-AR studies?
3. What are the measured outcomes addressed in AR for STEM learning studies?

2. Related Work
Augmented reality is a 3D technology which merges the physical and digital worlds in real
time. Applications based on this technology rest on three pillars: tools to track information
about real-world objects of interest; hardware and software to process information; and
devices to show the user the digital information integrated into the real environment (Azuma
et al., 2001; Carmigniani et al., 2011). AR technology is often described with reference to its
two predominant modes of tracking information from the physical world. The first is image- or

1
 ACCEPTED MANUSCRIPT
maker-based AR, which requires recognition of a marker or specific object to bring up digital
information; the second is location-based AR, which makes use of a device’s GPS to identify
locations at which computer-generated information should be superimposed.

Currently, AR is a popular technology that has become an important focus on research

education in recent years. Systematic reviews of research and applications have found that AR
has potential for pedagogical applications, have identified drawbacks of the technology and
suggested future research topics. These reviews are usually focused on the use of AR
technology within a broad range of educational contexts and provide relevant insights about

PT
how AR technology assists students’ learning. Indeed, Radu (2014) presents a meta-review of
26 publications and identifies a list of the positive and negative impacts of educational AR
technology on student learning. Radu concludes that AR is helpful in increasing students’

RI
motivation, fostering collaboration among students, developing spatial abilities, and improving
performance in physical tasks. Regarding AR’s negative impacts, Radu points out that AR
imposes an extra cognitive load on students and causes usability problems. In a recent study,

SC
Akçayır and colleagues (2017) present a systematic review of 68 research articles that used AR
in educational settings. They consider factors such as the publication year, learner type, and
technologies in AR, and compile the advantages and challenges of using AR in educational
settings. Their study reports a recent increase in the number of AR studies and concludes that

U
AR technology might potentially support learning and teaching when pedagogical issues such
AN
as “need for more class time, unsuitability in crowded classrooms, instructors’ inadequate
experience with technology“(Akçayır & Akçayır, 2017, pp. 8) and usability problems have been
overcomed. Finally, Bacca and colleagues (2014) present a systematic review that analyzes 32
studies published between 2003 and 2013 in six indexed journals; they analyze uses,
M

advantages, limitations, effectiveness, challenges, and features of AR in educational settings.

These authors found an increase in the number of published studies about AR in education
from 2009 to 2013, and identify science and the humanities and arts as the fields where AR has
D

been used the most. They claim that the main advantages of AR are learning gains, motivation,
interaction, and collaboration, whereas the main limitations of the technology are due to
TE

usability problems. Unlike the previous articles, Cheng and Tsai (2012) find valuable trends and
potential research directions for AR-related science learning. They review journals from the
Web of Knowledge and Scopus databases from 2004 to 2011, and select 12 articles or studies
EP

using AR in science learning. They note the AR features, educational context, participants and
affordances in science learning. Based on their findings, they propose that future research
should use mixed methods of investigating learning process and focus more on exploring
C

learner experiences rather than on usability.

The abovementioned studies offer a synthesis that is crucial to understanding the affordances,
AC

barriers and trends of AR technology in educational settings. However, most of these reviews
are not specific to STEM fields, or they do not report advances made in recent years.

3. Method
3.1. The manuscript selection process

To find relevant literature sources, seven well-known online research databases were used
which are related to education and technology (ACM Digital Library, ERIC, IEEExplore, ISI Web
of Science, ScienceDirect, Scopus, and Springer), using the query string: “augmented reality”
AND (education OR learning) AND (STEM OR science OR technology OR engineering OR
mathematics). The last search was conducted on 22 August 2017. The search produced 1358

2
 ACCEPTED MANUSCRIPT
results from previously-used search terms, including 616 duplicates, which were deleted. Titles
and abstracts were reviewed to determine whether they were suitable for the purposes of the
study. During this examination, a set of inclusion and exclusion criteria were adopted (see
Table 1). Authors developed this criteria list by adapting established criteria used in earlier
reviews (e.g. Akcayir, & Akcayir, 2017; Cheng, &Tsai, 2012; Zydney, & Warner, 2016) and
considering that previous studies (Akcayir & Akcayir, 2017; Zydney & Warner, 2016) already
presented complete reviews before 2010. By reviewing titles and abstracts, the number of
articles was narrowed down to 112. The selected articles then were examined by the first
author to determine whether they were suitable for the purposes of the study. The second

PT
author independently reviewed approximately 20% of the articles to confirm the reliability of
the coding method. The intercode agreement rate for coding was 96.6%. Disagreements
between the two coders were resolved through discussion and further review of the disputed

RI
studies. The articles were downloaded and the methods sections reviewed in order to verify
that the articles met the criteria for inclusion in the review. The second author independently
reviewed approximately 20% of the downloaded articles. The intercode agreement was

SC
initially 97.3%, and this was brought to 100% after discussion. A total of 28 articles met the
criteria for inclusion in the final review.

TABLE 1

U
Table 1: Inclusion and exclusion criteria
AN
3.2. Analysis

The articles that met the inclusion criteria were analyzed using a qualitative content analysis
M

method, which is a thematic analysis concentrating on the relationship between content and
context (Elo & Kyngäs, 2008; Hsieh & Shannon, 2005). The authors defined a group of
categories for analysis with their corresponding sub-categories, according to each research
D

question. Categories were useful for grouping studies according to their shared characteristics.
During the systematic review process, several sub-categories emerged and others were refined
TE

in order to reflect the emerging information. Two of the authors of this article manually coded
the studies separately according to their characteristics and classified them according to the
categories and sub-categories defined. In the case of discrepancy, the coders resolved this
EP

through discussion.

4. Results
C

In this section, we present the results of the three research questions, focusing on the themes
of the augmented reality application design, the main approaches followed by instructors
AC

when using augmented reality to facilitate learning, and the evaluation of the interventions.

4.1. Overview of reviewed studies

Of the 28 studies reviewed from 2010 through 2017, the majority of these studies (24 articles)
were published after 2013. National science standards [National Science Teachers’ Association,
2014] were used to classify the topics covered in the studies into the following areas: physical
sciences (10 articles), mathematics (seven articles), life sciences (seven articles), earth sciences
(three articles), and multidisciplinary (one article). The preferred educational context was the
in-class context (19 articles), followed by out-of-class activities with eight articles. Two of the
in-class interventions took place in laboratories, whereas the out-of-class interventions took
place in a science museum (three), botanical garden (two), in a nature center (one), one in

3
 ACCEPTED MANUSCRIPT
school (one), and one at home learning activity. The participants in the studies were mostly
middle school students (12 articles) and higher education students (seven articles). Other
studies focused on elementary students (three articles), primary school students (two articles),
high school students (one articles) technical students (one article), or cut across multiple
school levels (two articles). The number of student participants in the studies ranged from 20
to 874. Most of the studies were carried out with less than 90 participants (21 studies). Other
studies were carried out with a number of participants ranging from 92 to 146 (six studies); the
study of Bursztyn (2017) was carried out with 874 participants. In terms of the duration of
instruction, most of the studies were done in a unique session ranging from seven to 180

PT
minutes (14 articles), four studies did not give details about the duration of intruction, and 10
articles reported studies that lasted up to 18 weeks. An overview of the research studies is
provided in Table 2.

RI
TABLE 2

Table 2: Background information on reviewed articles

SC
4.2. General characteristics of AR applications

This section addresses the first research question. Twety-eight different augmented reality

U
applications were tested by researchers. Table 3 shows an overview of these applications in
terms of building tools, type of application, and augmented reality features included.
AN
4.2.1. Building tools

Regarding the building tools used in the 28 studies reviewed, most of these were self-
M

developed native applications which used self-programmed device sensors (11 studies). Eight
studies used augmented reality development tools: Vuforia (five studies), Metaio (one study),
Layar (one study) and Aurasma (one study). Finally, eight studies did not specify the building
D

tool used for the implementation of their testbed.

TE

4.2.3. Application type

The review of the literature highlights three categories of educative applications: (a)
exploration applications (13 applications) which in turn were divided into augmented books
EP

(two applications), augmented marks (seven studies), and the use of a point of interest to
trigger digital information (four studies); (b) simulation tools (12 applications); and (c) games
(two applications).
C

4.2.2 AR features
AC

In 71.4% of the interventions, image-base AR technology was used, whereas 25% were
location-based. One of the studies did not specify the AR feature used. The applications used a
great variety of digital information to augment the real world. More than half of the
applications reviewed used at least two types of digital information, eight applications used
three different digital elements, four applications used two elements, and three applications
used four elements. The rest of the applications (35.7%) used only one type of digital element.
Regarding the type of digital element used by the applications, 46.4% of the studies reviewed
used text, nearly half of them (42.8%) used 2D images, a nearly a third of the applications
(32.1%) used animations, and 3D objects and videos were used in 28.5% of the applications.
There were also applications that provided audio information (10.7%), and applications that
connected the real world with the World Wide Web (7.1%).

4
 ACCEPTED MANUSCRIPT

TABLE 3

Table 3: General characteristics of augmented reality applications

4.3. Instructional process

According to Akdeniz (2016), instruction can be defined as the whole process applied for
learning to occur and for the development of the target behavior that learners are expected to
show. The process should be carried out in a stimulating learning environment where students

PT
carry out activities that will facilitate learning and help develop behavior appropriate for the
gains students are supposed to make (Clark & Starr, 1968; Moore, 2007). The approaches
followed by teachers to achieve the fundamental aims of instruction are called instructional

RI
strategies (Akdeniz, 2016), and the rules, procedures, tools and skills used to deploy the
instructional strategies are called instructional techniques (Günduz, 2016).

SC
This section addresses the second research question. It presents the instructional strategies
and techniques used in the studies reviewed. For each feature described below, one or two
examples of applications are given to illustrate how the feature has been used in the AR-based

U
applications. An overview of these design features is provided in Table 4.

4.3.1 Instructional strategies

AN
Instructional strategies have been classified under a number of taxonomies according to how
the process functions, how the information is produced, how the information is acquired by
learners or how they are based on the instructional models that act as sources for strategies
M

(Akdeniz, 2016). Following this latest approach, the 28 studies reviewed can be grouped into
three categories: instruction through presentation; instruction through discovery; and
D

cooperative learning.

Eight of the applications reviewed provide students with general ideas about a subject; then,
TE

these ideas are progressively differentiated in terms of details and specificity following a
presentation instructional strategy as defined by Akdeniz (2016). Applications such as the one
presented by Gopalan (2014) are augmented books. In these augmented books, students can
EP

select the appropriate viewpoint for the 3D virtual models appearing out of the book pages
(Billinghurst, Kato & Poupyrev, 2001). Liou and colleagues (2016) present a more active
approach which takes into account the interactive characteristics of the technology. Their
application allows students to explore basic materials knowledge by providing three-
C

dimensional interactive animations. When students touch the interactive models on the
AC

handheld device, more detail information about the model appears as well as interactive
animations and videos.

Scientific discovery instructional strategy is a highly self-directed and constructivistic strategy

for learning. This instructional strategy requires that learners construct knowledge for
themselves by discovering, as opposed to being told about something (Bruner, 1961). The
scientific discovery instructional strategy was used in 13 of the studies reviewed. These studies
can be classified in three different families. Huang et al. (2016) is representative of the first
family of studies. These authors use AR technology to enhance the on-site information in a
botanical garden setting and to allow students to become immersed in environmental
exploration in an authentic outdoor environment. Students who participated in this family of
studies have an active role in building an understanding of the discovered information. Ibáñez

5
 ACCEPTED MANUSCRIPT
et al. (2014) present an in-class interactive AR application which is representative of the
second family of studies. This AR application follows Bruner’s (1961) guidelines for providing
effective instruction by allowing students to infer knowledge through their interaction with 3D
digital objects representing circuit elements, presenting relevant content in a structured way,
and giving immediate feedback (Ibáñez et al., 2014). Finally, a third family of applications uses
discovery instructional strategy in a blended learning environment where (1) the AR
application provides the information and/or data for measurement and (2) in-class activities
promote hypothesis formation, data analysis, and reflection (Kamarainen, Metcalf, Grotzer,
Browne, Mazzuca, Tutwiler & Dede, 2013).

PT
Finally, seven studies were guided by a cooperative/collaborative instructional process. These
studies used blended learning environments where the AR learning tool was used individually,

RI
and the collaboration was carried out in the real world. For example, in a study carried out by
Cascales-Martínez et al. (2017) students did individual activities using an AR tool while
collaborate in the real world to solve mathematical problems (Cascales-Martínez, Martínez-

SC
Segura, Pérez-López & Contero, 2017).

4.3.2 Factors considered when selecting instructional techniques

Although all the 28 studies reviewed consider the characteristics of the learners such as age,

U
duration of the intervention and physical learning environment, only one third of the studies
AN
consider specific characteristics of learners to select teaching techniques or discriminant
factors affecting learning outcomes.

When dealing with low- and high-achieving learners, AR technology has proved sometimes
M

useful in improving learners’ cognitive outcomes. For instance, Lin and colleagues (2015) found
that students with average and low academic achievement exhibit small and medium levels of
learning effectiveness, while students with high academic achievement do not benefit from
D

the use of AR technology. The former knowledge of learners also seems to have a cause-and-
effect relationship on students’ behavior during the teaching-learning process, and it was used
TE

in (Ibáñez, Di Serio, Villarán & Delgado, 2016) to guide students towards more effective
instruction. However, in AR-based learning environments, the relationship between learner
characteristics such as gender or learning style and learning outcomes has not yet been
EP

established.

Finally, Cascales-Martínez et al. (2017) carried out a study with students with special
educational needs in a real educational setting. The instructional content focused on
C

understanding and managing money, coins, and banknotes. The results of the study showed a
significant increase in the level of knowledge and motivation in students with learning
AC

disabilities, learning disorders not otherwise specified, and attention deficit hyperactivity
disorder.

6
 ACCEPTED MANUSCRIPT

4.3.3 Instructional techniques

Following the classification of instructional techniques provided by Gündüz (2016), we

identified five different instructional learning techniques in the studies reviewed: observation,
inquiry, game, role-play, and concept maps. A study was considered to use observation as an
instructional technique when learners triggered digital information such as image, text, video,
or animation by interacting with the AR-based tool. When using observation as an instructional
technique, the learner receives information in a passive way. A more active learner role was

PT
necessary for a study to be considered as using inquiry-based learning. This required the use of
an AR-based tool providing at least one of the operations of science inquiry, namely generating
hyphotheses, designing comparisons, collecting observations, analyzing data, and constructing

RI
interpretations (Quintana, Reiser, Davis, Krajcik, Fretz, Duncan, Kyza, Edelson & Soloway,
2004). Studies that used elements of game mechanics were classified in the category of games.
There were also studies that used a dramatization technique known as role-play, where

SC
students are asked to behave in fictional situations as the character they represent. Finally,
studies were considered to use a concept-map instructional technique when the AR-based tool
allowed the tagging and connection of concepts related to the instruction.

U
The instructional learning technique most often deployed was observation (14 studies). Six of
AN
these 14 studies deployed structured observation, whereas eight of them allowed students
free observation of the contents. Structured observation was used in studies supported by
augmented books (Martín-Gutiérrez, Saorín, Contero, Alcañiz, Pérez-López & Ortega, 2010), by
digital resources via mobile devices (Estapa & Nadolny, 2015), or by a set of augmented reality
M

learning activities organized as a learning flow (Ibáñez, Di Serio, Villarán & Delgado, 2014). Free
observations allowed students to choose both the content and the order in which to explore it,
and were deployed through AR-based simulators (Tarng, Lin, Lin & Ou, 2016) or through AR
D

learning tools based on marker images that simply served as triggers for AR content (Liou,
Bhagat & Chang, 2016).
TE

The second instructional technique most often used in the studies reviewed was inquiry (10
studies). One of these studies documents how the narrative structure of a board game, the
EP

physical floor materials, a student’s first-person embodied experiences, the third-person live
camera feed, and the augmented-reality symbols become integrated in a inquiry learning
activity (Enyedy, Danish & DeLiema, 2015). Laboratories and museums were used in four
studies as community spaces to gain expertise about STEM topics using augmented reality
C

technology. Two of these studies reporting interventions carried out in laboratories were
augmented reality tools supporting inquiry activities in real laboratory settings (Akçayır,
AC

Akçayır, Pektaş & Ocak, 2016; Frank & Kapila, 2017), and two further studies reported the use
of AR-based simulators in science museums (Salmi, Thuneberg & Vainikainen, 2017; Yoon,
Anderson, Lin & Elinich, 2017). Five studies used AR-based simulators to deploy the inquiry
instructional technique. The use of computer simulations in inquiry-based learning has been
found to be ineffective in improving learning processes and outcomes (Kirschner, Sweller &
Clark, 2006; Settlage, 2007). Consequently, some researchers have studied the difficulties that
learners might encounter in the discovery learning process (Rutten, Van Joolingen, & Van der
Veen, 2012) and have proposed scaffolding to overcome these difficulties. Despite this, only
one of the studies reviewed included scaffolding to support inquiry-based learning (Ibáñez, Di-
Serio, Villarán & Delgado, 2016).

7
 ACCEPTED MANUSCRIPT
Game formats were used as instructional techniques in two studies. One of these studies
deployed a virtual environment with outdoor activities that took advantage of the benefits of
games providing immersion-in-context, rewards for correctness, and immediate feedback in
response to student interaction (Bursztyn, Walker, Shelton & Pederson, 2017). The other study
presented an alternative reality game where students were asked to solve mathematical
problems using virtual money (Cascales-Martínez, Martínez-Segura, Pérez-López & Contero,
2017).

One study used role-play combined with inquiry as an instructional technique in the context of

PT
field trip learning. Role-play is a well-known and commonly used drama technique in education
as an instructional technique (Gündüz, 2016). The study reports how augmented reality helped
students to discover relevant information according to their specific role in an out-of-class

RI
activity (Kamarainen, Metcalf, Grotzer, Browne, Mazzuca, Tutwiler & Dede, 2013).

In one study (Chen, Chou & Huang, 2016), concept maps were integrated into AR learning to
develop a learning system for an elementary school science course. The system was developed

SC
to help students explore and organize what they learned in the course. The AR-based learning
tool allowed students to establish the structure of course knowledge by drawing digital links
connecting real images representing the main concepts to be studied.

U
TABLE 4
AN
Table 4: Instructional processes followed in the reviewed articles

4.4. Evaluation of interventions

M

This section addresses the third research question. It presents the research purposes,
methods, and outcomes used in the studies reviewed (see Table 5).
D

4.4.1. Research purposes

Four research purposes emerged from the studies reviewed. Evaluating the effects of AR
TE

learning was the main focus of 25 studies; investigating the affective domain during AR
learning was a goal in 15 studies; evaluating the influence of learner characteristics in the AR
learning process was the purpose of four studies, and assessing the impact of designing an AR
EP

system for AR learning was within the scope of five studies (see Fig. 1 and Table 5).

More than two thirds of the studies (67.8%) had more than one research purpose, and one of
them (Cascales-Martínez, Martínez-Segura, Pérez-López & Contero, 2017) had three different
C

purposes. More than half of the studies (53.5%) investigated the effects of AR learning and the
affective domain during AR learning. Evaluating the effects of AR learning was the sole
AC

research purpose of eight studies (see Fig. 1 and Table 5).

FIGURE 1
Figure 1. Research purpose frequency

4.4.2. Research approaches

Among the 28 analyzed proposals (see Table 4) there were quantitative studies (13),
qualitative studies (five) and mixed-method studies (10).

Quantitative studies were most often employed (46.4%) and these were divided into eight
quasi-experimental studies and five experimental design studies.

8
 ACCEPTED MANUSCRIPT
The qualitative studies included three content analysis studies which interpreted recorded
material to understand knowledge construction by learners in collaborative AR-based inquiry
learning environments (Chiang et al., 2014; Wang et al., 2014), and to gain insights about the
benefits of personalizing of learning in AR learning environments to help focus students’
attention (Ibáñez et al., 2016). The other two qualitative studies presented an interpretative
study and an ethnography study, respectively. The interpretative study examined the potential
of AR as a distributed learning interface for learning and warned about the risks of over-
distribution with respect to usability (Cuendet et al., 2013) whereas Enyedy et al. (2015)
documented a case study and traced how the narrative structure of a board game, the

PT
student’s first-person embodied experiences, and the augmented-reality symbols became
integrated into scientific inquiry activities.

RI
More than a third of the studies reviewed (35.7%) combined quantitative and qualitative
research methods for a better understanding of the learning process. Three different
combinations of quantitative and qualitative studies were found. The first type of combination

SC
was used in five studies, a representative example of which is the work presented by Yoon and
colleagues (2017) who combined a quasi-experimental method with interpretation. Although
the quasi-experimental part of their study showed that students in the AR condition showed
greater knowledge gains when compared with students in the non-AR condition, the

U
qualitative part of their study showed that AR capabilities enabled the visualization of typically
AN
invisible causal mechanisms that underlie complex phenomena, and played an important role
in the knowledge acquisition of the experimental group. The second type of mixed methods
(four studies) combined experimental design with interpretation. For instance, Huang et al.
(2016) presented quantitative results that suggested that an AR location-based intervention
M

made at a botanical garden helped students to develop a positive emotional attatchment to

the experimental activities, whereas interviews showed students’ willingness to learn more
about the environment. Finally, Zimmerman et al. (2016) used a quasi-experimental study to
D

prove that an experimental group using an AR learning tool gained more knowledge about the
life cycle of trees than a control group, and combined this quasi-experimental study with an
TE

ethnographic study to find three sociotechnical interactions that contributed to triggering

situational interests during the summer camp learning experience deployed.
EP

4.4.3. Learning outcomes

This section focuses on measures of student outcomes. Eighteen studies examined possible
effects of using AR-based learning environments on learners’ emotions while the remaining
C

studies did not included any assessment of emotional states. Motivation was the emotional
state that appeared in most studies (seven in total), followed by attitude, enjoyment and
AC

engagement, which appeared in five, four and four studies respectively. Other emotional
states considered were satisfaction, immersion, flow, and interest (see Fig. 2).

FIGURE 2

Figure 2. Frequency of affective outcomes

The 28 studies reviewed were classified into two broad categories according to the type of
cognitive outcomes measured: lower-level cognitive outcomes and higher-level cognitive
outcomes. The first category included works dealing with simpler cognitive processes stated on
the revised Bloom’s taxonomy (Anderson, L.W., & Krathwohl, D.R., 2001): remember, and
understand. The second category included works dealing with more complex cognitive
processes of the aforementioned taxonomy implying create, apply, analyze, and evaluate.

9
 ACCEPTED MANUSCRIPT
The most common cognitive outcome measured was the ability to remember information (18
studies). These studies typically used a pre/posttest format, with multiple-choice or short-
answer questions to assess retention of the presented material in much the same form in
which it was presented. One of these studies also assessed long-term retention (Cai et al.,
2017) by administering a delayed posttest one week after the instructional period concluded.
For three of the 18 studies mentioned above, augmented reality learning affordance related to
the acquisition of enhancing spatial skills was key to the understanding of scientific
phenomena (Cuendet et al., 2013; Lin et al., 2015; Martín-Gutiérrez et al., 2010). Three studies
measured learners’ understanding of STEM topics, through the construction of concept maps

PT
(Chen et al., 2016); pre- and posttests aiming to assess learners’ understanding of lunar
concepts after observing lunar phases over a month (Tarng et al., 2016); and pre- and posttests
to assess tasks related to the training of spatial skills (Cuendet et al., 2013).

RI
Six studies measured high-level cognitive outcomes related to students’ effectiveness in
applying procedures to solve STEM problems, analyze their results and self-evaluate their

SC
progress. In (Cascales et al., 2016), students with special needs were able to solve
mathematical problems by interacting with an AR-based learning tool and with their peers.
Their learning gains were measured through pre- and posttests that assessed whether they
could solve a set of mathematical problems by applying their knowledge. In (Chen et al., 2015)

U
students carried out a self-evaluation of their learning achievements in terms of solving
AN
problems related to earth science phenomena involving day, night, and seasons. Ibáñez and
colleagues (2014) used pre- and posttests to measure students’ learning effectiveness in a
problem-solving activity in physics, using a marker-based electromagnetic laboratory. Chiang
and colleages (2014) analyzed how students created knowledge, in their work students’
M

activities were recorded and then analyzed to study the knowledge construction phases during
an inquiry-based learning activity related to an ecological learning situation. In (Enyedy et al.,
2015), students worked in a blended learning environment, trying to conceptualize a physics
D

phenomenon. The learning effectiveness of the activity was documented using recordings of
students’ answers to open-ended questions on the subject. Wang and colleagues (2014)
TE

analyzed students’ behavior in an inquiry-based learning activity, using video recordings of the
intervention. Their results showed high frequencies for high-level inquiry behaviors, such as
interpreting experimental data or drawing conclusions in an AR simulation system and a
EP

traditional 2D simulation system. However, the AR simulation system engaged the students
more thoroughly in the inquiry process.

4.4.4. Problems reported

C

Only seven studies reported problems with augmented reality technologies that might affect
AC

the effectiveness of learning activities (see Table 5). A problem reported in three of the studies
reviewed was that students need to be trained in the use of augmented reality technology
before using it in learning activities. Another study goes further, suggesting that the training be
extended to instructors in order to expand the use of this technology in current courses. The
most frequent complaints in studies including usability measurement were the system did not
provide immediate feedback (two studies), the system was slow, or the interface was not
intuitive. A more serious concern arose from instructors who reported student distraction,
probably caused by the novelty effect (Frank et al., 2017; Ibáñez et al., 2016; Kamarainen et al.,
2013).

TABLE 5

10
 ACCEPTED MANUSCRIPT
Table 5. Research purposes, methods, and outcomes

5. Discussion
5.1. Design of augmented reality applications
The results show that the reviewed AR-based learning applications for STEM are evenly
distributed among physics, mathematics and life sciences topics. Most of the interventions
related to life sciences topics took place in out-of-class settings using location-based AR-
features, whereas physics and mathematics interventions usually were made in-class using

PT
marker-based or image-based location AR features. Moreover, life sciences topics were mostly
deployed using AR-based exploration tools, with inquiry learning activities orchestrated by
teachers, while physics and mathematics interventions were generally devoted to exploration

RI
or simulation, with an emphasis on the understanding of STEM phenomena. It would be
interesting to see more studies on the use of in-class AR technology incorporating scientific
inquiry orchestrated by teachers and supported more actively for the AR learning tools.

SC
It worth noting that assets were used differently in life sciences topics than in physics or
mathematics. In the first case, assets provided factual knowledge or helped students to
become immersed in the educational context by superimposing digital elements related to the

U
narrative of the lesson. In the second case, assets either allowed the student to see invisible
AN
phenomena or helped students to visualize 3D concrete or abstract objects. Therefore, assets
were mainly used to increase students’ visual perception, and some studies also included
tangible objects (Cuendet et al., 2013; Ibáñez et al., 2014). This is reasonable since students
are more likely to have stronger visual and read-write learning styles (Drago & Wagner, 2004).
M

However, it would be interesting to use audio assets, for instance, to achieve a higher
immersion in the learning context.
D

5.2. Instructional processes

A significant proportion of the studies reviewed promoted instructional strategies based on
TE

the students’ consumption of information in support of in-class learning activities in cross-

sectional studies. Even though these studies reported that learning instruction was effective, it
would be interesting to carry out further research to evaluate whether learning effectiveness is
EP

retained over time or by students familiar with the use of AR technology. It seems advisable to
shift to instructional strategies where students play a more active role in the teaching/learning
process, taking advantage of the potential learning affordances of this emergent technology.
C

Augmented reality has also driven instructional constructivist strategies where students have
the freedom to discover information through interaction, thus potentially promoting the self-
AC

construction of knowledge. However, more research is still needed in this area. In particular, it
is necessary to determine how AR-based tools can be designed to help students to construct
knowledge (Zydney & Warner, 2016) and to find instruments to measure whether students are
indeed building knowledge. On the other hand, researchers have reported learning problems
for some students when they have the freedom to discover information using AR technology in
STEM fields (Dunleavy, Dede & Mitchell, 2009; Ibáñez, Di-Serio, Villarán-Molina & Delgado-
Kloos, 2015). Therefore, it is necessary to find effective scaffolding mechanisms to help
students in the discovery process as well as in inquiry activities for scientific methods (Rutten,
Van Joolingen & Van der Veen, 2012).

11
 ACCEPTED MANUSCRIPT
Some studies used AR technology to support collaborative instructional processes by providing
knowledge-sharing features and the use of differentiated roles (Kamarainen et al., 2013).
Knowledge-sharing features included the sharing of pre-established text, images, 3D objects
and animations (Chen et al., 2015), and the possibility of adding new text and images to the AR
collaborative application (Chiang et al., 2014; Zimmerman et al., 2016). These can be
considered as initial attempts to use AR technology to facilitate collaborative instructional
processes. There is room for further improvement, as collaborative learning can be defined in
terms of procedural elements including group interaction, interdependency amongst group
members and individual accountability (Cuseo, 1992; Vassileva, 2008), which have not been

PT
considered in the studies reviewed.

The number of different instructional techniques used in the reviewed studies was limited to

RI
five. However, learning affordances of augmented reality on education seems to be suitable
for instructional techniques such as massive open online courses or flipped classroom. Indeed,
despite the high interest MOOCs generate, they have large number of dropouts (Gregori,

SC
Martínez, & Moyano-Fernández, 2018). Recent studies suggest the need to include active
learning support to break limitations of MOOC model. These studies point out that learning
materials should be able to ensure the continuous activity of the learners (Hew, 2018; Ollé, &
Namestovski, 2017). In this regard, AR technology could be used to facilitate students with

U
learning interactive tools that could be uses as support of MOOCs’ courses. On the otrher
AN
hand, flipped classroom instructional approach introduces changes in the use of in-class and
out-class activities. In particular, outside class time is designed for students to gain knowledge
at the remembering and understanding levels through the presentation of learning content
using taped lectures (Hwang, Lai, & Wang, 2015). However, some studies have point out that
M

students found taped lectures less engaging and more distracting than the typical classroom
lecture (Jensen, 2011). Recommendations to improve the flipped classroom approach include
the use of out-of-class learning activities to elicit students’ engagement and exploration (Lo,
D

2018). Since AR technology allows users the interactive exploration of augmented

environments, it seems interesting to explore the use of AR-technology to support learning
TE

activities for out-of-class activities in inverted classrooms.

Finally, most of the studies reviewed emphasized a description of the AR applications and how
students worked with them, rather than a justification for using them to support an
EP

instructional strategy. In fact, the studies rarely included factors such as learning goals or
characteristics of students, which may affect the selection of the instructional strategies
employed. This finding is consistent with the work of Cheung and Hew (2009) in mobile
C

learning.
AC

5.3. Measures of student outcomes

Evaluations made in quantitative studies suggest that augmented reality technology fosters
positive affective states of students, such as motivation, engagement, and attitudes toward
STEM subjects, that have proved to be effective in promoting learning benefits; these act
through the mediation of usability on learning outcomes (Choi & Baek, 2011; Kye & Kim, 2008;
Dunleavy et al., 2009). Some of the studies reviewed claim that AR helps to decrease cognitive
load or enhances spatial ability, but these do not provide conclusive results. These two
outcomes have been suggested as potential learning affordances for augmented reality (Cheng
& Tsai, 2013); hence, it would be interesting to carry out further investigation to determine
whether there is a real contribution from AR to the acquisition of the aforementioned
outcomes.

12
 ACCEPTED MANUSCRIPT
The cognitive outcome evaluated in the majority of quantitative studies is knowledge
acquisition. However, evaluations were made with ad-hoc questionnaires elaborated by
authors, which assessed low-level cognitive outcomes over brief periods of time. Although
these results are encouraging, it is expected that an interactive technology such as AR, with
proven capabilities to engage students in learning activities, can also support critical thinking,
problem-solving, collaboration, and self-directed learning (Jaramillo, 2017). On the other hand,
previous reviews have already warned of the possibility that the novelty effect is totally or
partially responsible for the successful results shown (Akçayır & Akçayır, 2017; Bacca et al.,
2014; Cheng & Tsai, 2012). Therefore, it is advisable that future works determine the

PT
effectiveness of AR in fostering high-level cognitive outcomes over longer periods of time.
Finally, the AR learning environments reviewed reveal that there is a maturity in the use of AR
to support constructivist or collaborative instructional strategies; what is missing is the

RI
provision of evaluations of the impact of AR technology on the acquisition of scientific inquiry
or collaborative specific competences.

SC
Most of the qualitative research carried out in the studies reviewed involved descriptive
accounts of how events, processes, and activities were perceived by participants. These basic
interpretative studies helped in a better understanding of learning processes by focusing on
the total picture rather than breaking it down into variables. However, only a few of these

U
studies provided new insights about how knowledge sharing, inquiry, or discussion processes
AN
were promoted by the use of AR technology using different instructional designs. Affordances
of augmented reality in STEM learning are encouraging; thus, it is important to continue an
exploration of how the learning experience occurs in different instructional designs, and to use
these new insights for the development of future AR-based STEM learning environments.
M

In terms of the types of learners selected for these research studies, only two articles focused
on gifted or special needs students. Thus, there is a need for future research to explore how
D

augmented reality learning environments can be used with more diverse populations of
students. These results are aligned with previous findings in other review studies in education
TE

(Cheng & Tsay, 2012; Zydney & Warner, 2016). Furthermore, most of the studies reviewed did
not consider learner characteristics such as gender, age difference, prior knowledge, or skills. It
would be interesting to understand the impact of AR features such as enhancement of
visualization and interactivity in affective or cognitive outcomes according to learner
EP

characteristics. An understanding of this, along with the interactivity capabilities of augmented

reality, will be useful in deploying adaptive learning interventions (Liu, McKelroy, Corliss &
Carrigan, 2017).
C

Finally, some studies report usability problems that are more closely related to the AR learning
AC

enviroment design than to the AR technology itself. Researchers should be committed to

finding new design techniques in a way that allows educators to take advantage of the main
affordances of this emergent interface and to avoid barriers to acceptance due to the design of
learning environments.

6. Conclusions
This study presents a review of the state of the art in AR as a promising technology for
supporting STEM learning; although it is not intended to be comprehensive, it provides
important findings that can be useful for instructional design and researchers.

13
 ACCEPTED MANUSCRIPT
The first research question aimed to identify the general characteristics and specific design
features of AR-based learning applications for STEM-AR studies. In this regard, three categories
of AR-based learning applications for STEM emerged from the literature: exploration,
simulation and, to a lesser extent, game-based applications. Exploration applications were
mainly used for life sciences topics in out-of-class settings, using location-based AR, whereas
simulation applications were used mainly for mathematics and physics instructional learning
environments, carried out in in-class settings using either image-based or marker-based AR.
Most of the studies reviewed reported applications which superimpose text, images and
animations to enhance the learning experience; however, these rarely stimulate senses other

PT
than sight. It would be interesting to explore the use of assets such as audio and haptic
elements to foster higher levels of immersion.

RI
The second research question was formulated to examine the instructional processes followed
by STEM-AR studies. In this regard, it was found that some studies deployed simple strategies
based on the consumption of information, others were devoted to instructional

SC
constructivistic strategies based mostly on simulations, and a small number included
collaboration between learners. Few studies provided students with assistance in carrying out
learning activities. Some of the drawbacks associated with AR technology in education are that
AR promotes distraction and increases cognitive load for students. Therefore, in addition to

U
access to information, students require assistance in selecting and interpreting information in
AN
AR-based learning environments. This review recommends that future research embed a
scaffolding mechanism to support better metacognitive processes, which might help with the
aforementioned drawbacks of this technology. Additionally, experimental support might also
be useful in those instructional processes that use inquiry-based learning. Finally, collaborative
M

instructional strategies might be enhanced by including features that support group

interaction, interdependence among group members, and individual accountability.
D

Finally, the third research question sought to determine the main learning outcomes measured
by STEM-AR studies. It was found that the studies measured mainly affective and cognitive
TE

outcomes using cross-sectional experiments. Although the reviewed studies offer a synthesis
crucial to understand affordances and the barriers of AR technology in educational settings,
few of them provide new insights about how learning experiences take place in STEM learning
environments using AR technology. There is therefore a need for researchers to diversify their
EP

measures and to include an assessment of deepened understanding that goes beyond

remembering facts and content. It may be also useful to include measures of the acquired
capacities to build hypotheses, to contrast them, and to build new knowledge from
C

interactions with the augmented environment and in pairs. It is also advisable to complement
quantitative measures with qualitative ones, in order to gain a better understanding of how AR
AC

technology can support STEM learning.

6.1. Limitations of the study

This review was limited in that it examined articles from seven databases: ACM Digital Library,
ERIC, IEEExplore, ISI Web of Science, ScienceDirect, Scopus, and Springer, from 2010 to 2017.
The articles in these databases are considered to have a high impact on the field; however, the
latest technical reports and business demonstrations of AR in STEM learning were excluded
from this review, which may limit the representation of the state of the art for these
applications. Although the number of papers included in this review was limited, the selection
process was completed using a systematic process in order to avoid bias.

14
 ACCEPTED MANUSCRIPT
References
Akçayır, M., & Akçayır, G. (2017). Advantages and challenges associated with augmented
reality for education: A systematic review of the literature. Educational Research Review, 20, 1-
11.

Akçayır, M., Akçayır, G., Pektaş, H. M., & Ocak, M. A. (2016). Augmented reality in science
laboratories: The effects of augmented reality on university students’ laboratory skills and
attitudes toward science laboratories. Computers in Human Behavior, 57, 334-342.

PT
Akdeniz, C. (2016). Instructional Process and Concepts. In Theory and Practice: Improving the
Teaching Process. (pp. 57-105). Springer Singapore.

Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing:

RI
A revision of Bloom's taxonomy of educational objectives. New York: Longman.

Azuma, R.T. (1997). A Survey of Augmented Reality. In Presence: Teleoperators and Virtual

SC
Environments, 6, pp. 355-385.

Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B. (2001). Recent
advances in augmented reality. IEEE Computer Graphics and Applications, 21(6), 34-47. IEEE

U
Computer Society.
AN
Bacca, J., Baldiris, S., Fabregat, R., & Graf, S. (2014). Augmented reality trends in education: a
systematic review of research and applications. Journal of Educational Technology & Society,
17(4), 133.
M

Billinghurst, M., Kato, H., & Poupyrev, I. (2001). The MagicBook: a transitional AR
interface. Computers & Graphics, 25(5), 745-753.
D

Bruner, J. S. (1961). The act of discovery. Harvard Educational Review, 31, 21-32.

Bujak, K. R., Radu, I., Catrambone, R., Macintyre, B., Zheng, R., & Golubski, G. (2013). A
TE

psychological perspective on augmented reality in the mathematics classroom. Computers &

Education, 68, 536-544.

Bursztyn, N., Walker, A., Shelton, B., & Pederson, J. (2017). Increasing undergraduate interest
EP

to learn geoscience with GPS-based augmented reality field trips on students’ own
smartphones. GSA Today, 27(5), 4-11.
C

Cai, S., Chiang, F. K., Sun, Y., Lin, C., & Lee, J. J. (2017). Applications of augmented reality-based
natural interactive learning in magnetic field instruction. Interactive Learning
AC

Environments, 25(6), 778-791.

Calle-Bustos, A. M., Juan, M. C., García-García, I., & Abad, F. (2017). An augmented reality
game to support therapeutic education for children with diabetes. PLOS ONE, 12(9), e0184645.

Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E., & Ivkovic, M. (2010).
Augmented reality technologies, systems and applications. Multimedia Tools and Applications,
51(1), 341-377. Springer Netherlands.

Cascales-Martínez, A., Martínez-Segura, M. J., Pérez-López, D., & Contero, M. (2017). Using an
Augmented Reality Enhanced Tabletop System to Promote Learning of Mathematics: A Case
Study with Students with Special Educational Needs. Eurasia Journal of Mathematics Science
and Technology Education, 13(2), 355-380.

15
 ACCEPTED MANUSCRIPT
Cheung, W. S., & Hew, K. F. (2009). A review of research methodologies used in studies on
mobile handheld devices in K-12 and higher education settings. Australasian Journal of
Educational Technology, 25(2), 153-183.

Chen, C. H., Chou, Y. Y., & Huang, C. Y. (2016). An Augmented-Reality-Based Concept Map to
Support Mobile Learning for Science. The Asia-Pacific Education Researcher, 25(4), 567-578.

Chen, C. P., & Wang, C. H. (2015). Employing augmented-reality-embedded instruction to

disperse the imparities of individual differences in earth science learning. Journal of Science
Education and Technology, 24(6), 835-847.

PT
Cheng, K. H., & Tsai, C. C. (2013). Affordances of augmented reality in science learning:
Suggestions for future research. Journal of Science Education and Technology, 22(4), 449-462.

RI
Chiang, T. H., Yang, S. J., & Hwang, G. J. (2014). Students' online interactive patterns in
augmented reality-based inquiry activities. Computers & Education, 78, 97-108.

SC
Choi, B., & Baek, Y. (2011). Exploring factors of media characteristic influencing flow in learning
through virtual worlds, Computers & Education, 57(4), 2382-2394. Elsevier Ltd.

Clarck, L. H., & Starr, I. (1968). Secondary School Teaching Method (3rd ed.). New York: Collier-

U
McMillan Ltd.
AN
Cuendet, S., Bonnard, Q., Do-Lenh, S., & Dillenbourg, P. (2013). Designing augmented reality
for the classroom. Computers & Education, 68, 557-569.

Cuseo, J. (1992). Cooperative learning vs. small-group discussions and group projects: The
M

critical differences. Cooperative Learning and College Teaching, 2(3), 5-10.

Di Serio, Á., Ibáñez, M. B., & Delgado, C. (2013). Impact of an augmented reality system on
D

students' motivation for a visual art course. Computers & Education, 68, 586-596.

Drago, W. A., & Wagner, R. J. (2004). Vark preferred learning styles and online
TE

education. Management Research News, 27(7), 1-13.

Dunleavy, M., Dede, C., & Mitchell, R. (2009). Affordances and Limitations of Immersive
Participatory Augmented Reality Simulations for Teaching and Learning, J. Sci. Educ-Technol.,
EP

18, 7-22.

Dünser, A., Walker, L., Horner, H., & Bentall, D. (2012, November). Creating interactive physics
C

education books with augmented reality. In: Proceedings of the 24th Australian Computer-
Human Interaction Conference (pp. 107-114). ACM.
AC

Elo, S., & Kyngäs, H. (2008). The qualitative content analysis process. Journal of Advanced
Nursing, 62(1), 107-115.

Gündüz, G. F. (2016). Instructional Techniques. In: Instructional Process and Concepts in Theory
and Practice (pp. 147-232). Springer Singapore.

Gutiérrez de Ravé, E., Jiménez-Hornero, F. J., Ariza-Villaverde, A. B., & Taguas-Ruiz, J. (2016).
DiedricAR: A mobile augmented reality system designed for the ubiquitous descriptive
geometry learning. Multimedia Tools and Applications, 75(16), 9641-9663.

16
 ACCEPTED MANUSCRIPT
Henderson, S., & Feiner, S. (2011). Exploring the benefits of augmented reality documentation
for maintenance and repair. IEEE Transactions on Visualization and Computer Graphics, 17(10),
1355-1368.

Enyedy, N., Danish, J. A., & DeLiema, D. (2015). Constructing liminal blends in a collaborative
augmented-reality learning environment. International Journal of Computer-Supported
Collaborative Learning, 10(1), 7-34.

Estapa, A., & Nadolny, L. (2015). The effect of an augmented reality enhanced mathematics
lesson on student achievement and motivation. Journal of STEM Education: Innovations and

PT
Research, 16(3), 40.

Frank, J. A., & Kapila, V. (2017). Mixed-reality learning environments: Integrating mobile

RI
interfaces with laboratory test-beds. Computers & Education, 110, 88-104.

Gopalan, V., Zulkifli, A. N., Faisal Mohamed, N. F., Alwi, A., Che Mat, R., Abu Bakar, J. A., &
Saidin, A. Z. (2015). Evaluation of e-star: An enhanced science textbook using augmented

SC
reality among lower secondary school student. Jurnal Teknologi.

Gregori, P., Martínez, V., & Moyano-Fernández, J. J. (2018). Basic actions to reduce dropout

U
rates in distance learning. Evaluation and program planning, 66, 48-52-

Hew, K. F. (2018). Unpacking the Strategies of Ten Highly Rated MOOCs: Implications for
AN
Engaging Students in Large Online Courses. Teachers College Record, 120(1), n1.

Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis.
Qualitative Health Research, 15(9), 1277-1288.
M

Huang, T. C., Chen, C. C., & Chou, Y. W. (2016). Animating eco-education: To see, feel, and
discover in an augmented reality-based experiential learning environment. Computers &
D

Education, 96, 72-82.

TE

Ibáñez, M. B., Di Serio, Á., Villarán, D., & Kloos, C. D. (2014). Experimenting with
electromagnetism using augmented reality: Impact on flow student experience and
educational effectiveness. Computers & Education, 71, 1-13.
EP

Ibáñez, M. B., Di-Serio, Á., Villarán-Molina, D., & Delgado-Kloos, C. (2015). Augmented reality-
based simulators as discovery learning tools: An empirical study. IEEE Transactions on
Education, 58(3), 208-213.
C

Ibáñez, M. B., Di-Serio, Á., Villarán-Molina, D., & Delgado-Kloos, C. (2016). Support for
Augmented Reality Simulation Systems: The Effects of Scaffolding on Learning Outcomes and
AC

Behavior Patterns. IEEE Transactions on Learning Technologies, 9(1), 46-56.

Jaramillo, S. G. (2017). Horizon Report-2017 Higher Education Edition. CUADERNO ACTIVA,

9(9), 171.

Jensen, S. A. (2011). In-class versus online video lectures: Similar learning outcomes, but a
preference for in-class. Teaching of Psychology, 38(4), 298-302

Kamarainen, A. M., Metcalf, S., Grotzer, T., Browne, A., Mazzuca, D., Tutwiler, M. S., & Dede, C.
(2013). EcoMOBILE: Integrating augmented reality and probeware with environmental
education field trips. Computers & Education, 68, 545-556.

17
 ACCEPTED MANUSCRIPT
Kerawalla, L., Luckin, R., Seljeflot, S., & Woolard, A. (2006). “Making it real”: Exploring the
potential of augmented reality for teaching primary school science. Virtual Reality, 10(3-4),
163-174.

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does
not work: An analysis of the failure of constructivist, discovery, problem-based, experiential,
and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Kye, B., & Kim, Y., (2008). Investigation of the Relationships between Media Characteristics,
Presence, Flow, and Learning Effects in Augmented Reality Based Learning. International

PT
Journal for Education Media and Technology, 2(1), 4-14.

Lin, H. C. K., Chen, M. C., & Chang, C. K. (2015). Assessing the effectiveness of learning solid

RI
geometry by using an augmented reality-assisted learning system. Interactive Learning
Environments, 23(6), 799-810.

Liou, W. K., Bhagat, K. K., & Chang, C. Y. (2016). Beyond the flipped classroom: A highly

SC
interactive cloud-classroom (HIC) embedded into basic materials science courses. Journal of
Science Education and Technology, 25(3), 460-473.

U
Liu, M., McKelroy, E., Corliss, S. B., & Carrigan, J. (2017). Investigating the effect of an adaptive
learning intervention on students’ learning. Educational Technology Research and
AN
Development, 1-21.

Liu, P. H. E., & Tsai, M. K. (2013). Using augmented-reality-based mobile learning material in
EFL English composition: An exploratory case study. British Journal of Educational Technology,
M

44(1).

Lo, C. K. (2018). Grounding the flipped classroom approach in the foundations of educational
D

technology. Educational Technology Research and Development, 1-19.

Martín-Gutiérrez, J., Saorín, J. L., Contero, M., Alcañiz, M., Pérez-López, D. C., & Ortega, M.
TE

(2010). Design and validation of an augmented book for spatial abilities development in
engineering students. Computers & Graphics, 34(1), 77-91.

Moore, K. D. (2007). Classroom teaching skills. McGraw-Hill Humanities, Social Sciences &
EP

World Languages.

Ollé, J., & Namestovski, Z. (2017, September). Student Performance and Learning Experience in
C

MOOCs: The Possibilities of Interactive Activity-Based Online Learning Materials. In

International Conference on Interactive Collaborative Learning (pp. 649-653). Springer, Cham.
AC

Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., Kyza, E., Edelson, D. &
Soloway, E. (2004). A scaffolding design framework for software to support science
inquiry. The Journal of the Learning Sciences, 13(3), 337-386.

Radu, I. (2014). Augmented reality in education: A meta-review and cross-media analysis.

Personal and Ubiquitous Computing, 18(6), 1533-1543.

Rutten, N., Van Joolingen, W. R., & Van der Veen, J. T. (2012). The learning effects of computer
simulations in science education. Computers & Education, 58(1), 136-153.

18
 ACCEPTED MANUSCRIPT
Salmi, H., Thuneberg, H., & Vainikainen, M. P. (2017). Making the invisible observable by
Augmented Reality in informal science education context. International Journal of Science
Education, Part B, 7(3), 253-268.

Settlage, J. (2007). Demythologizing science teacher education: Conquering the false ideal of
open inquiry. Journal of Science Teacher Education, 18(4), 461-467.

Sommerauer, P., & Müller, O. (2014). Augmented reality in informal learning environments: A
field experiment in a mathematics exhibition. Computers & Education, 79, 59-68.

PT
Squire, K. D., & Jan, M. (2007). Mad City Mystery: Developing scientific argumentation skills
with a place-based augmented reality game on handheld computers. Journal of Science
Education and Technology, 16(1), 5-29.

RI
Tarng, W., Ou, K. L., Yu, C. S., Liou, F. L., & Liou, H. H. (2015). Development of a virtual butterfly
ecological system based on augmented reality and mobile learning technologies. Virtual
Reality, 19(3-4), 253-266.

SC
Tarng, W., Lin, Y. S., Lin, C. P., & Ou, K. L. (2016). Development of a Lunar-Phase Observation
System Based on Augmented Reality and Mobile Learning Technologies. Mobile Information

U
Systems, 2016.

Vassileva, J. (2008). Toward social learning environments. IEEE Transactions on Learning

AN
Technologies, 1(4), 199-214.

Wang, H. Y., Duh, H. B. L., Li, N., Lin, T. J., & Tsai, C. C. (2014). An investigation of university
students’ collaborative inquiry learning behaviors in an augmented reality simulation and a
M

traditional simulation. Journal of Science Education and Technology, 23(5), 682-691.

Wu, H. K., Lee, S. W. Y., Chang, H. Y., & Liang, J. C. (2013). Current status, opportunities and
D

challenges of augmented reality in education. Computers & Education, 62, 41-49.

TE

Yoon, S., Anderson, E., Lin, J., & Elinich, K. (2017). How augmented reality enables conceptual
understanding of challenging science content. Journal of Educational Technology &
Society, 20(1), 156.
EP

Zimmerman, H. T., Land, S. M., & Jung, Y. J. (2016). Using Augmented Reality to Support
Children’s Situational Interest and Science Learning During Context-Sensitive Informal Mobile
Learning. In Mobile, Ubiquitous, and Pervasive Learning (pp. 101-119). Springer International
C

Publishing.

Zydney, J. M., & Warner, Z. (2016). Mobile apps for science learning: Review of research.
AC

Computers & Education, 94, 1-17.

19
 ACCEPTED MANUSCRIPT

Acknowledgments
This work was supported in part by the Spanish project EEE CICYT (TIN2011-28308-C03-01),
RESET-UC3M: Reformulando Ecosistemas Escalables Educativos under grant no. CICYT
(TIN2014-53199-C3-1-R), eMadrid under grant no. S2013/ICE-2715.

PT
RI
U SC
AN
M
D
TE
C EP
AC
 ACCEPTED MANUSCRIPT

Inclusion criteria Exclusion criteria

(a) Published between January 2010 and August (a) Studies that mentioned the term “augmented
2017. reality” but were about “virtual reality” or “mixed
reality”.
(b) Peer-reviewed journal article. (b) Used for professional learning (e.g., teacher
education).
(c) Available in full-text. (c) Emphasized application design and
development as opposed to students’ outcomes.
(d) Empirical research.
(e) Related to science learning.

PT
(f) Described applications for augmented reality in
science education as primary component.
Table 1: Inclusion and exclusion criteria

RI
U SC
AN
M
D
TE
C EP
AC
 ACCEPTED MANUSCRIPT

Primary author Science domain Educational level Educational context No. of Duration of
(year of publication) students instruction

Akçayır, M. (2016) Physics Higher education In-class (Laboratory) 76 15 h

Bursztyn, N. (2017) Earth Sciences High school Out-of-class 874 20 min
Cai, S. (2017) Physics Middle In-class 42 1h
Cascales, A. (2017) Mathematics Primary In-class 22 9h
Chen, C.H. (2016) Life Sciences Primary In-class 71 180 min

PT
Chen, C.P. (2015) Earth Sciences Middle In-class 144 55 min
Chiang, T.H. (2014) Life Sciences Elementary In-class 57 120 min
Cuendet, S. (2013) Mathematics Technical In-class 24 90 min

RI
(Geometry)
Dünser, A. (2012) Physics Middle In-class 10 18 min
Enyedy, N. (2015) Physics Elementary In-class 43 15 weeks

SC
Estapa, A. (2015) Mathematics Higher education In-class 61 -
Frank, J.A. (2017) Physics Higher education In-class (Laboratory) 75 10 min
Gopalan, V. (2014) Multidisciplinary Middle In-class 70 -

U
Gutiérrez, E. (2016) Mathematics Higher education In-class 20 -
(Geometry)
AN
Huang, T.C. (2016) Life Sciences Middle Out-of-class 21 8h
(botanical garden)
Ibáñez, M.B. (2014) Physics High school In-class 64 40 min
Ibáñez, M.B. (2016) Physics Middle In-class 82 100 min
M

Kamarainen, A. M. (2013) Life Sciences Middle In-class 71 35 h

Lin, H.C.K. (2015) Mathematics Middle In-class 76 -
(Geometry)
D

Liou, W.K. (2016) Life Sciences Higher education Out-of-class 92 18 weeks

Martín, J. (2010) Mathematics Higher education In-class 24 9 hours
TE

(Geometry)
Salmi, H. (2017) Physics Middle Out-of-class 146 1 month
(science museum)
Sommerauer, P. (2014) Mathematics Primary, middle, Out-of-class 101 90 min
EP

high school (science museum)

Tarng, W. (2015) Life Sciences Middle Out-of-class 120 6 weeks
(campus butterfly
garden)
Tarng, W. (2016) Earth Sciences Elementary In-class 56 1 month
C

Wang, H.Y. (2014) Physics Higher education In-class 40 15 min

AC

Yoon, S. (2017) Physics Middle Out-of-class 58 11 min

(science museum)
Zimmerman, H.T. (2016) Life Sciences Elementary, Out-of-class (nature 42 7 weeks
families center)

Table 2: Background information on reviewed articles

 ACCEPTED MANUSCRIPT
Primary author Building tools AR Features Type Assets
(year of publication)
Akçayır, M. (2016) Metaio Image-based Simulator Text, video, animations
(markers)
Bursztyn, N. (2017) Native app. Location-based Game Text, images, videos
(Android, iOS)
Cai, S. (2017) Native app. (3D Image-based Simulator Animations
modeling tool 3DS (Kinect RGB
Max and graphics camera)
engine Java 3D)
Cascales, A. (2017) Native app. (TUIO Image-based Game Text
protocol) (video projector
and two cameras)

PT
Chen, C.H. (2016) Not specified Image based Exploration Text, image, 3D model
Chen, C.P. (2015) Not specified Image-based Simulator Image
(turntable, PC,
webcam)

RI
Chiang, T.H. (2014a) Native app. (iOS) Location-based Exploration Image, text, websites

Cuendet, S. (2013) Not specified Image-based Simulator Image

(markers)

SC
Dünser, A. (2012) Native app. Image-based Exploration Text, video, animations
Enyedy, N. (2015) Not specified Image-based Simulator Video
(markers)
Estapa, A. (2015) Layar Creator Location-based Exploration Video, websites, audio,

U
image
Frank, J.A. (2017) Native app. (iOS) Image-based Exploration Video, text
(markers)
AN
Gopalan, V. (2014) Not specified Image-based Exploration Text, image, video
Gutiérrez, E. (2016) Vuforia Image-based Exploration 3D model
Huang, T.C. (2016) Not specified Location-based Exploration Text, image
M

Ibáñez, M.B. (2014) Unity 3D, Vuforia Image-based Simulator Text, image, animation
Ibáñez, M.B. (2016) Unity 3D, Vuforia Image-based Simulator Text, image, animation
Kamarainen, A. M. (2013) FreshAiR, Vuforia Location-based Exploration Text, image, audio, video
D

Lin, H.C.K. (2015) Not specified Image-based Exploration 3D object

(markers)
Liou, W.K. (2016) Unity 3D, Vuforia Image-based Exploration 3D object, animation,
TE

(markers) video, audio

Martín, J. (2010) Native app. Image-based Exploration 3D object
Salmi, H. (2017) Not specified Not specified Simulator Not specified
EP

Sommerauer, P. (2014) Aurasma Image-based Exploration Video

Tarng, W. (2015) Native app. Location-based Simulator 3D model, animation
(Android)
Tarng, W. (2016) Native app. Location-based Simulator 3D model, animation
(Android)
C

Wang, H.Y. (2014) Native app. Image-based Simulator Text, 3D model, animation
(markers)
AC

Yoon, S. (2017) Native app. Image-based Simulator Animation

Zimmerman, H.T. (2016) Native app. Image-based Exploration Text, images

Table 3: General characteristics of augmented reality applications

 ACCEPTED MANUSCRIPT
Primary author (year of Instructional strategies Learner characteristics considered Instructional
publication) techniques

Akçayır, M. (2016) Discovery ICT competences Inquiry

Bursztyn, N. (2017) Discovery Student major Game
Student initial interest in the topic
Gender
Cai, S. (2017) Presentation Student’s physics achievements Observation
Cascales, A. (2017) Cooperative/Collaborative Students with special needs Game
Chen, C.H. (2016) Discovery - Concept map
Chen, C.P. (2015) Cooperative/Collaborative Learning style Inquiry
ICT competences
Chiang, T.H. (2014) Cooperative/Collaborative - Observation

PT
(field trip)
Cuendet, S. (2013) Cooperative/Collaborative - Structured
observation
Dünser, A. (2012) Presentation - Observation
Enyedy, N. (2015) Cooperative/Collaborative - Inquiry

RI
Estapa, A. (2015) Discovery - Structured
observation
Frank, J.A. (2017) Presentation - Inquiry
Gopalan, V. (2014) Presentation - Structured

SC
observation
Gutiérrez, E. (2016) Presentation - Structured
observation
Huang, T.C. (2016) Discovery - Field trip
Ibáñez, M.B. (2014) Discovery ICT competences Structured

U
observation
Ibáñez, M.B. (2016) Discovery Previous knowledge Inquiry
Kamarainen, A. M. (2013) Discovery - Role-play
AN
(field trip)
Lin, H.C.K. (2015) Presentation Low/medium/high-performing Observation
students
Liou, W.K. (2016) Presentation - Observation
Martín, J. (2010) Presentation - Structured
M

observation
Salmi, H. (2017) Discovery Low/medium/high-performing Inquiry
students
Gender
D

Sommerauer, P. (2014) Discovery - Observation

Tarng, W. (2015) Discovery - Inquiry
Tarng, W. (2016) Discovery - Observation
TE

Wang, H.Y. (2014) Cooperative/Collaborative - Inquiry

Yoon, S. (2017) Discovery - Inquiry
Zimmerman, H.T. (2016 Cooperative/Collaborative - Observation
)
Table 4: Instructional processes followed in the reviewed articles
C EP
AC
 ACCEPTED MANUSCRIPT
Primary Research Research approach Affective Cognitive outcomes Problems
author (year purpose outcomes reported
of
publication)
A C D L
Akçayır, M. X X Mixed method Satisfaction Remembering Students need to
(2016) (quasi-experimental Motivation information be trained in
study; Enjoyment technology
interpretative study) Attitude
Bursztyn, N. X X Quasi-experimental Motivation Remembering -
(2017) study information

PT
Cai, S. (2017) X Quasi-experimental Attitude Remembering Students need to
study information be trained in
technology
System instability

RI
System too slow
Lack of user-
friendly interface
Cascales, A. X X X Quasi-experimental Motivation Applying knowledge -
(2017) study

SC
Chen, C.H. X Experimental design Motivation Understanding -
(2016) Attitude
Chen, C.P. X X Mixed method Enjoyment Self-evaluate progress -
(2015) (quasi-experimental
study;

U
interpretative study)
Chiang, T.H. X X Content analysis Engagement Creating knowledge -
AN
(2014) Immersion
Cuendet, S. X Interpretative study - Understanding -
(2013)
Dünser, A. X X Quasi-experimental - Remembering -
(2012) study information
M

Enyedy, N. X X Case study Engagement Creating knowledge -

(2015)
Estapa, A. X X Quasi-experimental Motivation Remembering -
(2015) study information
D

Frank, J.A. X Mixed method Engagement Remembering Difficult to use

(2017) (experimental design; information Students do not
TE

interpretative study) read

Gopalan, V. X X Mixed method Motivation - System too slow
(2014) (quasi-experimental Engagement Interface is not
study; Enjoyment intuitive
interpretative study)
EP

Gutiérrez, E. X X Experimental study - Remembering -

(2016) information
Huang, T.C. X X Mixed method Engagement Remembering Students need to
(2016) (experimental design; Interest information be trained in
C

interpretative study) technology

Ibáñez, M.B. X X Experimental design Flow Applying knowledge -
(2014)
AC

Ibáñez, M.B. X Content analysis - Remembering Distraction

(2016) information
Kamarainen, X X Mixed method Self-efficacy Remembering Teachers need to
A. M. (2013) (experimental design; Engagement information be trained
interpretative study) Attitudes Novelty effect
Lin, H.C.K. X X X Mixed method Attitude Remembering -
(2015) (experimental design; information
interpretative study)
Liou, W.K. X Quasi-experimental - Remembering -
(2016) study information
Martín, J. X X Mixed method Satisfaction Remembering -
(2010) (quasi-experimental; information
interpretative study)
Salmi, H. X X Quasi-experimental Enjoyment Remembering -
(2017) study Interest information

Sommerauer, X Experimental design - Remembering -

 ACCEPTED MANUSCRIPT
P. (2014) information
Tarng, W. X Quasi-experimental Motivation Remembering .
(2015) X study information
Tarng, W. X X Experimental design - Understanding .
(2016)
Wang, H.Y. X Content analysis - Creating knowledge -
(2014)
Yoon, S. X Mixed method - Remembering -
(2017) (quasi-experimental information
study;
interpretative study)
Zimmerman, X X Mixed method Attitude Remembering -
H.T. (2016) (quasi-experimental information

PT
study;
ethnography)
A: Investigating the affective domain during AR learning.
C: Evaluating the influence of learner characteristics in the AR learning process.

RI
D: Designing an AR system for learning.
L: Evaluating the effects of AR learning.
Table 5. Research purposes, methods, and outcomes

U SC
AN
M
D
TE
C EP
AC
 ACCEPTED MANUSCRIPT
14
12
10

Frquency
8
6
4
2
0

PT
Research purpose

RI
A: Investigating the affective domain during AR learning.
C: Evaluating the influence of learner characteristics in the AR learning process.

SC
D: Designing an AR system for learning.
L: Evaluating the effects of AR learning.
Figure 1. Research purpose frequency

U
AN
8
7
M

6
Frequency

5
4
3
D

2
1
TE

0
EP

Affective outcome

Figure 2. Frequency of affective outcomes

C
AC
 ACCEPTED MANUSCRIPT
Research highlights

1. A review of the literature on AR technology to support STEM education is shown.

2. Most AR applications use trigger mechanisms to aid the consumption of information.
3. Some AR applications allow the exploration of digital assets to build knowledge.
4. AR STEM education research should focus on science skill-based outcomes.
5. AR STEM applications should offer metacognitive and experimental support.

PT
RI
U SC
AN
M
D
TE
C EP
AC