http://journal.kamall.or.kr/wp-content/uploads/2022/6/Kim_25_2_02.pdf http://www.kamall.or.

kr

Multimedia-Assisted Language Learning

25(2) 44-63

A Comparative Study on AI-directed and Human-directed MALL:

The Effects on L2 Comprehension Skills

Na-Young Kim (Sehan University)

Kim, Na-Young. (2022). A comparative study on AI-directed and human-directed MALL: The effects on
L2 comprehension skills. Multimedia-Assisted Language Learning, 25(2), 44-63.

This study aims to determine whether MALL directed by artificial intelligence positively affects second
language (L2) learning. By comparing the effects of human-directed MALL, this study particularly explores
the effectiveness of AI-directed MALL in increasing L2 comprehension skills. The participants were 98
freshmen who were taking a TOEIC course at a university in Korea. For the treatment, they were divided
into the AI-directed MALL group (n = 51) and the human-directed MALL group (n = 47) at random.
According to their treatment conditions, the participants took the online learning course with their mobile
phones directed by either the AI or the human teacher. At the beginning and the end of the study, pre and
post-TOEIC tests were administered. Paired-samples t-tests as well as independent samples t-tests were
run to compare the mean scores. The findings reveal that all participants in both two groups increased their
test scores at a significance level. However, there was a statistically significant difference in mean scores
between the two groups. The AI-directed MALL group outperformed the human-directed MALL group.
Based on these findings, implications are discussed.

Key words artificial intelligence, mobile-assisted language learning, second language learning, TOEIC,
comprehension skills
doi: 10. 15702/mall.2022.25.2.44

I. INTRODUCTION
During COVID-19, there has been a significant shift from traditional face-to-face learning
programs to emergency online learning programs in many educational institutions to reduce
disruptions in tertiary classrooms (Darsih et al., 2021). In fact, in tertiary education settings, an
increasing amount of instruction has already been delivered online (Murray et al., 2015).
However, due to the pandemic, the transition to online learning has been forced, exposing
significant gaps in education systems (Maskey, 2020). Although there have been significant
technological advancements in the past years, many schools have been slower to adapt. Many
teachers still lack basic resources like textbooks and internet access. These small cracks have
turned into wide gaps and become more and more difficult to close.
Meanwhile, the integration of artificial intelligence (AI) in class has successfully transformed
learning environments (Maskey, 2020). The limitations combined with the lack of teaching
resources such as textbooks and internet access have been solved with this technology. The
challenges regarding the class time limit and imbalanced student-teacher ratio have also been
overcome through AI. By acting as a personal tutor, AI has provided students with individual
support and feedback when teachers lack the bandwidth. Aside from the personalized learning
plan, AI involvement in class has allowed students to access an enormous amount of online
information. According to Istrate (2019), machines equipped with AI have been designed,
programmed, and developed for self-learning.
The current classroom settings have limitations combined with an imbalanced
student-teacher ratio. Class time is also limited. Furthermore, teachers often lack basic
resources such as textbooks and internet access. However, AI systems playing a role as personal
tutors can be used in areas where teachers are limited (Jones et al., 2018). They can process
information more quickly in larger quantities than humans can by leveraging algorithms that
learn from the accumulated data. Using these algorithms, the AI systems provide suggestions
relevant to the teacher when determining subjects their students have excelled in or struggled
so they can be redesigned effectively. They also provide customized feedback and support
when the teacher does not have the bandwidth. By doing so, AI can solve the student-teacher
ratio problem. Considering that learning environments consist of these social and economic
factors (Maskey, 2020), the adoption of AI eliminates the socioeconomic discrepancies in
schools.
With the development of mobile technology, it has been requested to apply AI in mobile
learning. According to Liu et al. (2010), the adoption of AI technology can improve mobile
learning. In particular, Huang et al. (2022) claimed that AI should be implemented in
mobile-assisted language learning (MALL) environments to enhance language learning.
Specifically, the scholars insisted that the integration of AI technology with mobile applications
can increase language learners’ listening, speaking, reading, and writing skills as well as their
vocabulary, grammar, meaning negotiation, and critical thinking skills (Huang et al., 2022).
Regarding this issue, empirical studies have been conducted to explore the effectiveness of
MALL directed by AI. The previous L2 studies have discovered that AI-directed MALL

Na-Young Kim 45
applications can enhance students’ learning in different aspects and skills of a language (Yang &
Kim, 2021). In particular, it has been reported that these AI-directed applications have played
an important role in increasing students’ official language test scores including TOEIC (Jones et
al., 2018; Kim et al., 2020; Kusumaningrum & Pertiwi, 2021; Loh et al., 2021; Obari et al.,
2020).
While there have been positive views on MALL incorporated with AI technology, there has
been debate over the effectiveness of MALL. With the absence of full exploitation of its
portability and mobility, Kukulska-Hulme and Shield (2008) emphasized the necessity of
empirical evidence to prove the efficacy of mobile technology in language learning. Kim (2018)
also emphasized the necessity of examining its effectiveness through various mobile
applications involving different features. Considering the recent development of AI technology,
empirical studies are needed to contribute to the literature regarding AI-directed mobile
applications. However, Yang and Kim (2021) pointed out that there is still a dearth of empirical
evidence to prove the effectiveness of AI applications in L2 learning. Although Liu et al. (2010)
noted that it is necessary to incorporate AI technology with mobile learning, few research
studies on MALL have been conducted regarding the use of AI. Hong et al. (2016) also noted
that there is a lack of AI studies about L2 comprehension skills.
Taken all together, the present study aims to determine whether MALL directed by AI
technology positively affects second language learning. By comparing the effects of
human-directed MALL, this study examines the effectiveness of AI-directed MALL regarding
L2 comprehension skills. In particular, given the importance of the TOEIC test in Korea (Kim,
2018), this study focuses on the effectiveness of the use of AI in MALL on Korean students’
TOEIC learning. The purpose of this study is specified in the following research questions:

a. What are the effects of AI-directed MALL on L2 comprehension skills? To what extent
does AI-directed MALL affect Korean students’ TOEIC learning?
b. Are there any differential effects of human-directed MALL and AI-directed MALL on
Korean students’ TOEIC learning? What are some different effects of the two types of
MALL on L2 comprehension skills?

46 A Comparative Study on AI-directed and Human-directed MALL: The Effects on L2 Comprehension Skills
II. LITERATURE REVIEW
1. MALL and L2 learning

Mobile learning is not only about delivering digital learning content but also paves the way
for learning by providing digitally enhanced experiences across different contexts. According to
Chinnery (2006), mobile devices have also been put into practice in language classrooms for
various educational purposes. In particular, as the use of mobile technology has been
considered beneficial for second language learning, MALL has become popular in L2
classrooms (Kim, 2018). Researchers have explored how to use them to support English as a
second language learning. Regarding different aspects and skills of a language, previous studies
have explored the effectiveness of MALL in L2 settings.
In general, MALL studies have revealed the positive impacts of using mobile devices on
second language learning. Many of them have indicated its positive impacts on L2 vocabulary
(Stockwell, 2010). For example, after exploring mobile phone use for language learning, Brown
(2001) found that vocabulary lessons, quizzes, and translations were commonly available and
they were beneficial in improving English language learning. Language students have also found
the use of mobile devices most useful for learning L2 vocabulary, grammar, and translation
(Steel, 2012).
In China, Xu (2020) conducted action research at a Chinese university to investigate the
effects of MALL on English listening and speaking. In this study, the participants’ attitudes
toward listening and speaking practices on mobile devices became more positive. They also
made significant progress in English listening. However, there was no significant change in
speaking. It was explained by the previous study (Ahn & Lee, 2016), suggesting that speaking is
a challenging skill, and therefore, it requires longer periods of time for positive changes to
occur.
In Korea, however, different results were obtained regarding L2 speaking. Kim et al. (2019)
investigated the effects of students’ interaction through mobile devices on English speaking and
writing performance and they found that mobile interaction improved both the students’
speaking and writing performance. After comparing the impacts of different types of mobile
interaction – spoken and written – the researchers concluded that MALL helped students’
speaking and writing, and its positive effects on L2 learning did not differ by the types of
mobile interaction.
Regarding English listening and reading skills, Kimura et al. (2011) explored the

Na-Young Kim 47
effectiveness of MALL in Japan. On an online English learning website, the researchers
uploaded TOEIC questions that consist of multiple-choice questions with answer keys. For the
treatment, a set of questions was sent to students when they accessed the service. If a student
solved the questions, their answers were sent and scored through the Internet. Their results
returned immediately. They were also able to get feedback. As a result, the students benefitted
from mobile TOEIC programs by increasing their L2 listening and reading scores.
In spite of all these positive advancements, it seems that the debate on the effectiveness of
MALL in L2 learning will continue (Kim, 2018). MALL is particularly subject to disturbances of
many kinds. Despite the widespread use of mobile technology and its sophisticated functions,
MALL has obvious disadvantages such as small screen size and limited presentation of graphics
(Albers & Kim, 2002). Mobile devices also have restricted memory. In addition, it depends on
networks that may not provide very high transmission capacity. Some carriers even have a lack
of network connectivity standards all the time. Kimura et al. (2011) added that there can be
geographical limitations. In this vein, Kukulska-Hulme and Shield (2008) emphasized the
absence of full exploitation of its portability and mobility. They cast doubt on the sufficiency of
empirical evidence to prove the efficacy of mobile technology in L2 learning.
It has also been assumed that mobile technology cannot justify its expense. Salaberry (2001)
pointed out that it is unclear that technology has offered the same pedagogical advantages as
conventional language classroom instruction. Fallahkhair et al. (2007) noted that the same
learning outcomes can also be attained through textbooks, desktop computers, or even TV or
radio programs. Kukulska-Hulme and Shield (2008) also pointed out that MALL supports
one-way teacher-to-student communication as mobile devices are mostly used to deliver
content rather than encouraging students to communicate with their teacher or peers.
Although the previous scholars (e.g., Kimura et al., 2011) found an improvement in the
students’ TOEIC scores, there were several limitations regarding the mobile-based language
learning program. Since it was a “ready-made” lesson programmed beforehand, the immediate
feedback on learning management system (LMS) was available but the learning contents were
not customized for each student. The students became proactive at the end of the study
regarding the conduct and conditions of the program. They wished that they could have
options to choose the access and delivery time, frequency, and quantity of practice materials.
Taking all into account, more research is necessary to contribute to the literature, eliciting
empirical evidence for its practical use and effectiveness in L2 classrooms. It should be
discovered more considering the benefits of MALL to the education system (Okumuş Dağdeler
et al., 2020). Kim (2018) also emphasized the necessity of examining its effectiveness through

48 A Comparative Study on AI-directed and Human-directed MALL: The Effects on L2 Comprehension Skills
various applications involving different features. Considering the recent development of AI
technology, empirical studies are needed to contribute to the literature regarding AI
applications in MALL.

2. AI in MALL

There are some examples of AI applications relevant to language learning. Dragon

transcription software was one of the first AI tutoring applications. It was integrated with
Windows in the 1990s. With speech recognition, Dragon software transcribed text from speech.
According to Campbell and Geertsema (2017), this dictation tool was beneficial for second
language learners, particularly related to their pronunciation. Using natural language processing
(NLP) and automatic speech recognition (ASR) techniques, Duolingo has also provided foreign
language training. Beyond speech-to-text transcription, this downloadable software and the
online system recognizes language errors and helps users to correct them. By doing so, it has
the potential for the emergence of learner autonomy and the promotion of self-regulation
(Kannan & Munday, 2018). The European Commission also designed a research project with
an AI robot named L2-TOR. As a social robot, it naturally interacts with its users, combining
the socio-affective and cognitive aspects of learning (Kanero et al., 2018).
As mobile technology advances, AI comes to be more and more popular. Liu et al. (2010)
suggested that it is important to integrate AI technology with mobile learning. According to
them, the use of AI applications improves mobile learning environments. Greene (2017)
introduced some mobile applications that offer language learning services solely through AI.
One of them is Glossika, a downloadable mobile AI application. It has provided language
lessons to help students to speak a language better and faster by using smart technology,
structured content, and adaptive learning algorithm. Glossika tries to tailor learning material to
the students’ specific levels and needs. The students can choose the topics that most matter to
them and study at a comfortable pace using their mobile phones. By offering a unique learning
method guided by AI, this application helps the students to focus on the most ideal lessons,
saving their learning time (Sumo & Bah, 2021).
Another mobile AI application found to be helpful for students learning English is ELSA
Speak: English Accent Coach (Darsih et al., 2021). Like Glossika, it can also be downloaded on
mobile phones via Play or App store. ELSA is designed to help students to speak English in a
simple, confident, and fluent manner. The features of this application are intended to allow the
students to correctly say words from vowel and consonant to word stress. According to Jones

Na-Young Kim 49
et al. (2018), ELSA has indeed helped the students to correct mistakes in their pronunciation.
In Darsih et al.’s (2021) study, the participants were not able to reach their speaking teacher in
person due to the pandemic and practiced their speech with this application to continue their
English learning at home. The majority of the participants in their study also agreed that ELSA
Speak App is useful to improve their speaking ability, particularly regarding pronunciation.
In the field of virtual assistants, Google Assistant, Amazon Alexa, and Apple Siri are the
most recent developments. These AI applications are also downloadable on mobile devices.
According to Jones et al. (2018), they have also helped language learning more easily and more
quickly. Scholars in L2 settings have investigated the use of these AI systems on mobile phones
to enhance the students’ English language proficiency. For example, Google Home Mini and
Amazon Alexa have been introduced to students of English in Japan to increase their TOEIC
test scores. A number of related studies have confirmed that the two AI applications were
beneficial for improving the Japanese EFL students’ TOEIC level (Obari et al., 2020).
In other L2 settings, AI technology has also been adopted for TOEIC test preparation. In
Indonesia, for example, Kusumaningrum and Pertiwi (2021) adopted an AI-directed mobile
program in class. The authors introduced an AI application, Listening for the TOEIC Test, and
investigated its effects on Indonesian L2 students’ TOEIC scores. As expected, the students
showed significant improvement in their test scores. With direct and accurate feedback, the
application provided the students with autonomous and flexible learning environments. After
learning important strategies and tips for the test through AI, the students excelled in the test.
Similarly in Korea, Santa TOEIC has been introduced to the students preparing for the
TOEIC test (Jones et al., 2018). According to Loh et al. (2021), Santa TOEIC provided the
most efficient learning content to those preparing for the TOEIC test. As a 1:1 TOEIC tutor,
this AI-directed tutoring service helped the students of English to prepare for the test. It
diagnosed the students’ current level, provided expected TOEIC scores, and suggested
level-appropriate learning items based on the test scores. As a personalized AI tutor, Santa gave
individual feedback, tailored lecture, and customized explanation to the students. Important test
strategies and skills were also provided. As Jones et al. (2018) stated, students were able to
increase their TOEIC scores by more than 100 points within 5 to 7 hours. In this regard, the
AI-directed program can be of great help to Korean students who need good TOEIC scores to
prove their employability (Kim, 2018).
As can be seen above, there have been positive views on MALL integrated with AI
technology in L2 settings. Nevertheless, there is still a need for empirical evidence to prove the
impacts of AI mobile applications on English language learning (Yang & Kim, 2021). While

50 A Comparative Study on AI-directed and Human-directed MALL: The Effects on L2 Comprehension Skills
scholars have emphasized the necessity of integrating AI technology with mobile learning (Liu
et al., 2010), few experimental studies have been carried out regarding AI-directed MALL. In
particular, Hong et al. (2016) noted that there is a lack of AI studies about L2 listening and
reading comprehension skills. Thus, future studies are requested to contribute to the literature
on the use of AI mobile applications. The present study, therefore, aims to determine whether
MALL directed by AI technology positively affects L2 comprehension.

III. METHOD
1. Participants

A total of 98 freshmen who were taking a TOEIC course in two classes were chosen to be
the subjects of the current study. There were 46 male and 52 female students. The participants’
age ranged from 20 to 25. On average, they were 21.31 years old. The participants majored in a
variety of areas. The highest percentage (41.25%) majored in aeronautics followed by police
safety (29.63%), and arts (23.12%).
For the study, the participants were randomly divided into two experimental groups:
AI-directed MALL and human-directed MALL groups. There were 51 students in the
AI-directed MALL group while 47 in the human-directed MALL group. To make the study
more consistent, the two types of MALL groups were assigned to the same teacher. The
students participated in the TOEIC course with their mobile phones directed by either AI or a
human teacher.
To check the participants’ proficiency levels, a TOEIC pre-test was administered to all the
participants before the treatment. Their proficiency level was determined according to the
results of the test. The participants of the study scored 350.15 on average for the pre-test.
According to Mason and Krashen (2014), it was assumed that the participants of the study were
at the elementary proficiency level. That is, the students had functional but limited English
proficiency, being able to keep very simple face-to-face conversations on familiar topics.

2. Two Types of MALL Programs

For the study, the participants were assigned to the AI-directed MALL group and the
human-directed MALL group at random. For the AI-directed MALL group, an AI-based online

Na-Young Kim 51
learning program was used. As an intelligent tutoring system (ITS), the program was developed
by Janua Educational Services (JES). This AI system interacted with the students and tested
their knowledge while providing some information. It also taught each student individually
based on their prior knowledge and priorities.
For the treatment, the students engaged in a diagnostic test, and following the test results, a
differentiated curriculum was provided. The students’ learning progress was different from
student to student. Furthermore, the students’ data were analyzed individually during the
process. Based on their traits and patterns, the different number of tasks and the different
amounts of learning were provided. In addition, activities were automatically adapted according
to the students’ needs.

[FIGURE 1] AI-directed MALL Program

The AI-directed TOEIC learning program used for the present study automatically learned
about the students and told them what their weaknesses were. Through the personalized
course, the program helped the students to improve on their weaknesses in the actual exams. It
seemed to be efficient to focus on the parts that the students were weak at, rather than those
they had already known well enough. From the weakness analysis report, the students were able
to learn and practice their vulnerable TOEIC questions. By intensively studying the areas the

52 A Comparative Study on AI-directed and Human-directed MALL: The Effects on L2 Comprehension Skills
students were weak at, they appeared to spend less time but learn more through this
technology. For the first two to three weeks, the researcher provided technical support and
guidance for those with the difficulty with using this AI program.
For the human-directed MALL group, a human teacher or a course instructor uploaded
TOEIC listening and reading multiple-choice questions on the school website. Answers and
explanations were also provided along with the questions. They were adopted from a textbook,
Longman Preparation Series for the TOEIC Test (Lougheed, 2017). There were a variety of
practice items reflecting the test format and content. There were also drill-and-practice activities
for grammar and vocabulary related to the TOEIC test. According to the lesson plan, the
teacher selected and controlled the learning materials when uploading them to the site.
Once the students accessed the website through mobile phones, the same questions were
delivered to them. When the students solved the questions, their answers were automatically
scored. The students’ learning progress and learning timelines were monitored and tracked
through LMS. Based on this, the students received instant feedback regarding their TOEIC
learning. Even though the students accessed the same learning materials, the amount of
learning varied; some just checked vocabulary words for meaning while others completed all
the questions uploaded.

3. Data Collection and Procedure

This study intended to see if AI-directed MALL makes a significant contribution to L2

students’ comprehension skills. For the study, 98 freshmen students who were taking a
mandatory TOEIC course were recruited. The class was held every week for two hours. The
students were from two different classes. For the treatment, they were divided into the
AI-directed MALL group (n = 51) and the human-directed MALL group (n = 47) at random.
The two types of MALL groups were assigned to the same teacher. However, depending on
their treatment conditions, the students took the TOEIC course with their mobile phones
directed by either the AI or the human teacher.
For the present study, the TOEIC test was used to explore whether the AI-directed online
learning program was effective in increasing TOEIC scores. The test was a multiple-choice test
and entirely in English. It was composed of two timed sections, listening comprehension and
reading comprehension, with 100 questions each. 45 minutes were provided for the listening
comprehension section whereas 75 minutes were for the reading comprehension section. In
total, the students were required to solve 200 questions within 120 minutes. All the listening

Na-Young Kim 53
and the reading comprehension questions used for the TOEIC test of the current study were
adopted from one of the popular TOEIC preparation books, Taking the TOEIC 1: Skill and
Strategies (Anderson et al., 2017).
At the beginning and the end of the study, pre and post-TOEIC tests were administered. As
the test was a paper-and-pencil test, all participants were requested to take the tests offline in
the classroom. According to the number of correct answers, the students’ test scores were
determined. They were then converted to a scaled score. The scaled score range was from 5 to
495 for the listening and reading sections, respectively. The total score ranged from 10 to 990.

4. Data Analysis

The empirical data collected for the current study was pre and post-TOEIC test scores. To
explore whether the AI-directed MALL program increased L2 students’ comprehension skills,
the tests were administered before and after the treatment. The collected data were computed
and analyzed with SPSS version 21.0 program. To compare the pre and post-test results,
descriptive statistics were calculated. To check whether the mean score changes from the
pre-test to the post-test were statistically significant, performed were paired-samples t-tests.
Independent samples t-tests were run to compare the mean differences between groups. P value
was set at .05.

IV. RESULTS AND DISCUSSION

1. Effects of Human-directed and AI-directed MALL on TOEIC Learning

The purpose of the study was to investigate the impacts of AI-directed MALL on L2
learning by comparing it with those of human-directed MALL. The first research question was
whether and to what extent the two types of MALL affected students’ TOEIC scores. To
compare the pre and post-test scores of the students, descriptive statistics were administered.

[TABLE 1] Descriptive Statistics for AI-directed MALL and Human-directed MALL

Pre-test Post-test
Group M SD M SD
AI-directed MALL 363.92 105.51 417.16 114.36
Human-directed MALL 336.38 85.77 363.30 119.88

54 A Comparative Study on AI-directed and Human-directed MALL: The Effects on L2 Comprehension Skills
 The descriptive statistics summarized in Table 1 represent that there were positive changes
in TOEIC scores between the pre and post-tests. It depicts that the post-test mean scores
obtained by the participants in the current study are higher than their pre-test mean scores,
which indicates that the treatment was successful. To be more specific, the descriptive statistics
show that the participants in the AI-directed MALL group increased their test scores from
363.92 points (SD = 105.51) on the pre-test to 417.16 points (SD = 114.36) on the post-test.
In terms of the human-directed MALL group, the participants also increased their
comprehension test scores from the pre-test to the post-test. As displayed in Table 1 above, the
mean score on the pre-test was 336.38 (SD = 85.77) while that on the post-test was 363.30 (SD
= 119.88).
In the next step, paired samples t-tests were run to determine whether the differences
between the pre- and post-test mean scores were statistically significant for both groups. Table
2 shows the result of the t-tests for the AI-directed MALL group and the human-directed
group, respectively.

[TABLE 2] The Results of the Paired-Samples t-Tests for the Two Groups
Mean Std. Std. t df Sig.
Group
Deviation Error Mean (2-tailed)
AI-directed MALL -53.24 68.45 9.59 -5.554 50 .000
Human-directed MALL -26.91 86.15 12.57 -2.142 46 .038

By comparing the pre and post-test scores in the two experimental groups, it was found that
the mean differences were statistically significant (p < .05). In other words, all participants in
the current study significantly increased their TOEIC scores after the treatment. To be specific,
the result of the paired-samples t-test of the AI-directed MALL group indicates that the
participants in this group enhanced their L2 comprehension skills at a significance level (t =
-5.554, p = .000). Accordingly, the score changes between the pre- and post-test scores of the
human-directed group also appeared to be statistically significant (p < .05). The result of the
t-test for this group represents that the participants in this group improved their TOEIC skills
at a significance level (t = -2.142, p = .038).
Focusing on different aspects and skills of a language, there have been many studies
investigating the effectiveness of MALL (Brown, 2001; Okumuş Dağdeler et al., 2020; Steel,
2012; Stockwell, 2010). Particularly, scholars have shown its positive impacts on L2 learning. To

Na-Young Kim 55
be specific, they have proved that MALL can positively affect L2 proficiency in the four major
skills including speaking, writing, reading, and listening (Kim, 2018; Kim et al., 2019;
Kukulska-Hulme & Shield, 2008; Xu, 2020). The current study is in line with these previous
MALL studies, suggesting that L2 students can benefit from MALL in listening and reading
comprehension skills.
To be more specific, this study confirmed that L2 students can enhance their TOEIC skills
through mobile phones. The findings of the study corroborate the previous research indicating
the positive effects of MALL on L2 comprehension skills. In Japan, Kimura et al. (2011)
investigated the use of mobile technology for improving L2 students’ listening and reading
comprehension skills. The scholars uploaded TOEIC sample questions online and asked the
participants to download and solve the questions using their own mobile phones. In accordance
with the current study, their findings also proved that the students benefitted from mobile
TOEIC programs by increasing their comprehension test scores.
It is more important to note that the present study discovered not only human-directed
MALL but also AI-directed MALL played a positive role in increasing the participants’ TOEIC
scores. The findings of the study support the previous studies suggesting that L2 students can
benefit from AI technology in terms of their comprehension skills. For example,
Kusumaningrum and Pertiwi (2021) applied AI technology to develop Indonesian students’
TOEIC skills and found that the AI-directed learning program helped the students learn the
language skills and test strategies to succeed on TOEIC. The researchers emphasized the
advantages of employing AI with its flexibility, accuracy, and direct feedback. The authors
concluded that AI helped to improve the students’ English proficiency. Their research findings
are in accordance with the current study in that AI-directed programs also played a beneficial
role in TOEIC learning.

2. Differences in the Effects of Human-directed and AI-directed MALL

Because the participants engaged in two different experimental conditions, it was necessary
to evaluate how the two types of treatment differently affected L2 comprehension skills. For
the purpose of examining which treatment type more improved students’ TOEIC scores in
comparison to the other, independent samples t-tests were run for both the pre- and post-tests.
Table 3 below presents the results of this analysis.

56 A Comparative Study on AI-directed and Human-directed MALL: The Effects on L2 Comprehension Skills
 [TABLE 3] The Results of the Independent-Samples t-Tests
Levene’s Test for t df Sig. Mean Std.
Test Equality of Variances (2-tailed) Difference Error Mean
F Sig.
Pre-test 3.853 .053 1.411 96 .162 27.54 19.52
Post-test 0.001 .976 2.276 96 .025 53.86 23.67

Table 3 above displays that the mean scores of the two groups were close at the beginning
of the study, with a mean difference of 27.54 for the pre-test. To examine whether the
difference between the means was statistically significant, an independent-samples t-test was
run, but the difference was turned out to be statistically insignificant (t = 1.411, p = .162). In
other words, the participants in the AI-directed MALL and the human-directed MALL groups
were similar in their TOEIC scores before the treatment. Based on the pre-test results, they
were considered homogeneous at the outset of the study.
On the other hand, the mean scores of the two groups seemed not that close at the end of
the study. As can be shown in Table 3, the mean difference between the AI-directed MALL
group and the human-directed MALL group was 53.86 on the post-test. The mean score of the
AI-directed MALL group was higher than that of the human-directed MALL group (See Table
1). In order to examine whether this mean difference was statistically significant, another
independent-samples t-test was run. The results of the t-test indicate that there was a significant
difference in mean scores between the two groups (t = 2.276, p = .025). It was found that the
participants in the AI-directed MALL group more benefitted than those in the human-directed
MALL group. In other words, MALL directed by the AI was more beneficial than that by the
human teacher.
For the human-directed MALL group of the current study, TOEIC multiple-choice
questions with answers were uploaded on the website. Through the school LMS, the students’
learning progress was monitored. Regarding their TOEIC learning, the students received
instant feedback. The MALL programs also included drill-and-practice programs for grammar
and vocabulary. As Brown (2001) reported, the commonly used in MALL were vocabulary
lessons, quizzes, or grammar translations. After conducting a similar study on the use of mobile
technologies for L2 learning, previous scholars (Kimura et al., 2011) reported that students still
benefitted from this kind of MALL program regarding English comprehension skills. In their
study, the scholars uploaded TOEIC questions and asked the students to solve the questions
using their mobile phones. It was found that the MALL programs played a beneficial role in

Na-Young Kim 57
increasing TOEIC scores.
However, it is important to note that a one-size-fits-all approach cannot be effective
considering the wide variety of learner interests and backgrounds. Although Kimura et al.
(2011) found an improvement in the students’ TOEIC scores after engaging in the MALL
programs, there were also limitations regarding the mobile-based language learning program.
The scholars criticized that the mobile programs only provided the participants with a
ready-made lesson, and the learning contents were not customized for each participant. The
participants also requested to be allowed to select the frequency, delivery time, and quantity of
TOEIC practice materials.
From this point of view, the integration of AI in MALL seems necessary. As a personalized
tutor, AI-directed programs can give individual feedback, tailored lecture, and customized
explanation to each student after diagnosing the student’s current level (Loh et al., 2021).
Depending on the students’ conditions, the number of tasks and the amount of learning are
differently provided. Since the AI-directed programs provide level-appropriate learning items
based on the student’s test scores, the students can increase their TOEIC scores within less
time. By playing a role as a 1:1 individualized TOEIC tutor, the program directed by AI can
help the students to prepare for the test more effectively (Jones et al., 2018).
For the AI-directed MALL group in the current study, the participants engaged in TOEIC
learning via an AI program. After taking a diagnostic test, they were given a differentiated
curriculum according to their test results. The students’ data were individually analyzed during
their learning process with AI. In particular, this program helped the students to improve on
their weaknesses in TOEIC through the personalized course. They studied and practiced their
vulnerable TOEIC questions. This AI-directed program individually provided the students with
learning materials based on their prior knowledge and priorities. The number of tasks and the
amount of learning differed based on their traits and patterns. Consequently, the students in
this group were able to increase their test scores more than their counterparts.
Taken all together, it can be said that the present study provided empirical evidence to
prove the educational value of an AI-directed program in L2 learning. With AI, the entire
learning time was devoted to the personalized one for each student. By comparing with the
human-directed program which only provided the students with a ready-made lesson not
customized for each student, the current study confirmed the superior effects of the AI
program on comprehension skills. In this regard, the current study guides the use of AI
technology and mobile learning.

58 A Comparative Study on AI-directed and Human-directed MALL: The Effects on L2 Comprehension Skills
V. CONCLUSION
Kim (2018) claimed that the literature lacks sufficient research on the effects of MALL on
L2 learning. She emphasized the necessity of examining its effectiveness through various
mobile applications involving different features. Considering the development of AI
technology, more empirical studies are needed to contribute to the literature regarding AI
mobile applications (Yang & Kim, 2021). In light of this, the present study aimed to find an
effective way of integrating MALL with AI technology positively affects L2 learning. In
particular, Hong et al. (2016) noted that there is a lack of AI studies about L2 comprehension
skills. Given the importance of the TOEIC test in Korea (Kim, 2018), this study focuses on the
effectiveness of the use of AI mobile applications on TOEIC skills.
Major findings are as follows: First, the results of the paired-samples t-tests indicate that all
participants in both the two groups – AI-directed and human-directed groups – increased
their TOEIC scores at a significance level. It was found that the mean differences between the
pre and post-test scores were statistically significant for both groups. More importantly, the
result of the independent t-test indicates that there was a statistically significant difference in
mean scores between the two groups. The participants in the AI-directed MALL group
increased their TOEIC scores more than those in the human-directed MALL group did. It was
found that MALL directed by AI was more effective than that by a human. This confirmed the
superior impacts of AI integration in mobile-assisted language learning.
According to Okumuş Dağdeler et al. (2020), it should be discovered more given the
benefits of MALL to the education system. Previous scholars have insisted that more research
is necessary to contribute to the literature, eliciting empirical evidence for its practical use and
effectiveness in L2 classrooms (Kim et al., 2019). In this regard, the current study provides
insight into the use of mobile applications in language learning. In particular, this study revealed
that L2 students can benefit from MALL whether it is AI-directed or human-directed. What’s
more, the current study showed the superiority of the AI-directed MALL program over the
human-directed MALL one.
Yang and Kim (2021) noted that there is a need for experimental studies to confirm the
impacts of AI on English learning. Previous scholars have emphasized the necessity of
employing AI technology to enhance mobile learning (Liu et al., 2010), but little work has been
done regarding AI-directed MALL. In light of this, the findings of the current study contribute
to the literature on the integration of AI in MALL. The present study provided empirical
evidence to confirm the educational value of an AI-directed program in TOEIC learning. With

Na-Young Kim 59
the advantages of customized learning of AI, this study provides guidance for the use of AI
technology and mobile learning.
The study, however, has several limitations. First, while the TOEIC test consists of two
sections, listening and reading, the current study did not investigate the effects of AI-directed
MALL on listening and reading proficiency separately. Although the current approaches to
teaching and learning a language strive to integrate the four language skills, the separation of the
four skills – speaking, listening, reading, and writing – has lied at the core of linguistic
research and language testing (Hinkel, 2010). Hong et al. (2016) also noted that there is a lack of
AI studies about listening and reading skills. Given this, it might be worth examining the
impacts of AI programs on language skills separately for future study.
Furthermore, human factors that might affect language learning should also be investigated.
According to Liu (2011), gender and major are the main factors that influence
technology-integrated learning. Gender and major are seen as major factors that influence
language learning (Cha et al., 2022). In this light, it seems essential to consider the effects of
these demographic data when using technology in language learning settings. Considering this,
it seems necessary to consider human factors for future study.
Last but not least, the current study did not adopt qualitative methods of analysis. To find
out the more detailed differences between the effects of AI-directed MALL and
human-directed MALL on L2 comprehension skills, qualitative data can be collected through
individual or focus group interviews with students and teachers. The interview results might
provide new insights into the MALL programs from both students’ and teachers’ points of
view.

REFERENCES
Ahn, T. Y., & Lee, S. (2016). User experience of a mobile speaking application with automatic speech
recognition for EFL learning. British Journal of Educational Technology, 47(4), 778-786.
Albers, M., & Kim, L. (2002). Information design for the small-screen interface: An overview of web
design issues for personal digital assistants. Technical Communication, 49(1), 45-60.
Anderson, P., Chadwick, M. K., & Shin, W. (2017). Taking the TOEIC 1: Skill and strategies. Compass
Publishing.
Brown, E. (2001). Mobile learning explorations at the Stanford learning lab. Board of Trustees of the Leland
Stanford Junior University.

60 A Comparative Study on AI-directed and Human-directed MALL: The Effects on L2 Comprehension Skills
Campbell, C., & Geertsema, M. (2017). Improving student language learning in adult education
through the use of mobile learning: Barriers, challenges and ways to move forward. In Mobile
learning in higher education in the Asia-Pacific region (pp. 541-553). Springer.
Cha, Y., Kim, N.-Y., & Kim, H.-S. (2022). Effects of EFL learners’ perspectives on online English
classes: Gender, major, and proficiency. STEM Journal, 23(1), 42-57.
Chinnery, G. M. (2006). Going to the MALL: Mobile assisted language learning. Language Learning &
Technology, 10(1), 9-16.
Darsih, E., Wihadi, M., & Hanggara, A. (2021). Using ELSA app in speaking classes: Students’ voices.
http://eprints.eudl.eu/id/eprint/1451/1/eai.12-12-2020.2304993.pdf
Fallahkhair, S., Pemberton, L., & Griffiths, R. (2007). Development of a cross platform ubiquitous
language learning service via mobile phone and interactive television. Journal of Computer Assisted
Learning, 23(4), 312-325.
Greene, T. (2017). AI-powered language learning promises a fast track to fluency.
https://thenextweb.com/apps/2017/12/01/ai-powered-languagelearning-promises-to-fast-t
rack-fluency/
Hinkel, E. (2010). Integrating the four skills: Current and historical perspectives. Oxford Handbooks Online.
https://doi.org/10.1093/OXFORDHB/9780195384253.013.0008
Hong, Z. W., Huang, Y. M., Hsu, M., & Shen, W. W. (2016). Authoring robot-assisted instructional
materials for improving learning performance and motivation in EFL classrooms. Journal of
Educational Technology & Society, 19(1), 337-349.
—
Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning Are they really useful?
A systematic review of chatbot supported language learning. Journal of Computer Assisted Learning,
38(1), 237-257.
Istrate, A. M. (2019). The impact of the virtual assistant (VA) on language classes. In the 15th International
Scientific Conference eLearning and Software for Education (Vol. 15, No. 1, pp. 296-301). Carol I
National Defence University Publishing House.
Jones, S., Richards, A., Cho, Y., & Lee, Y. J. (2018). The effect of the fourth industrial revolution on
general English education: A case study of native English instructors’ perceptions at a Korean
university. Korean Journal of General Education, 12(2), 169-197.
Kanero, J., Geçkin, V., Oranç, C., Mamus, E., Küntay, A. C., & Göksun, T. (2018). Social robots for early
language learning: Current evidence and future directions. Child Evelopment Perspectives.
https://onlinelibrary.wiley.com/doi/abs/10.1111/cdep.12277
Kannan, J. & Munday, P. (2018). New trends in second language learning and teaching through the lens
of ICT, networked learning, and artificial intelligence. Círculo de Lingüística Aplicada a la
Comunicación, 76, 13-30 http://dx.doi.org/10.5209/CLAC.62495

Na-Young Kim 61
Kim, B., Suh, H., Heo, J., & Choi, Y. (2020). AI-driven interface design for intelligent tutoring system
improves student engagement. In Woodstock ’18: ACM Symposium on Neural Gaze Detection (pp.
1-15). ACM.
Kim, N. Y., Cha, Y., & Kim, H. S. (2019). Future English learning: Chatbots and artificial intelligence.
Multimedia-Assisted Language Learning, 22(3), 32-53.
Kim, N. Y. (2018). A study on chatbots for developing Korean college students’ English listening and
reading skills. Journal of Digital Convergence, 16(8), 19-26.
Kimura, M., Obari, H., & Goda, Y. (2011). Mobile technologies and language learning in Japan. In M.
Levy, F. Blin, C. B. Siskin, & O. Takeuchi (Eds.), WorldCALL: International Perspectives on
Computer-Assisted Language Learning (pp. 38-54). Routledge.
Kukulska-Hulme, A., & Shield, L. (2008). An overview of mobile assisted language learning: From
content delivery to supported collaboration and interaction. ReCALL, 20(3), 271-289.
Kusumaningrum, M. A. D., & Pertiwi, D. R. (2021). An intelligent tutoring system for learning listening
TOEIC. Jurnal Berdaya Mandiri, 3(1), 456-467.
Lougheed, L. (2017). Longman preparation series for the TOEIC test . Pearson Education ESL
Liu, L. (2011). Factors influencing students’ preference to online learning: Development of an initial
propensity model. International Journal of Technology in Teaching & Learning, 7(2).93-108.
Liu, Q., Diao, L., & Tu, G. (2010). The application of artificial intelligence in mobile learning. In 2010
International Conference on System Science, Engineering Design and Manufacturing Informatization (Vol.
1, pp. 80-83). IEEE.
Loh, H., Shin, D., Lee, S., Baek, J., Hwang, C., Lee, Y., ... & Choi, Y. (2021). Recommendation for
effective standardized exam preparation. In LAK21: 11th International Learning Analytics and
Knowledge Conference (pp. 397-404). https://doi.org/10.1145/3448139.3448177
Mason, B., & Krashen, S. (2014). Can second language acquirers reach high levels of proficiency
through self-selected reading? An attempt to confirm Nation’s (2014) results. International Journal
of Foreign Language Teaching, 10(2), 10-19.
Maskey, S. (2020, June 8). Artificial intelligence in education transformation. Forbes.
https://www.forbes.com/sites/forbestechcouncil/2020/06/08/artificial-intelligence-in-educ
ation-transformation/?sh=7ecc0d5d32a4
Murray, D., Koziniec, T., & McGill, T. J. (2015). Student perceptions of flipped learning. Australian
Computer Science Communications, 37(2), 57-62.
Obari, H., Lambacher, S., & Kikuchi, H. (2020). The impact of using AI and VR with blended learning
on English as a foreign language teaching. In K.-M. Frederiksen, S. Larsen, L. Bradley & S.
Thouesny (Eds.), CALL for Widening Participation: Short Papers from EUROCALL 2020 (pp.
253-258). Research-publishing.net.

62 A Comparative Study on AI-directed and Human-directed MALL: The Effects on L2 Comprehension Skills
Okumuş Dağdeler, K.; Konca, M. Y., & Demiröz, H. (2020). The effect of mobile-assisted language
learning (MALL) on EFL learners’ collocation learning. Journal of Language and Linguistic Studies,
16(1), 489-509.
Salaberry, M. R. (2001). The use of technology for second language learning and teaching: A
retrospective. The Modern Language Journal, 85(1), 39-56.
Steel, C. (2012). Fitting learning into life: Language students’ perspectives on benefits of using mobile
apps. In Brown, M. Hartnett, M. and Stewart, T. (Eds.), Future challenges, sustainable futures (pp.
875-880). Proceedings Ascilite Welllington.
Stockwell, G. (2010). Using mobile phones for vocabulary activities: Examining the effect of platform.
Language Learning & Technology, 14(2), 95-110.
Sumo, D. S., & Bah, M. L. (2021). Chinese language education in the era of artificial intelligence:
Innovation development, pedagogy & the smart classroom. Education Quarterly Reviews, 4(4),
264-270.
Xu, Q. (2020). Applying MALL to an EFL listening and speaking course: An action research approach.
Turkish Online Journal of Educational Technology-TOJET, 19(4), 24-34.
Yang, H., & Kim, H. (2021). Development and application of AI chatbot for cabin crews. Korean Journal
of English Language and Linguistics, 21, 1085-1104.

Applicable level: tertiary education

Author: Kim, Na-Young (Sehan University, Assistant Professor, 1st author);
[email protected]

Received: April 30, 2022

Reviewed: May 31, 2022
Accepted: June 15, 2022

Na-Young Kim 63