See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/344440906

THE IMPACT OF LEARNING MANAGEMENT SYSTEMS ON STUDENTS'

ACHIEVEMENT IN LANGUAGE EXAMS

Article in Turkish Online Journal of Distance Education · October 2020

DOI: 10.17718/tojde.803410

CITATIONS READS

20 2,365

2 authors:

Emel Akay Eylem Koral Gümüşoğlu

Anadolu University Anadolu University
22 PUBLICATIONS 107 CITATIONS 9 PUBLICATIONS 192 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Emel Akay on 01 October 2020.

The user has requested enhancement of the downloaded file.

Turkish Online Journal of Distance Education-TOJDE October 2020 ISSN 1302-6488 Volume: 21 Number: 4 Article 12

THE IMPACT OF LEARNING MANAGEMENT SYSTEMS ON

STUDENTS’ ACHIEVEMENT IN LANGUAGE EXAMS

Dr. Emel AKAY

ORCID: 0000-0003-4322-1615
School of Foreign Languages
Anadolu University
Eskisehir, TURKEY

Eylem KORAL GUMUSOGLU

ORCID: 0000-0002-2681-5166
School of Foreign Languages
Anadolu University
Eskisehir, TURKEY

Received: 10/12/2019 Accepted: 12/03/2020

ABSTRACT
The use of technology in or beyond classrooms has created a stupendous paradigm to bring down the
classroom walls and open up for new and unique learning opportunities. With the plethora of materials
and rich learning object repositories, LMSs gripped attention of many educational institutions in different
levels. Although LMS has been heralded as a great platform to support learning, the impact of it on
student achievement has always been under question. Therefore, the aim of this study was to investigate
the relationship between university students’ achievement in an English preparatory program and the use
of LMS in language learning process. In a mixed-method design, quantitative data were collected from 321
students in the School of Foreign Languages, and qualitative data were gathered from interviews with 30
randomly selected students. The results revealed that the use of LMS had a much greater impact on midterm
exams than the proficiency exam although it had an effect on both. The analyses of the interviews indicated
that participants had mostly positive attitudes towards the use of LMS as they thought it contributed to their
language learning processes.

Keywords: LMS, language, EFL, ELT, exam, achievement.

INTRODUCTION
It has been realized that until the nature of educational relationships changes in the classroom and in the
institutional level, the value of information will not be fully understood. Educational experience should be
transferred in such a way that it becomes meaningful to the information-age learner (Frand, 2000). Thus,
education has faced a digital transformation with the technological aspects used in every part of education.
Green and Gilbert (1995) states that “The old competitive reference points describing information resources
that used to distinguish between institutions – the numbers of science labs and library books -are being
replaced by a new one: information resources and tools available to students (p.15) Like all the fields in
education, English Language Teaching (ELT) has also undergone a change. A lot of changes have been
brought into materials and technology devices used in language learning process. The use of technology as a
tool to develop different language skills has received great attention. (Chapelle, 2001) Among all these tools
and interfaces, Language Management Systems (LMS) have rapidly emerged and have been widely used in
second language learning. The first introduction of the LMS dates back to late 1990s. However, the content
and format of LMS was nothing close to what it is today. The LMS picked up quick with innovations right
before the turn of the century and they have become ubiquitous at higher education all around the world.

206
LITERATURE REVIEW
“The Learning Management System is an online program that serves as a learning and communication
platform for students” (Borboa, Joseph, Spake & Yazdanparast, 2017, p.19). Also, Reigeluth et al. (2008)
maintain that LMSs could provide “a variety of instructional features that allow teachers to truly customize
learning for each learner, and to facilitate choice and control for the learners as they work towards mastery
of required attainments and deep knowledge of all standard subjects and skills” (p.38). According to Ellis
(2009) an LMS should do the following:
• centralize and automate administration
• use self-service and self-guided services
• consolidate training initiatives on a scalable web-based platform assemble and deliver learning content
rapidly
• support portability and standards
• personalize content and enable knowledge reuse
To be able to fulfill its functions, LMSs have communication and cooperation tools (announcement areas,
e-mail, chat, list servers, instant messaging and discussion forums); content development and delivery
tools (learning resources, development of learning objects repositories and links to internet resources);
management tools (registering, enrolling, displaying timetables, managing students activities and electronic
office hours); and assessment tools (submission, multiple choice testing, collaborative work and feedback)
(Coates,et al., 2015).
Over the years, with the changes experienced in education, ELT approaches methods and techniques have
moved from teacher- centered approach to learner-centered ones and Watson, Lee and Reigeluth (2007)
indicate that “the learner –centered paradigm of education cannot be effectively implemented without
technology and by the same token, technology cannot approach its potential contribution to education and
learning without a learner-centered paradigm of education” (p.70). Some contributions of technology to
education like customizing assessment, analyzing student progress, evaluating student performance, tracking
academic achievement, and identifying areas for additional scaffolding or assistance are clearly the support
of learner-centered paradigm (Taylor, 2004; Watson &Watson 2007). Among the available technologies for
instruction and learning, LMSs appear to be the most promising tool to facilitate learner-centered instruction
in language education. The use of LMS in the lecture shifts the traditional way of learning (teacher-centered
approach) to a more student-centered approach in which a student is the constructor of knowledge (Lippert,
1993). According to Chittaro and Ranon (2007), with the use of LMS in teaching and learning, students
become responsible for their learning and identify and interact with a variety of technologies to construct
and discover their own knowledge. Other benefits that LMS offer in the classrooms are student-centered,
team-centered collaborative learning, and a high level of student-to student interaction (Kiesler, 1992).
LMS enables collaboration-based learning activity among students (Cavus, 2007). According to Roschelle
(2003), with the help of collaborative activities, students are able to deal with more complex problems and
understand the subject matter better. It also allows learners to communicate and interact with their teachers
and peers in order to work together in a new and enjoyable way (Aljarrah, 2011). Actually, one of the driving
forces for adopting LMS is often related to improved communication between lecturers and students in
large classes (Eyitayo, 2005). Most LMSs include some or all of the following core components facilitating
the interaction and collaboration among the students course management tools (syllabus, calendar, drop
boxes, announcements), content tools (content pages, quizzes, assessments), and communication tools
(asynchronous e-mail, discussion forums, chat), all of which allow instructors to provide content and learning
activities, test learning, receive assignments, and conduct discussions and other course-related activities in
a principally asynchronous online environment (Simonson, Smaldino, Albright & Zvacek, 2006). As it is
claimed by Govender (2010), all these positive aspects of LMS seem to better engage students individually
in the learning process and this affects the success of students in a positive way. According to the result of the
studies conducted by Hardy et all., (2005) and Govender&Grayson (2008), there is a positive correlation
between use of LMS and students’ achievement level. Students who performed well in exams used LMS
almost twice as much as others.

207
Although LMS has become an important part of language learning process all over the world, there are
still some problems that need to be covered by the institutions. According to Bates (2005) low rates of
participation, learner resistance, high non-completion rates and poor learner performance are among these
problems. Also, Mott& Wiley (2009) believe that it did not give the expected success because of several
reasons. One of the reasons of this failure is the institutional resistance to change. Digital transformation
of the higher education institutions is not always easy and takes effort and time to implement. In some
cases, institutions are trying to apply this technology just to be competitive among others and for the sake
of using technology though it is not seen as a part of the solution (Chadwick,2001). Also, as Coates et
al., (2005) state institutions do not have overall control over content. Without ultimate control over the
source code that runs the program, pedagogical content may no longer serve to the purpose of the teacher
and the institution. Another problem is the gap in terms of digital literacy between the teacher and the
students (Bennett, Maton, &Kervin, 2008). The new generation called digital natives (Prensky, 2001) are
tech-savvy individuals and they are more inclined to use technology in their learning process than their
teachers who are mostly used to conventional methods of teaching. This gap could create some problems in
the implementation of LMS. No matter new generation is good at using technology, students who are new
to this particular technology might still exhibit some concerns initially and this could cause a reluctance
to actively participate in the online classroom areas (CEDL, 1999).Thus, multiple factors must be taken
into consideration while implementing LMS into language learning process and enough attention must be
given to computer skills and the knowledge of both the teachers and students (Zanjani, Nkyvist, & Shlomo,
2013).

METHOD
Research Design
This study investigates the relationship between university students’ achievement in an English preparatory
program and student related variables as well as the use of LMS in language learning process. Therefore,
the design of the study is descriptive and cross-sectional without any experimental manipulation. Since the
study combines the quantitative data and qualitative data related to students’ perceptions regarding the use
of LMS, mixed method research was used. The study aims to answer the following research questions:
1. Is there a significant difference in LMS scores according to the student attitudes (with regard to time
spent for LMS, having technical equipment, having technical knowledge, convenience of the system,
technical support, teacher help, teacher encouragement, pleasure and benefit)?
2. Is the student achievement in English Proficiency Exam associated with student related variables
(education year, level, faculty, program type) and the use of LMS?
3. Is the student achievement in Midterm Exam associated with student related variables (education
year, level, faculty, program type) and the use of LMS?
4. What are the students’ perceptions considering the use of LMS in a language learning process?

Participants
The participants of the study were university students registered in the 1- year English Preparatory Program
of a state university in Turkey during the interval 2017-2018, Spring Semester. Students contributed to the
study on voluntary basis, therefore the ones who provided the required data were all included. In total, 321
students aged between 18-24 partook in the research.

Measurements
Dependent Variables
Achievement: To assess students’ achievement, the overall scores of English Proficiency Exam, which was
administered at the end of spring term, were used. The items of the exam were prepared by the Testing
Unit of the institution based on the learning outcomes defined by Global Scale of English in the range of

208
24-66. The exam consists of three parts: 1) Multiple-choice proficiency test includes items for listening
comprehension, reading comprehension, language and vocabulary use. 2) Writing test requires students to
showcase their writing skills on the given genres which are also in the outcomes of the course. 3) Speaking
test includes individual and pair discussion questions for which students perform in front of a jury and their
performances are video recorded. The products of writing and speaking tests are evaluated by the instructors
based on predetermined rubrics after standardization and norming sessions, whereas the multiple-choice test
is evaluated in a computerized system. Students in different levels can take the proficiency exam when they
finish one-year program. To reach the overall proficiency score, 60% of the multiple-choice test, 20% of the
writing test, and 20% of the speaking test are added. Students who receive 60 points or over are considered
successful. In this respect, students’ overall proficiency scores were used as the indicator of achievement in
this research. The scores were recoded as a dichotomous variable with the values 0= FAIL and 1= PASS.
Descriptive statistics revealed that out of 321 students, 37,1 % received passing score, while 62,9% failed
(See Table 1).
In addition, scores (out of 100) of the first Midterm Exam in spring term were also received. Similar to
the Proficiency Exam, the items were prepared by the Testing Unit of the institution based on the learning
outcomes defined by Global Scale of English. Since this is an achievement test, the content of the exam
is specific to each level and covers what has been taught. The exam consists of three parts: 1) Multiple-
choice test includes items for listening comprehension, reading comprehension, language and vocabulary
use. 2) Writing test requires students to showcase their writing skills on the given genres which are also in
the outcomes of the course. 3) Speaking test includes individual and pair discussion questions for which
students perform in front of a jury and their performances are video recorded. The products of writing and
speaking tests are evaluated by the instructors based on predetermined rubrics after standardization and
norming sessions, whereas the multiple-choice test is evaluated in a computerized system.

Independent Variables
LMS Scores
In the preparatory program, students are required to attend two hours of laboratory lessons where they have
to complete several productive tasks to be done with the help of computer and internet by using several
web tools and the exercises from a publisher provided Learning Management System (LMS) based on the
outcomes of the course (the name of the publishing company or the LMS platform will not be mentioned
here in order to avoid copyright infringement and biased conceptions of the results). These LMS exercises are
assigned by the class teacher every week to be done in the lab hours or sometimes as self-study materials that
students can complete at home at their own pace. The system keeps track of students’ scores in each exercise
and produces a final report including overall assignment score. In this study, overall assignment scores in
LMS for spring period were used as one of the independent variables to predict their effect on achievement
in proficiency and midterm exams. LMS scores were also taken as one of the dependent variables to analyze
the impact of student attitudes on the variation of LMS scores.

Level
Students in the English Preparatory Program are placed in 5 different proficiency levels based on the results
they receive from the placement test which is administered at the beginning of the education year. These
levels are named as A Success, A, B, C, and D which are equivalent to upper intermediate, intermediate, pre-
intermediate, elementary and beginner levels respectively. In this study, data from A, B, and C level students
in spring term were used (levels were coded as 1, 2 and 3 in the same order). Data from A Success and D
levels were excluded since the number of students was limited and the data from them included missing
values. On this ground, out of 321 students 90 A Level, 107 B Level, and 124 C Level students provided
data for the study (See Table 1).

209
Year
Students in the preparatory program are categorized in two groups as obligatory and optional. In the
obligatory group, one-year English program is a requirement of the faculties prior to the studies in the
fields students are registered. In the optional group, students join in the English program with their own
request although it is not a requirement of their faculties. In this case, students who are unsuccessful in the
obligatory group have to repeat the year, while the ones in optional group can continue their education in
their faculties without the certificate of proficiency. Students participated in this study were 88,8% first year
students (either obligatory or optional coded as 1) and 11,2% repeat students (coded as 2).

Faculty
Students registered to various faculties at this university receive one-year English education either optionally
or mandatorily. The majority of students in this study were registered to Engineering Faculty (34%),
Economics (17,8%) and Communication (13,4%) respectively as can be seen in Table 1.

Program Type
This variable categorizes students into two groups: the ones who receive education in English preparatory
program optionally (coded as 1), and obligatorily (coded as 2) based on the requirements of their faculties.
In accordance with the relevant regulations of the university, all the departments of Engineering Faculty and
certain departments of Literature, Education, Science, Economics, and Communication Faculties require
students to receive 1 year English Preparatory Education obligatorily. On the other hand, preparatory
programs in Faculty of Law, Architecture, Tourism, Aviation, Health Sciences, Sports Sciences and Vocational
School of Transportation are optional. Students in both groups are placed in classes based on the placement
exam scores and they are responsible for the same curriculum according to their levels. As can be seen in
Table 1, out of 321 students 73 are registered in the optional program whereas 248 are in the obligatory
program.

Table 1. Total sample by pupil variables

Variable Value N %
Proficiency Result Pass (1) 119 37.1
Fail (0) 202 62.9
Level A (0) 90 28.0
B (1) 107 33.3
C (2) 124 38.6
Year First (0) 285 88.8
Repeat (1) 36 11.2
Program Type Optional (0) 73 22.7
Obligatory (1) 248 77.3
Faculty Literature (0) 11 3.4
Science (1) 31 9.7
Aviation (2) 25 7.8
Education (3) 8 2.5
Law (4) 4 1.2
Economics (5) 57 17.8
Communication (6) 43 13.4
Architecture (7) 19 5.9
Engineering (8) 109 34.0
Health Science (9) 2 0.6
Tourism (10) 5 1.6
Vocational (11) 7 2.2

210
Data Collection Instruments and Analysis
Both qualitative and quantitative data were collected from students at the end of Spring Term in 2018 for
the study. In the last laboratory lessons of the term, students were given an online questionnaire which
includes items to evaluate the LMS procedure they used during the year. The questionnaire designed by
the researchers included 5 items for general information (group, level, year, type of program, faculty) and
9 items to investigate participants’ attitudes towards the use of LMS. These 9 items were created based on
the literature and controlled by three experts in the field prior to the study in terms of language and clarity.
In addition, the questionnaire was piloted on two classes (43 students apart from the actual participants of
the study) and necessary changes were made in terms of language. As the questionnaire does not examine a
single phenomenon similar to the scales, but was designed in order to collect data about the different aspects
of students’ LMS use and their attitudes towards them, exploratory and confirmatory factor analyses were
not conducted.
The LMS scores of students in the platform are not publicly available. Assigned laboratory teachers could
only see them when they sign in their account. To collect students’ LMS scores, a detailed guideline on how
to export reports in the system was sent to the class teachers. Teachers downloaded student reports from the
LMS system for their classes in an Excel Sheet format and emailed to the researchers when the term finished.
The scores of the students who completed the questionnaire were filtered from the reports.
Similarly, when the grading procedures finished in the second week of June, students’ Midterm grades and
Proficiency Exam scores were received from the administration as they are not publicly available. Midterm
and Proficiency Exam scores of students who completed the questionnaire were filtered as well.
Also, semi-structured interviews were conducted with 30 students (10 from each level) who participated in
the study in the same week. These students were randomly selected from the participant group, and joined
the interviews on voluntary basis. Interview questions were designed by the researchers according to the
literature and checked by the same three experts in the field. Necessary changes were made on the questions
based on their comments.
The data collected for the study were computed and analyzed by using IBM SPSS 22 statistical software.
In the quantitative phase of the analyses, nonparametric tests and two different regression analysis were
conducted. The association between LMS scores and student attitudes on the items in questionnaire was
examined with Kruskal-Wallis and Mann-Whitney U tests. In order to analyze the relationship between
Midterm scores and independent variables, multiple regression analysis was conducted. The association
between Proficiency Exam scores and independent variables was analyzed with logistic regression method.
Qualitative data collected from semi-structured interviews were coded and general themes were created. In
order to increase the validity and reliability of the results, two researchers and another academician with a
PhD on Distance Education independently coded the interviews and results were compared. Also, detailed
descriptions of the participants were given above.

FINDINGS
The Effect of Student Attitudes on LMS Scores
Data collected with student questionnaire were analyzed to examine the effect of student attitudes on LMS
scores. Since the data did not show a normal distribution, proven in both Kolmogorov- Simirnov and
Shapiro-Wilk tests (p<.05), nonparametric tests (Mann Whitney U and Kruskal Wallis Test) were used
instead of t-test and ANOVA to make comparisons.

211
 Table 2. Kruskal-Wallis Test Results
Variable Category N Rank X2 p Significant
mean difference
How many hours a week do None 68 55.11 16.358 .001*** None / 1-2 hours
you spend for LMS? 1-2 hours 211 86.69 None / 3-4 hours
3-4 hours 36 99.24
5-6 hours 6 106.50
Do you like studying English Strongly Disagree 67 65.69 6.741 .150
with LMS? Disagree 54 73.64
Neutral 115 88.13
Agree 65 85.83
Strongly Agree 20 91.50
Is it beneficial to study English Strongly Disagree 39 64.84 6.950 .139
with LMS? Disagree 48 88.43
Neutral 124 75.55
Agree 83 93.49
Strongly Agree 27 77.44
(p<.001***, p<.01**, p<.05*)

As the variables included more than 2 categories, Kruskal-Wallis test on k independent samples was
conducted. The results, as can be seen in Table 2, revealed that LMS scores significantly vary according to the
time spent for LMS (X2=16.358, p=0.001). When the pairwise comparisons were examined, it was detected
that students who spent 1or 2 hours on LMS received better scores compared to the ones who did not spend
any time, which was not a surprising fact. Similarly, students who spent 3 or 4 hours on LMS received better
scores compared to the ones who did not spend any time on LMS. On the other hand, neither pleasure
about studying English with LMS nor its benefit on the studies reported significant results (p>0.5).

Table 3. Mann-Whitney U Test Results

Variable Category N Mean Rank Sum of U p
Ranks
Do you have technical equipment No 62 70.65 1907.50 1529.500 .203
to use LMS? Yes 259 83.09 11133.50
Do you have technical knowledge No 19 49.05 490.50 435.500 .024*
to use LMS? Yes 302 83.12 12550.50
Do you get technical support when No 90 84.48 3801.50 2453.500 .553
you need? Yes 231 79.65 9239.50
Does your teacher help when you No 12 80.70 403.50 388.500 .988
need? Yes 309 81.01 12637.50
Does your teacher encourage you No 24 91.65 916.50 648.500 .453
to use LMS? Yes 297 80.29 12124.50
Is it easy to use LMS? No 25 62.43 437.00 409.000 .278
Yes 296 81.84 12604.00

(p<.001***, p<.01**, p<.05*)

As these were dichotomous variables, Mann-Whitney U test on 2 independent samples was conducted.
According to the estimations given in Table 3, it was seen that students who have technical knowledge
to use LMS received significantly better scores (p<.05). It can be implied that the knowledge on how to
use computer and the Internet significantly affects LMS scores. However, having technical equipment,
convenience of the system, technical support, teacher help and teacher encouragement did not create a
significant variance on LMS scores.

212
Thus, it can be summarized that students who know how to operate computer systems and the Internet,
as well as the ones who spend more time to study English with LMS, receive better scores on the platform.

Student Achievement in Proficiency Exam

In the previous section, students’ attitudes which affect the LMS scores were identified. The major aim in
this section was to see whether LMS scores have an impact on the English Proficiency Exam results. In order
to determine the relationship between the achievement in English Proficiency Exam and the independent
variables, a logistic regression model was designed. Prior to the analyses, residual statistics and collinearity
diagnosis were performed to detect outliers and multicollinearity problems among the variables and to check
the assumptions of the regression models used. The results of the residual statistics reveal that the values
reached are in an acceptable range (between -2.163 and +2.279). In this study, VIF values range between
1.033 and 4.292, which proves there is an acceptable level of multicollinearity. In summary, based on the
values reached it is assumed that the error terms are independent and predictors are not correlated with
each other (Barnett and Lewis, 1978; Cook & Weisberg, 1982; Field, 2009; Menard, 1995; Myers, 1990).
Therefore, the logistic regression analysis was conducted and the results were given in Table 4.

Table 4. Logistic regression on proficiency exam scores

Model 1 Model 2
Variables B df Sig Wald Exp(B) B df Sig Wald Exp(B)
Constant -.162 1 .306 1.047 .851 -.726 1 .567 .328 .484
LMS .018 1 .004** 8.503 1.018
Level (1) -1.579 1 .063 3.457 .206
Level (2) -2.299 1 .000*** 23.241 .100
Faculty (1) -1.478 1 .051 .3.802 .228
Faculty (2) 1.206 1 .415 .664 3.339
Faculty (3) 1.430 1 .241 1.377 4.181
Faculty (4) 21.108 1 1.000 .000 1.469E+9
Faculty (5) -.690 1 .186 1.749 .502
Faculty (6) -.260 1 .689 .160 .771
Faculty (7) .192 1 .901 .015 1.212
Faculty (8) 1.006 1 .426 .633 2.735
Faculty (9) -18.871 1 1.000 .000 .000
Faculty (10) 3.281 1 .066 3.382 26.613
Faculty (11) -20.415 1 .999 .000 .000
Type (1) 1.097 1 .396 .721 2.996
Year (1) -.792 1 .175 1.841 .453
-2 Log 222.143 161.168
Likelihood:
Reference category: First
R2= 0.274, Cox & Snell R2=.315, Nagelkerke R2=.421, Model X2=60.974, p<0.000 (p<.001***, p<.01**,
p<.05*)

When the logistic regression was carried out with only the constant variable (proficiency) in Model 1, it was
detected that only 54% of the participants were classified correctly as pass or fail as can be seen in Table 6.
Wald test result also indicated that the model was not significant (p=0.306). In addition, the overall statistics
result for the variables which were not in the equation was found 51.255 (p<0.001), and this implies that
other variables should be added to the model. Model 2 was carried out using Enter method and independent
variables (LMS, level, faculty, type, year) were added to the regression model. The results revealed that the
classification of participants according to the proficiency exam increased to 76.4%. -2 Log Likelihood value
decreased from 222.143 to 161.168 indicating that the regression model improved. When the independent
variables were examined, it was determined that LMS scores positively contributed to the proficiency exam

213
result significantly (p=0.01). Beta value implies that when LMS scores increase one point, possibility of
passing from proficiency increases 0.018. In addition, according to the odds ratio, students who have higher
LMS scores have the possibility to pass from the proficiency exam 1.018 times more. Students’ level of
English was another factor which was included in the regression model. According to the results, it was seen
that level variable negatively influenced the achievement in the proficiency exam. While the statistics did not
show a significant result for B level students (p=0.63), findings for C level students were highly significant
(p<0.001). Also, adjusted odds ratio explains that when the level decreases to C level, the possibility to
pass from proficiency exam is 10 times less compared to A level students. On the other hand, none of the
categories in the faculty, type and year variables show a significant outcome (p>0.05). To analyze how much
the model improved after the inclusion of independent variables, Hosmer & Lemeshow Test was conducted.
In general terms, this value ranges between 0 and 1 and 1 indicates a perfect fit. In our case, the significance
of Hosmer& Lemeshow Test result was found 0.838 (X2=4.209, df=8), which shows the model predicts the
achievement in proficiency exam in a good level.

Student Achievement in Midterm Exam

The impact of independent variables on midterm exam was analysed with multiple regression method. For
the variables measured in nominal scale, dummy variables were created and included in the model. A level,
first year students, optional program, and Literature Faculty were taken as the reference categories to make
comparisons.

Table 5. Summary of multiple regression analysis for midterm scores (N=321)

Variable B SE(B) b t Sig.(p)
LMS .098 .027 .256 3.576 .000*
B Level -2.035 3.806 -.039 -.535 .594
C Level -2.248 2.087 -.089 -1.077 .283
Repeat -5.324 2.847 -.145 -1.870 .064
Obligatory 21.737 5.184 .694 4.193 .000*
Science 13.452 3.224 .309 4.172 .000*
Aviation 18.968 6.065 .362 3.127 .002*
Education .815 4.734 .012 .172 .864
Law 22.811 12.118 .142 1.882 .062
Economics -4.538 2.435 -.143 -1.864 .064
Communication -1.171 3.088 -.029 -.379 .705
Architecture 22.732 5.922 .454 3.838 .000*
Engineering 21.729 6.024 .682 3.894 .000*
Health Sciences 14.547 12.011 .090 1.211 .228
Tourism 42.457 8.038 .454 5.282 .000*
Vocational 12.839 9.458 .113 1.358 .177
R= .569 R2= .324, DR2=.254, *p<0.01, FChange=4.625, p<0.001, Durbin-Watson=1.939

The overall model of the multiple regression analysis was conducted using Enter method. Multiple correlation
coefficient between the independent variables and midterm scores was found R=0.569. According to the R2
value, it was identified that 32.4% of the variability in midterm scores is accounted for by the predictors
included in the model. F-ratio of the model was found 4.625 which is significant with a probability p<0.001.
In addition, Durbin-Watson statistic reports the value of 1.939. When these change statistics were examined,
it can be assumed that the inclusion of the independent variables improved the regression model. Also,
ANOVA test result indicates that the regression model significantly predicts the midterm scores (F=4.625,
df=15, p<0.001).

214
In the next step of the analysis, parameters included in the model were examined. Beta value for LMS was
found .098. This result indicates that when LMS scores increase one point, midterm scores increase 9.8
points (p<0.001). Another significant predictor was the program type and it was found that students in the
obligatory program receive better scores from the midterm exams compared to the ones in optional program.
As the positive beta value shows, when students are in the obligatory program, their midterm scores increase
21.7 points (p<0.001). The analysis of the faculties revealed that being in the Science Faculty, Aviation,
Architecture, Engineering and Tourism Faculties have positive impact on midterm scores. According to the
beta values, students who are registered to the Science Faculty receive 13.45 more points from the midterm
exams compared to the Literature Faculty students (p<0.001). Also, students in Aviation Faculty receive 18.9
points more (p<0.01), Architecture Faculty students get 22.73 points more (p<0.001), Engineering students
get 21.72 more (p<0.001) and Tourism students get 42.45 points more (p<0.001) from the midterm exams
compared to the Literature Faculty students. On the other hand, it was seen that other variables related to
faculties, education year and level did not create a significant impact on midterm scores (p>0.05).

The comparison of the regression analyses of proficiency exam and midterm exam shows that LMS scores
have a much greater impact on midterm exam scores although it has an effect on both. Besides, while the
level factor was a significant predictor on proficiency scores, it was not associated with the midterm scores.
The comparisons also indicate that program type does not influence proficiency results significantly whereas
it has an effect on midterm scores. Also, faculties students are registered to are significant in predicting
midterm scores while they do not predict proficiency exam results. On the other hand, being a first year or
repeat student predicts neither proficiency exam nor midterm exam results significantly.

Perceptions of Students Related to LMS

In order to reach a deeper insight about the students’ perceptions on the use of LMS, semi-structured
interviews were conducted with 30 randomly selected students. Since the interviews were conducted at the
end of the school year, students commented on the questions according to their hands-on experience with a
publisher-provided LMS. The numbers given in the tables do not match the total number of the participants
since students have mentioned more than one theme in their comments.
Initially, students were asked about the easy and difficult aspects of LMS. As can be seen in Table 6, 25
students found the use of LMS easy, while the rest (5) thought it is difficult to use.

Table 6. Easy and difficult aspects of LMS

What are the easy and difficult aspects of LMS? f f
Easy 25 Difficult 5
-access 14 -spelling 3
-interface/web design 13 -listening recordings 2
- types of exercises 4 -using headphones 1
-uploading homework 1

The majority of the students who thought it is easy to use LMS mentioned about the easy access to the
platform. Similarly, 13 students thought the interface of the platform is easy to use as they can navigate
easily in the system and find appropriate tasks. Also, 4 students mentioned that types of the exercises are
easy as they are quite similar to the ones they use in their course book. These ideas can be seen in the student
comments given below.

“Everything in the system is within easy reach. I can access with my password even at home with my
mobile phone.”
“The web design is not complicated, it is very simple. I can find what I want easily.”
“Exercises are similar to the ones in our course book, so I can understand the instructions easily.”

215
Students who found it difficult to use mentioned mostly about the spelling rules that the system accepts or
not. It was indicated that the system requires correct capitalization and punctuation while writing answers
for the open-ended questions. Also, as LMS is an internet-based platform, the problems caused by the speed
or quality of connection are indicated. Due to the connection speed, students found it difficult to upload
files or listen to recordings. On the other hand, the hardware required to use LMS, such as headphones, was
mentioned as the source of difficulty.

“The system considers my answers incorrect because of the smallest punctuation or capitalization
mistakes even when my answers are correct.”
“Sometimes when I upload my homework, the system freezes and I can’t do anything.”
“Because of the internet speed, listening tracks frequently stop. It is difficult to listen or watch
something.”
“Most of the earphones in the language labs do not work, so I skip the listening parts.”

Participants were asked about the technical support they might have needed during their LMS use. 24
students did not need any technical support, while 6 of them needed help for different problems they faced.

Table 7. Technical support for LMS

Have you ever needed technical support? f f

Yes 6 No 24
-registration 3
-password 2
-voice recording 2

Three students who needed technical support, faced with problems at the initial stage of their experience
while they were signing up for the LMS. They received support from class teachers at this stage. These students
mentioned that the codes provided by the publisher did not work or registration process did not work. Two
of the students forgot their passwords and had to contact with the online help to reset a new password. In
addition, two of the participants required technical help while they were recording and uploading their
voice. On the other hand, majority of the participants found the system quite simple or solved the technical
problems themselves. Students’ comments related to this question are given below:

“In the first lesson while we were signing up for LMS, I could not manage to register. It was the first
time I started using LMS, so I needed my teacher’s help.”
“I forgot my password and I could not access the platform. I chatted with the online help desk and
they sent a link to my email. I used it and created a new password.”
“While I was recording my voice, the program gave errors frequently and the page froze. My teacher
helped me about it.”

Students were also asked whether the use of LMS contributed to their language learning process or not. Out
of 30 participants 23 gave positive answers while the rest responded negatively about it (see Table 8).

216
 Table 8. Contribution to language learning process
Do you think LMS contributes to your language
learning process? f f
Yes 23 No 7
-reinforcement 8 -boring exercises 3
-skills practice 4 -mismatch with exams 2
-match with course book 4 -waste of time 2
-advantage for quizzes 4
-easy recall 3
-exposure to language 2
-detecting weaknesses 2

8 of the students who gave positive answers mentioned that the use of LMS provided a good reinforcement
for the things they learned during the course. 4 of them indicated that it was a good chance to practice
different language skills and 4 thought LMS contributed to their learning process as the learning object
repository of the platform was similar to the materials given in their course book. Also, 4 students asserted
that studying with LMS helped them in quizzes and in-class assessment procedures. 3 students mentioned
that they could easily recall what they studied on the platform as they were visual learners. While 2 of the
participants thought they were exposed to the language more, 2 of them indicated that they were able to
increase their awareness about their weaknesses in different skills. However, 3 of the participants mentioned
that the exercises in the platform were boring. Also, 2 of them asserted that there was a mismatch between
the exercises in the system and the question types in the midterm or proficiency exams. In addition, 2 of
the participants felt extremely negative about the contribution of LMS and thought it was a waste of time.
Comments given on this interview question depict the attitudes of students:

“The platform gives simple explanations of the topics I learn in class. In this way, it helps me to
remember and reinforce what I learn. I practice at home after the class.”
“Vocabulary and listening exercises were really good. They gave the examples of everyday language.”
“The exercises were designed in accordance with the things we learned in our book. So, it helped me
to practice.”
“When I studied with LMS before the quizzes, I received better grades.”
“We were exposed to the language more. You know, it is not possible to hear English on the street
here.”
“The exercises are very boring and childish.”
“Exam questions were not similar to the exercises, so they were not useful.”

Another interview question sought an answer about the specific skills that LMS might have contributed to
(see Table 9).

Table 9. Contribution to language skills

Which language skills did LMS contribute the most? f
All 23
Grammar 18
Vocabulary 14
Pronunciation 14
Listening 11
Writing 8
Reading 4
None 7

217
As can be seen in Table 9, while 7 of the participants thought it was helpful for none of the skills, majority
of the students felt it was useful for improving all skills. Students mostly mentioned about the positive
effect of LMS on grammar (f = 18), vocabulary (f = 14), pronunciation (f = 14), and listening (f = 11) skills
improvement. 8 students mentioned that their writing skills improved the most and only 4 participants
indicated the reading skill. However, none of the participants mentioned about the effect of LMS on
speaking skill. This might have resulted due to the lack of emphasis on speaking skill in the system or just
because of the students’ preferences.
In addition, students were asked about the sections they enjoyed a lot in LMS.

Table 10. Enjoyable sections of LMS

What are the most enjoyable sections of LMS? f
Listening exercises 23
Vocabulary games 20
DVDs / videos 16
Grammar exercises 12

As it is given in Table 10, majority of the participants mentioned they enjoyed the listening exercises the
most (f = 23). Also, students liked the vocabulary games since these games helped them to remember and use
the words better (f = 20). DVDs and videos were also popular among participants (f = 16) as they exemplified
the authentic use of language. Besides, grammar exercises caught students’ fancy in the LMS.

“I liked the listening sections in LMS. It was enjoyable to hear and complete the exercises.”
“Vocabulary games are just my cup of tea. Learning words by playing games is really enjoyable. I
loved puzzles.”
“Watching the videos of native people were really fun.”

In the last interview question, participants were asked whether they recommend the use of LMS in language
learning process or not. Out of 30 participants, 23 mentioned that they would recommend it to language
learners, while 7 responded negatively (see Table 11).

Table 11. Recommendation of LMS

Do you recommend using LMS in a language learning process? f f
Yes 23 No 7
-beneficial 20 -expensive 6
-reinforcing 19 -boring 5
-fun 12 -waste of time 2

Among the students who responded positively, 20 participants mentioned that LMS was beneficial for
their studies, 19 of them thought the exercises were reinforcing, and 12 found the use of LMS fun. On the
other hand, among the students who responded negatively, 6 participants mentioned that it was expensive,
5 found LMS boring, and 2 indicated that it is a waste of time. Some of the comments that support these
findings were given below:

“Studying with LMS after class is really reinforcing. I think other students should use it, too.”
“Instead of listening to the lectures or doing traditional exercises, practicing in front of a computer
by myself was more enjoyable. I recommend it other students as well.”
“The cost of the access code was really expensive. We can find free online websites instead of LMS.”
“I don’t think it is beneficial. Course book and supplementary pack are more than enough. I think
LMS is a waste of time.”

218
DISCUSSIONS AND CONCLUSION
This study investigated the relationship between university students’ achievement in an English preparatory
program and student related variables as well as the use of LMS in language learning process in a mixed
method design. It has been found out that these variables had an impact on students’ achievement in
language proficiency and midterm exams. For the proficiency exam, students’ language level significantly
predicted the scores. Students who have lower level of English received lower scores on the proficiency exam.
This result is also supported by the study conducted by Amua-Sekyi & Nti (2015) in which the findings
suggest that students’ weak background in English contributed to students’ poor performance in the English
examination. A similar study conducted in Turkish context by Aydin (2012) investigated the role of English
proficiency level of students from a state university and found out that students who had high proficiency
level achieved better scores in English Proficiency Exam.
For the midterm exam, program type (whether the program is obligatory or optional) and faculties that
students were registered also affected the scores significantly. According to this study, students in the
obligatory program received better scores from the midterm exams compared to the ones in optional program.
As opposed to our study, El-Omari (2016) indicated that there were no statistical differences between
compulsory and optional status of programs in terms of students’ language achievement. In addition, it was
observed in our study that achievement of students from different faculties varied which is also pointed out
by the studies of Akay & Toraman (2015) and Yu& Shen (2012).
In addition to the abovementioned student related variables, this study mainly showed that the use of LMS
by students in their language learning process predicted the proficiency and midterm scores significantly.
Our findings asserted that the students who received higher scores in LMS better performed in these exams,
which is also supported by literature. The studies conducted by Hardy et all., (2005) and Govender&Grayson
(2008) proved that there is a positive correlation between the use of LMS and students’ achievement level.
Students who used LMS more than others received better exam results. As Freeman & Anderson (2011)
states, LMS provides the students with more access to the target language as they can find many authentic
materials. Another study revealed that incorporating LMS activities increased students’ motivation to the
class content (Anwaruddin,2013). Similarly, a study which examined the contribution of technology in
teaching English Language proved that LMS could enhance students’ language proficiency ad facilitate
teaching process (Navidinia, Bidaki& Hekmati, 2016). Freeman & Anderson (2011) also pointed out that
use of technology in language teaching could be beneficial not only for students but also for teachers as it
can provide authentic written and audio-visual materials online.
The positive impact of LMS on students’ exam scores may have resulted from several reasons just as stated in
the literature. Since LMSs have different tools for communication and cooperation, content development
and delivery, management and assessment (Coates,et al., 2015), students could have control over their own
language learning processes. Thus, this may have a positive reflection on their language achievement.
Although the results were only specific to a university and difficult to generalize to other contexts, it is worth
considering LMS as a valuable tool to improve students’ motivation and achievement in language learning
after analyzing the advantages and drawbacks it may create.
As a result, LMS might bring new efficiencies to learning and teaching. With the use of LMSs, higher
education institutions may have the opportunities to reduce course management expenses and physical space
demands. Also, these tools may enhance knowledge management, enrich learning and teaching environments
by unifying fragmented information technologies, accelerate the access to information, improve quality
assurance procedures for course design and delivery (Brown 2001; Dutton & Loader 2002; Katz 2003; King
2001)

219
BIODATA and CONTACT ADDRESSES of AUTHORS
Dr. Emel AKAY is a Doctor of Educational Sciences at School at Foreign Languages,
Anadolu University. Akay gained her BA in English Language Teaching at Hacettepe
University in 1999. In 2002, she received her MA degree in MATEFL program at
Bilkent University. Dr. Akay gained her Ph.D. in Research Methods and Statistics in
2017. She is currently continuing her second Ph.D. in Educational Administration.
Her academic interest areas are research methods, multilevel analysis, educational
administration, educational technology and e-learning. She has 10 journal articles
published in national and international indexes, 2 international book chapters, 2
books and papers submitted to international meetings.

Emel AKAY
School of Foreign Languages
Address: Anadolu University, 26470, Eskisehir, Turkey
Phone: +90 2223350580-6170
E-mail: [email protected]

Eylem KORAL GUMUSOGLU is an instructor at School of Foreign Languages,

Anadolu University. Gumusoglu gained her BA in English Language Teaching in
Middle East Technical University in 2000. In 2003 she received her MA degree in
MATEFL program at Bilkent University. In 2005 she worked as a Foreign Language
Teaching Assistant at Governors State University in Chicago, USA. Currently,
Gumusoglu is a Ph.D. student in Distance Education Department at Social
Sciences Faculty, Anadolu University. Her academic interest areas are augmented
reality, e-learning, massive open online courses, learning management systems.

Eylem KORAL GUMUSOGLU

School of Foreign Languages
Address: Anadolu University, 26470, Eskisehir, Turkey
Phone: +90 2223350580-6170
E-mail: [email protected]

REFERENCES

Akay, E., & Toraman, C. (2015). Students’ attitudes towards learning English grammar: A study of scale
development. Journal of Language and Linguistic Studies, 11(2), 67-82.
Aljarrah, A.(2011). University of Jordan Students’ Attitudes Towards the use of the Learning Management
System as a learning tool. Dirasat Journal, educational sciences. 38(4). 1293 – 1304.
Amua-Sekyi, E.T.&Nti, S.K. (2015). Factors affecting students’ performance in English at colleges of
education in Ghana. International Journal of Research in Humanities, Arts and Literature, 3(10),
29-44.
Anwaruddin, S. M. (2013). Web 20 and Language Learners’ Motivation: An Action Research Study.
Canadian Journal of Action Research. 14(1):51–68.
Aydin, G. (2012). The role of English proficiency level, personal and affective factors predicting language
preparatory school students’ academic success. Unpublished Master’s Thesis, Middle East Technical
University, Ankara, Turkey.
Barnett, V., & Lewis, T. (1978). Outliers in statistical data. New York: Wiley.
Bates, A. W. (2005). Technology, e-learning and distance education (2nd ed). New York: Routledge Falmer.

220
Bennett, S., Maton, K., & Kervin, L. (2008). The ‘digital natives’ debate: A critical review of the evidence.
British Journal of Educational Technology, 39, 775–786.
Borboa, D., Joseph, M., Spake, D., & Yazdanparast, A. (2017). Perceptions and Use of Learning Management
System Tools and Other Technologies in Higher Education: A Preliminary Analysis. Journal of
Learning in Higher Education, 10(2), 17-23.
Brown, S. (2001). Campus Re-Engineering. In F. Lockwood & A. Gooley (eds), Innovation in Open and
Distance Learning: Successful Development of Online and Web Based Learning. London: Kogan Page.
Chadwick, C. (2001). Computadoras en la educacion: Problemas y precauciones. Revista Latinoamericana
de Estudios Educativos, XXXI (001), 87–98.
Chapelle, C. (2001). Computer Applications in Second Language Acquisition. Cambridge University Press,
Cambridge.
Chittaro L. and R. Ranon. (2007). Web3D technologies in learning, education and training: Motivations,
issues opportunities. Computers and Education, 49(1):3–18.
Cavus, N. (2007). Assessing the success rate of students using a learning management system together
with a collaborative tool in web-based teaching of programming languages. Journal of Educational
Computing Research, 36(3), 301-321.
CEDL (1999). Lucent Technologies Centre for Excellence in Distance Learning. Student attitudes towards
distance learning. Available: http:/www.lucent.com/cedl/stdtatt.html.
Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of learning management
systems on university teaching and learning. Tertiary education and management, 11, 19-36.
Cook, R. D., & Weisberg, S. (1982). Residuals and influence in regression. New York: Chapman & Hall.
Dutton, W.H. & Loader, B.D. (2002). Introduction: New Media and Institutions of Higher Education and
Learning. In W.H. Dutton & B.D. Loader (eds), Digital Academe: The New Media and Institutions
of Higher Education and Learning. London: Routledge.
Ellis, R. K. (2009). A Field Guide to Learning Management Systems. American Society for Training &
Development (ASTD).
El-Omari, A. H. (2016). Factors affecting students’ achievement in English language learning. Journal of
Educational and Social Research, 6(2), 9.
Eyitayo, O. (2005). Experimenting eLearning with a large class. International Journal of Education and
Development using ICT, 1(3).
Field, A. P. (2009). Discovering statistics using SPSS (and sex and drugs and rock’ n’ roll) (3rd ed.). London:
Sage.
Frand, J. L. (2000). The INFORMATION-AGE Mindset. Educause review, 35(5), 14-20. Freeman, L. D,
Anderson M.(2011) Techniques and Principles in Language Teaching. Oxford University Press
199-218.
Govender, I., & Grayson, D. J. (2008). Pre-service and in-service teachers’ experiences of learning to program
in an object-oriented language. Computers & Education, 51(2), 874-885.
Govender, D. W. (2010). Attitudes of students towards the use of a Learning Management System (LMS) in
a face-to-face learning mode of instruction. Africa Education Review, 7(2), 244-262.
Green, K. C., & Gilbert, S. W. (1995). Great expectations: Content, communications, productivity, and
the role of information technology in higher education. Change: The magazine of higher learning,
27(2), 8-18.
Hardy, J., Bates, S., Antonioletti, M., & Seed, T. (2005, September). Integrating e-learning and on-
campus teaching II: evaluation of student use. In J.Cook and D.Whitelock (eds.): Exploring the
frontiers of e-learning. Research Proceedings of the 12th Association of Learning Technology Conference
(Manchester), pp. 140-153.

221
 Katz, R.N. (2003). Balancing Technology and Tradition. Educause Review 38(4), 48-59.
Kiesler, S. (1992). Talking, teaching, and learning in network groups: Lessons from research. In A. Kaye
(Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp. 117-136).
Berlin: Springer-Verlag.
King, B. (2001). Making a Virtue of Necessity - A Low Cost, Comprehensive Online Teaching and Learning
Environment. In F. Lockwood & A. Gooley (eds), Inno- vation in Open and Distance Learning:
Successful Development of Online and Web Based Learning. London: Kogan Page.
Menard, S. (1995). Applied logistic regression analysis. Sage university paper series on quantitative applications
in the social sciences, 07-106. Thousand Oaks, CA: Sage.
Mott, J., & Wiley, D. (2009). Open for learning: The CMS and the open learning network. In Education –
Exploring our connective educational landscape, 15, 3–22.
Myers, R. (1990). Classical and modern regression with applications (2nd ed.). Boston, MA: Duxbury.
Navidinia, H., Bidaki, M. Z., & Hekmati, N. (2016). Incorporating E-learning in teaching English language
to medical students: exploring its potential contributions. Medical journal of the Islamic Republic
of Iran, 30, 462.
Lippert, C. R. (1993). Computer-based education and training in South Africa. Pretoria: J.L. Van Shaik
Academic.
Prensky, M. (2001). Digital natives, digital immigrants part 1. On the horizon, 9(5), 1-6.
Reigeluth, C.M., Watson, W.R., Watson, S.L., Dutta, P., Chen, Z., & Powell, N.D.P. (2008). Roles for
technology in the information-age paradigm of education: Learning management systems.
Educational Technology, 48, 32–39.
Roschelle, J. (2003). Learning by collaborating: conceptual change. The Journal of the Learning Sciences, 2,
235-276.
Simonson, M., Smaldino, S., Albright, M., & Zvacek, S. (2006). Teaching and learning at a distance:
Foundations of distance education (3rd ed.). Upper Saddle River, NJ: Prentice Hall.
Taylor, J.S. (2004). An analysis of the variables that affect technological literacy as related to
selected Technology Student Association activities (Unpublished doctoral dissertation). North Carolina
State University.
Watson, W.R., Lee, S., & Reigeluth, C.M. (2007). Learning management systems: An overview and roadmap
of the systemic application of computers to education. In F.M.M. Neto & F.V. Brasileiro (Eds.),
Advances in computer-supported learning (pp. 66–96). London: Information Science Publishing.
Watson, W.R., & Watson, S.L. (2007). An argument for clarity: What are learning management systems,
what are they not, and what should they become? TechTrends: Linking Research and Practice to
Improve Learning, 51, 28–34.
Yu, B., & Shen, H. (2012). Predicting roles of linguistic confidence, integrative motivation and second
language proficiency on cross-cultural adaptation. International Journal of Intercultural Relations,
36(1), 72-82.
Zanjani, N., Nykvist, S. S., & Shlomo, G. (2013). What makes an LMS effective: a synthesis of current
literature. Paper presented at the Proceedings of CSEDU 2013–5th International Conference on
Computer Supported Education.

222

View publication stats