Pegem Journal of Education and Instruction, Vol. 11, No. 4, 2021 (pp.

126-137)

RESEARCH ARTICLE WWW.PEGEGOG.NET

Automated Writing Evaluation (AWE) in Higher Education:

Indonesian EFL Students’ Perceptions about Grammarly
Use across Student Cohorts
Delsa Miranty1, Utami Widiati2*
1
Graduate Program in ELT, Universitas Negeri Malang, alan Semarang 5, Malang, East Java, Indonesia
2
Universitas Negeri Malang, +62341-551312, Jalan Semarang 5, Malang, East Java, Indonesia

A b s t r Ac t
Automated Writing Evaluation (AWE) has been considered a potential pedagogical technique that exploits technology to
assist the students’ writing. However, little attention has been devoted to examining students’ perceptions of Grammarly use in
higher education context. This paper aims to obtain information regarding the writing process and the merits and drawbacks
of Grammarly use among Indonesian undergraduate EFL students. A hundred (n=100) students majoring in English education
from a public university in Banten Province were involved in this research. They were divided into three groups of users, i.e.,
first-, second-, and third-year student cohorts, to test whether the frequency of using Grammarly can affect their perceptions of
Grammarly use in a writing class. A questionnaire and an interview guide were used to obtain the data. While the questionnaire
results were analyzed using SPSS version 20, the interview results were coded and categorized based on common themes. The
findings showed that there is no difference among student cohorts in their perceiving that the use of Grammarly was considered
necessary to compose and revise their writing because they still dealt with several writing constraints. They thought they
got immediate and comprehensive feedback, notifications of errors, and suggestions to revise the errors. Furthermore, the
frequency of using Grammarly, as viewed from the student cohorts, may affect their perceptions of usefulness and drawbacks
of Grammarly use, especially whether Grammarly’s feedback is always helpful or not. Examining the opportunities of using
Grammarly in sparking a constructive learning atmosphere in EFL writing class is worth-researching further.
Keywords: Automated Writing Evaluation (AWE), Grammarly use, Indonesian EFL writing students

IntroductIon their capacity in composing writing from the results of the

corrective feedback. They can also maintain their writing
The interests among scholars in researching the ways corrective
performance from the reflection on the feedback given by
feedback is informed when evaluating English as a foreign
the teachers. The benefits of corrective feedback have been
language (EFL) students’ academic writing have partly been
empirically justified by previous scholars (e.g., Isnawati,
motivated by the importance of grammatical accuracy in
Sulistyo, Widiati & Suryati, 2019; Wang, Shang, & Briody, 2013;
academic writing. Grammatical accuracy helps students
Yang, 2010), asserting that feedback can influence students’
convey the intended meaning by considering the grammar
writing quality. The study by Isnawati et al. (2019) revealed
forms, which will help determine the quality of the whole text.
that when teacher corrective feedback was combined with
The forms might include tenses, word choices, word order,
a teacher-student conference, the students under the study
subject-verb agreement, punctuations, and spellings. The
had a greater mean score of writing performance, implying
importance of performing an adequate grammar accuracy
level is also articulated in the Common European Framework
of Reference (Council of Europe, 2001) under the linguistic Corresponding Author e-mail: [email protected]
repertoire category to succeed in the meaning-making https://orcid.org/0000-0002-8603-4556
process. Azar (2007) highlights that grammatical accuracy How to cite this article: Miranty D, Widiati U, (2021). Automated
might contribute to the increased intelligibility in dealing Writing Evaluation (AWE) in Higher Education: Indonesian EFL
with the four English skills, including writing. Cavaleri and Students’ Perceptions about Grammarly Use across Student
Cohorts. Pegem Journal of Education and Instruction, Vol. 11,
Dianati (2016) also argued that quality academic writing is
No. 4, 2021, 126-137
influenced by grammatical accuracy. It can maintain the
clarity and precision of the writers’ intended ideas expressed in Source of support: Nil
the writing. Hence, providing corrective feedback to improve Conflict of interest: None.
the students’ writing appears to be an endless and essential DOI: 10.47750/pegegog.11.04.12
discourse. Received: 23.05.2021
Corrective feedback serves as the gatekeeper of the quality
Accepted: 03.08.2021 Publication: 01.10.2021
of students’ writing. The students can recognize and measure
 Automated Writing Evaluation (AWE) in Higher Education:

a possibility of positive impacts of teacher written corrective Debates on the use of AWE tools remain in the context
feedback combined with a teacher-student conference. of L2 learning. Some scholars (Bai & Hu, 2017; Kern &
Recent rapid technological changes have affected how Warschauer, 2000; Rezaei & Lovorn, 2010; Wang, Shang, &
corrective feedback can be provided, especially in writing Briody, 2013; Ware & Warschauer, 2006; Zhang & Hyland,
classes. Lim and Phua (2019) stated that digital tools can 2018) consider AWE a helpful means of assessing the students’
improve the students’ ability to write well and promote writing. The automated rating engines allow the students
constructive feedback in the learning context. Remarkably, to get immediate scores and written corrective feedback.
the technological development in this globalized world has The scores and feedback are considered more objective
triggered teachers, especially in the higher education context, and consistent than human raters. The feedback can be
to utilize automated writing evaluation (AWE) tools to execute quantitative and qualitative. Therefore, AWE tools can elevate
written corrective feedback. students’ motivation and raise their awareness to do several
revision stages. It is also designed to be consistent over time
AWE in L2 Classrooms: Potentials and Challenges so that the evaluation process can be less time-consuming.
By definition, automated writing evaluation (AWE) assesses Alternatively, the students can use the score and feedback
written texts using automatic rating engines developed by to have self-reflection and self-revision; in other words, they
computational linguistics (Wilson & Andrada, 2016). The can be trained to develop their autonomy. The opportunity
assessment may range from lexical, syntactic, and grammatical to do self-reflection and self-revision through AWE tools can
levels to semantic and discourse levels (Chen & Cheng, 2008). also lessen their writing anxiety because it is self-paced and
The inception of AWE was in the 1960s, widely known as Page personalized. The personalization of the evaluation process
Essay Grade (PEG). The program applied multiple regression makes the students not worry about negative judgment from
analysis to measure the writing quality using previously either teachers or peers. From the teachers’ perspectives, AWE
hand-scored essays as the referred texts (Shermis, Mzumara, tools can help teachers provide corrective feedback for the
Olson, & Harrington, 2001). Then, Educational Testing Service students. Detailed evaluation results from AWE tools enable
(ETS) collaborated with Vantage Learning developed e-rater the teachers to interact with the students to deal with the
and Intellimetric in the 1990s. Both scoring programs were revision process. While the tools can inform the form-focused
considered more thorough in evaluating the lexical, syntactic, feedback such as grammatical accuracy and mechanics, the
and discourse aspects of writings (Elliot & Mikulas, 2004). teachers can devote their time and efforts to evaluate the text-
Pearson Knowledge Technologies also purchased Intelligent focused feedback such as content and organization.
Essay Assessor to score essays. What differentiates this However, doubts about the use of AWE tools to foster
program from other AWE programs is that this program uses learner autonomy remain on the discussion. They were found
latent semantic analysis. The semantic meaning of the analyzed not to help develop the students’ attitudes toward learner
texts is compared to a corpus under a similar topic. Nowadays, autonomy. Cheng and Cheng (2008) asserted that the AWE
several AWE tools have been used to rate and evaluate different tools did not necessarily foster the students’ autonomy because
forms of written genres, such as Criterion and My Access the feedback was not concerned with content. Hence, the
(Li, Link, & Hegelheimer, 2015). Previously, AWE tools were negotiation process and interaction about the feedback might
used for high-stakes testing, such as summative tests (Cotos, be even discriminating against the students who are not
2014; Stevenson & Phakiti, 2019). In line with technological familiar with the technology. Consequently, the motivation
advancement and the importance of technology in providing to improve their writing can be less evident. The independent
quick and thorough corrective feedback, many AWE tools use of the tools was also questioned because the students’
have been used as an alternative pedagogical technique in L2 writing would be unauthentic. The students tended to rely
writing classrooms. on the feedback given by the tools much. Inspired by these
At least, there are two essential components of AWE. First, inconclusive previous research findings, the present study
AWE tools generate automated scores. They are designed to demystifies potential benefits of AWE tools for EFL writing
provide quick and thorough evaluation results of the analyzed instruction.
texts so that the evaluation process is not time-consuming.
Teachers can use the scores as a source of reflection to decide Grammarly in L2 Classrooms
on future actions. Second, AWE tools provide automated, As a form of AWE, Grammarly is the center of attention
written corrective feedback (Ranalli, 2018). The students can in academic writing in the higher education context. It is
use the feedback to revise their papers autonomously, but the an American technology company in the English language
teachers’ role is still vital to clarify the feedback. Such a teacher digital writing tool, developed by the original Ukrainian
role implies the benefits of implementing teacher-student authors Alex Shevchenko, Max Lytvyn, and Dmytro Lider
conferences as explored by Isnawati et al. (2019). in 2009. This tool combines artificial intelligence and natural

Pegem Journal of Education and Instruction, ISSN 2146-0655 127

 Automated Writing Evaluation (AWE) in Higher Education:

language processing under the machine and deep learning echoed the previous research in that using Grammarly
algorithms to provide real-time, quick, and comprehensive encouraged the users to activate independent learning and
writing evaluation results. Grammarly’s products offer self-revision process in their writing. This all highlights writing
not only grammar checking but also spell checking and as a process.
plagiarism detection. Now, six products have been launched: However, the Grammarly tool is not without flaws. This
The Grammarly Editor, Browser Extensions, Grammarly tool seems to lack comprehensive results, especially related
for Microsoft Office®, Grammarly for Your Desktop, The to the quality of the content and organization. It cannot
Grammarly Keyboard, and Grammarly for iPad. Generally, capture the coherence level between sentences and paragraphs.
the tools are intended to help people strengthen their writing Misleading feedbacks were also noted in Nova’s (2018) study
abilities and express ideas clearly and correctly (Grammarly, in which the students identified inaccuracies because the
2020). In Indonesia, the use of Grammarly is not obligatory results did not conform to their intention. Grammarly’s
especially in the context of higher education. The universities over-checking feature is another challenge because it may
are given a discretion to determine automated writing consider the reference writing incorrect, mostly the authors’
evaluation tools beneficial for their language courses. Hence, names. As Nova (2018) found out, this tool’s most noticeable
Grammarly seems not immensely used, considering no specific weakness is that Grammarly can only approve American
regulation for such purposes. English to automatically consider British English, particularly
A considerable body of research has examined the merits the spelling incorrect.
and drawbacks of using Grammarly tool in EFL writing Moreover, doubts also remain on whether this technology
classes. Regarding the merits of using Grammarly in writing is beneficial for actualizing meaningful learning. Interaction,
classes, Japos (2013) stated that Grammarly could substantially discussion, and collaborative reflection as vital elements in the
decrease grammar errors. Likewise, Cavaleri and Dianati writing process seem to be not adequately addressed using
(2016) argued that Grammarly was perceived by the students Grammarly since AWE programs are designed to be used as
as a valuable, helpful, and easy-to-operate AWE tool. It helped self-study and self-paced learning media. In other words, it
them understand grammar rules. Grammarly allows the lessens the opportunity for the students and their teachers to
students to understand the errors and the explanation; they talk about the feedback provided by the tool as a critical factor
can consider the suggestions given by the tool to revise their in developing writing quality and second language acquisition
writing. Darayani, Karyuatry, and Rizqan (2018) and O’Neill (Zhang & Hyland, 2018).
and Russell (2019) justified the previous research that the Although the integration of Grammarly in writing
students perceived Grammarly as a powerful tool enabling classes has been quite extensively examined, little attention
user writing to be easily checked for grammar possible stylistic is devoted to examining this issue in the Indonesian higher
errors. Their study also revealed that the students were more education context. There is no obligation for the university or
pleased with the feedback from Grammarly than that from higher education institution in Indonesia to use AWE tools
teachers. Hence, writing quality can be increased. This positive such as Grammarly because it is at the teacher’s discretion.
effect of using Grammarly was reported in several studies It is also not compulsory for the students to use AWE
(El-Ebyary & Windeatt, 2010; Li, Link, & Hegelheimer, 2015; tools such as Grammarly before submitting writing tasks
Liao, 2015; Parra & Calero, 2019; Wang et al., 2013). The to the faculty. Grammarly’s incorporation might be new
significance of using Grammarly in developing the students’ for some undergraduate students because their writing is
attitudes and behaviors was also evident. usually checked manually by their teachers. Raised by this
Cavaleri and Dianati (2016) found that the students were circumstance, the efforts to delve into the students’ perceptions
willing to continue using the tool. Ventayen and Orlanda- of using Grammarly in writing classes can help evaluate the
Ventayen (2018) unveiled that the students could get valuable usability of the tool. Additionally, the results can reflect the
lessons about grammatical errors so that their grammatical students’ attitudes toward Grammarly use so that teachers can
awareness could lead toward a better revision process. It is make evaluative actions to maintain the usefulness in writing
corroborated by Parra and Calero (2019) that the students’ classes. In short, this study seeks to explore the perceptions of
understanding of the errors increased. As a result, their writing Indonesian undergraduate EFL students from different cohorts
accuracy can be enhanced autonomously. In the Indonesian after using Grammarly in their writing class. The following
context, Ghufron and Rosyida (2018) examined Grammarly’s research questions drive the present study.
usefulness by conducting a quasi-experimental study involving
40 third-semester EFL students. The results showed that in Research Questions
addition to the reduced errors in diction, grammar, spelling, • RQ 1: Is there any difference between the students across
and punctuation, Grammarly could enhance the students’ cohorts regarding their perceptions about the writing
motivation to be autonomous learners. Nova’s (2018) study process?

128 Pegem Journal of Education and Instruction, ISSN 2146-0655

 Automated Writing Evaluation (AWE) in Higher Education:

• RQ 2: Is there any difference between the students across Table 1: Description of the questionnaire items.
cohorts regarding their perceptions about the use of Number of items Type Domain
Grammarly in the writing class? Five items 5-point Students’ perceptions about the
Likert writing process
Method Four items scale Students’ perceptions about the
Research Design usefulness of Grammarly in writing
classes
This study employed a quantitative approach in obtaining a
Six items Students’ perceptions about the
numerical description of the students’ perceptions about the drawbacks of Grammarly in writing
use of Grammarly in writing classes. Inferential statistical classes
results were also brought to reveal the differences between
groups of students. Additionally, a qualitative approach was questionnaire items are valid and moderately reliable. Then,
used to examine further the students’ more profound and an interview guide was formulated based on the questionnaire
broader perspectives regarding the use of Grammarly as a form items to obtain more comprehensive data.
of AWE tools. It appears crucial to understand whether there
are variations of perceptions among the students of different Data Collection
year cohorts. The data collection procedure started with getting the students’
consent to follow the overall research process. Then, the
Study Group questionnaire was sent to the students using Google Forms.
The study involved at first 133 undergraduate students (students Out of 133 students, 100 students resent the questionnaire
of the frst year = 51 students; second year = 51 students; third completely, comprising 40 from the first year, 38 from the
year = 31 students) from the English education department of second, and 22 from the third. The remaining thirty-three
a public university in Banten Province, Indonesia. Accessibility questionnaires were then discarded from the analysis.
was the main reason for involving those research participants. Fifteen students who returned the questionnaire sheets were
This study compared the perceptions of different student further involved in the semi-structured interview sessions.
cohorts, that is, to examine whether there were significant The students were selected based on the categorization of
differences among the students who used Grammarly for the their GPA (1.00-4.00): lower-achievers (2.95-3.45), middle-
first time and those who had already used it before, and among achievers (3.46-3.71), and higher-achievers (3.72-4.00). Five
those who used Grammarly purposefully in a writing course. students from each category were invited to participate in the
While the first-year students experienced using Grammarly for interview sessions.
the first time, the second and the third-year students may have
experienced using such an AWE tool for more than one time. Data Analysis
The data analysis procedure started with the responses to the
Data Collection Tools questionnaire items. The results were calculated descriptively
Two instruments were employed to obtain adequate data using SPSS version 20 to obtain the frequency and the mean
to address the research questions. A 5-point Likert-scale score of each questionnaire item. The mean scores were then
questionnaire was adapted from Cavaleri and Dianati (2016); used to determine three categories regarding the frequency of
the questionnaire consisted of fifteen items. Table 1 depicts Grammarly use as a learning strategy. A Strategy Inventory
the details of the questionnaire items. The questionnaire was of Language Learning (SILL) developed by Oxford (1995)
formulated initially in English for two reasons. First, the was employed to divide the mean scores of the questionnaire
English education department students already had basic items into three categories: high, medium, and low levels
English knowledge to understand the questionnaire items. of Grammarly use, as depicted in Table 2. Based on the
Second, as the questionnaire was adapted from Cavaleri’s categorization from SILL, the categories identified in the
and Dianati ‘s (2016) study, the English-version questionnaire analysis were divided into three levels: (1) High level of
items were preferred to avoid losing the original meaning Grammarly use (HG); (2) Medium level of Grammarly use
from the primary source. The validity and reliability tests of (MG); and (3) Low level of Grammarly use (LG).
the questionnaire items were performed by using SPSS 20. Furthermore, a one-way ANOVA test was run to measure
The Pearson correlation coefficient value is above the r table the extent to which each item of the questionnaire responses
.194 with the significance level 0.05 (two-tailed) for the items was different within each group of the students or between
with either positive or negative meaning. The Cronbach’s the three groups, i.e., first, second, and third years. The mean
Alpha value is .767 for the items with positive meaning and scores of each item were recalculated. Here are the hypotheses
.461 for the items with negative meaning. It means that the to be tested.

Pegem Journal of Education and Instruction, ISSN 2146-0655 129

 Automated Writing Evaluation (AWE) in Higher Education:

• H0: There is no difference of perceptions between and including the mean scores and standard deviation values of
within first-, second-, third-year student groups about the each questionnaire item.
use of Grammarly in the writing class. The results showed that the students from all groups
• H1: There is a difference of perceptions between and within perceived proofreading service as not considerably essential
first-, second-, third-year student groups about the use of factor in the writing process (X = 2.41, ). They still preferred
Grammarly in the writing class. feedback from others to assist them to write in English.
After the questionnaire responses were processed using However, based on the results of questionnaire item 5, the
descriptive and inferential statistical measurements, the students still believed that the feedback given by their lecturers
researchers analyzed the interview transcripts to enrich the was understandable enough in the writing class (X = 2.61, ). It
quantitative data. The researchers manually analyzed the might imply that they needed a tool to offer feedback in their
interview transcripts. First, the interview audio recordings writing class, particularly their papers’ grammar. As perceived
were transcribed. Because the interview sessions were by the students, the importance of proofreading service was
conducted in English, the transcript was not edited unless associated with the students’ writing difficulties. Most of
there were severe grammatical errors that influenced its clarity. the students also agreed that their vocabulary and grammar
Second, the transcript was coded verbatim. Third, the coding knowledge was still not excellent (X = 3.47, and expressing
results were categorized into several groups to corroborate thoughts in writing was not an easy task (X = 3.37, , regardless
the questionnaire results to address the research questions. of their year level. The students’ need to get assistance was
A member checking was conducted by having the transcripts related to the fact that they did not have adequate confidence
proofread and reviewed by the respondents to avoid biased in formulating correct sentences X = 3.49.
meaning and completeness. The respondents agreed that the The ANOVA test results revealed that the significance value
transcripts were already eligible to be analyzed and presented. for each of the five questionnaire items is lower than α=0.05
(Item 1 .994; Item 2 .365; Item 3 .343; Item 4 .565; Item 5 .591).
Findings The results reinforce that they find the teacher’s assistance in the
• RQ 1: Is there any difference between the students across writing process is crucial for them. They are still not confident
cohorts regarding their perceptions about the writing enough with their vocabulary and grammar knowledge as well
process? as their ability to express ideas in their writing, regardless of
Table 3 depicts the questionnaire results regarding the their year level. They also perceived that the teacher’s feedback
students’ perceptions of themselves in the writing process, on their writing is understandable and easy to follow.
The fact that most of the students were still not excel at
Table 2: Categorization of Grammarly Use. grammar and vocabulary was corroborated by the following
Category Indicator Interpretation interview excerpts.
High 4.5 to 5.0 Always or almost always used My English grammar and vocabulary are not too weak and
3.5 to 4.4 Usually used not too advanced. It’s kind of medium level. Sometimes, I’m
Medium 2.5 to 3.4 Sometimes used confused to use them. It’s all based on what I want to express.
(A third-year student)
Low 1.5 to 2.4 Generally not used
Sometimes, I am confused to express my ideas in writing
1.0 to 1.4 Never or almost never used
because my weaknesses in English Grammar. So, I think very

Table 3: The Students’ Perceptions of Statements about Themselves in the Writing Process.
Group A Group B Group C
Statement Mean SD Mean SD Mean SD X Mean X SD
1. I do not need any help with writing in English; 2.40 .97 2.42 .93 2.40 .79 2.41 .90
I just need a proofreading service
2. My knowledge of English grammar and 3.57 .65 3.33 .85 3.50 .56 3.47 .70
vocabulary is weak
3. I do not always feel confident that I have 3.31 1.05 3.54 .75 3.62 .87 3.49 .90
written the correct sentences
4. I am fine with English grammar, but I find it 3.42 1.06 3.21 99 3.46 1.07 3.37 1.04
difficult to express my ideas in writing
5. I do not always understand the feedback I 2.71 .89 2.48 .83 2.62 1.03 2.61 .91
get in my writing
Note: Group A: 1st year, Group B: 2nd year, Group C: 3rd year

130 Pegem Journal of Education and Instruction, ISSN 2146-0655

 Automated Writing Evaluation (AWE) in Higher Education:

Table 4: The One-Way ANOVA Test Results.

Sum of Squares df Mean Square F Sig.
Item 1 Between Groups .011 2 .005 .006 .994
Within Groups 80.179 97 .827
Total 80.190 99
Item 2 Between Groups 1.005 2 .503 1.018 .365
Within Groups 47.905 97 .494
Total 48.910 99
Item 3 Between Groups 1.765 2 .883 1.081 .343
Within Groups 79.225 97 .817
Total 80.990 99
Item 4 Between Groups 1.255 2 .627 .574 .565
Within Groups 106.055 97 1.093
Total 107.310 99
Item 5 Between Groups .905 2 .452 .529 .591
Within Groups 82.885 97 .854
Total 83.790 99

important for mastering English Grammar before you start the tool. These questionnaire items were under the category
writing. (A third-year student) of high level.
Some of the students reported that their grammar skills Table 6 depicts the one-way ANOVA test results of the
and vocabulary mastery were not advanced enough. Both questionnaire items. The results revealed that the significance
weaknesses affected their ability to write in which sometimes value of each of the questionnaire items presented before was
they felt confused at using both language aspects in their lower than α=0.05 (Item 6 .975; Item 7 .447; Item 8 .102; Item
writing well. They also thought that mastering English 9 .999; Item 14 .258; Item 15 .911). These findings imply that
grammar was of great importance when dealing with writing generally the students perceived usefulness of Grammarly
activity. regarding its details of feedback to understand grammar rules
• RQ 2: Is Ttere any difference between the students across and suggestions to revise the writing. However, their perceived
cohorts regarding their perceptions about the use of usefulness of the explanations of the errors given by such AWE
grammarly in the writing class? tool still varies within or across the three groups, i.e., first-,
Table 5 displays the questionnaire results regarding the second-, and third-year student groups. In other words, the
use of Grammarly. The students agreed that Grammarly was “high category” of Grammarly use may not automatically result
not difficult to operate. In particular, using Grammarly could in conformed perceptions among the students of the usefulness
assist their writing process because the AWE tool gave them of Grammarly in this case.
detailed feedback (X = 3.82, ). They also found that Grammarly The interview results corroborated the perceived usefulness
provided explanations of the writing errors, not only the feedback of using Grammarly in that the students experienced the
(X = 3.97,). As a result, it was no surprise that the students perceived benefits in writing class. They said that they used Grammarly
Grammarly as a helpful tool to understand the grammar rules ( to help them overcome problems related to the projects and
X = 3.88,), leading to the positive perception among them about materials provided in the writing class, support them by
the role of Grammarly’s suggestions to revise and improve their automatically correcting their paper, check their writing
works (X = 4.00,). All in all, the usefulness of using Grammarly assignments, and improve their writing performances.
was quite evident among the students (X = 3.65, ). To be honest, for the first time I used Grammarly, I was
The mean scores were further linked with SILL to shocked. Then, I decided to continue using Grammarly because
determine the frequency of using Grammarly per item. Table for me it is very good and very quick to give corrections on my
2 in the method section, the questionnaire items number 6, 7, writing. (A first-year student)
8, 9, 14, 15 indicate that their considerably positive perceived Grammarly can be installed on our smartphone, and
usefulness of using Grammarly was because they usually used we can change our keyboard with Grammarly so that it can

Pegem Journal of Education and Instruction, ISSN 2146-0655 131

 Automated Writing Evaluation (AWE) in Higher Education:

Table 5: Students’ Perceptions about Ways that Grammarly is Helpful.

Group A Group B Group C
Statement Mean SD Mean SD Mean SD X Mean X SD
6. Grammarly gives detailed feedback 3.80 .79 3.81 .80 3.84 .80 3.82 .79
7. Grammarly makes helpful suggestions for improving my work 4.05 .72 3.87 .69 4.06 .56 4.00 .66
8. Grammarly gives good explanations about my errors 3.88 .67 3.84 .83 4.18 .53 3.97 .70
9. Grammarly has helped me understand grammar rules 3.88 .79 3.87 .89 3.87 .79 3.88 .81
14. I do agree that student get usefulness with the use of 3.45 .88 3.81 .80 3.68 1.02 3.65 .91
Grammarly in writing class.
15. Grammarly is easy to use, especially in writing class. 3.65 .83 3.75 1.06 3.71 .99 3.71 .95

Table 6: The One-Way ANOVA Test Results.

Sum of Squares df Mean Square F Sig.
Item 6 Between Groups .032 2 .016 .025 .975
Within Groups 62.728 97 .647
Total 62.760 99
Item 7 Between Groups .724 2 .362 .812 .447
Within Groups 43.276 97 .446
Total 44.000 99
Item 8 Between Groups 2.250 2 1.125 2.338 .102
Within Groups 46.660 97 .481
Total 48.910 99
Item 9 Between Groups .002 2 .001 .001 .999
Within Groups 66.558 97 .686
Total 66.560 99
Item 14 Between Groups 2.280 2 1.140 1.374 .258
Within Groups 80.470 97 .830
Total 82.750 99
Item 15 Between Groups .175 2 .087 .094 .911
Within Groups 90.415 97 .932

automatically correct my typing. It can be used anytime and check you get with most word processors. (A third-year
anywhere we want, including in our writing class. (A third- student)
year student) I think the user can’t rely on this application over and
Grammarly corrects my writing and I can learn from it over again. It only addresses common mistakes. So, learning
what us the correct sentence structure. (A second-year by ourselves would be better than by using an AI assistance.
student) (A third-year student)
Meanwhile, the difference between the student cohorts in The excerpts above demonstrated that the first exposure to
perceiving the helpfulness of Grammarly was corroborated Grammarly experienced by the students in the second semester
by the following excerpts. made them feel surprised. In a more comprehensive way, one of
To be honest, in the first time I used Grammarly, I was the students in the fourth semester argued that Grammarly was
surprised. (A first-year student) not considerably helpful in terms of checking the plagiarism
Well, that’s a plus point from Grammarly. However, in rate of the content. As time goes by, the students perceived
my experience, sometimes Grammarly missed a few points that Grammarly was a more powerful tool, as compared
in terms of plagiarism. (A second-year student) to the average similar tools. This discrepancy may indicate
It provides you with the basics you need to lay down your the role of repetition of exposure to get the students more
writing foundation, spelling, grammar, and punctuation familiar with the use of Grammarly to assist their writing
checker, which is more powerful than the average spell process independently. What was also found from the student

132 Pegem Journal of Education and Instruction, ISSN 2146-0655

 Automated Writing Evaluation (AWE) in Higher Education:

Table 7: Students’ Perceptions about the Drawbacks of Grammarly.

Group A Group B Group C
Statement Mean SD Mean SD Mean SD X Mean X SD
10. The feedback is not always helpful 3.05 .76 2.57 .79 2.59 .75 2.75 .79
11. I do not agree with some of the suggestions 3.08 .50 2.72 .83 2.75 .76 2.86 .72
12. I cannot understand the explanations 2.68 .96 2.51 .75 2.56 .66 2.59 .80
13. I have technical issues with Grammarly 3.20 .83 3.03 .76 2.75 .84 3.00 .82

Table 8: The One-Way ANOVA Test Results.

Sum of Squares df Mean Square F Sig.
Item 10 Between Groups 5.085 2 2.542 4.277 .017
Within Groups 57.665 97 .594
Total 62.750 99
Item 11 Between Groups 2.752 2 1.376 2.708 .072
Within Groups 49.288 97 .508
Total 52.040 99
Item 12 Between Groups .530 2 .265 .404 .669
Within Groups 63.660 97 .656
Total 64.190 99
Item 13 Between Groups 3.430 2 1.715 2.577 .081
Within Groups 64.570 97 .666

cohort in the sixth semester was another student articulating 11 and 13 was slightly higher than α=0.05 (.072 and .081,
a contrasting view on Grammarly. The student suggested the respectively), indicating slightly similar responses between
users not relying much on that application. Instead, the student the three groups about their disagreement with either some
preferred learning by themselves to utilizing the AI-based suggestions provided by Grammarly or its ease of use. However,
writing correction tool to assist their writing process and there was a difference between the students across year levels
progress. This evidence indicates that the length of experience in terms of how Grammarly’s feedback was considered not
may not determine the conformity of perceptions among the always helpful because the significance value of questionnaire
students in the same cohort. item 10 was lower than α = 0.05 (.017). It means that more
Furthermore, Table 7 shows the students’ perceptions students were still in between whether they experienced the
about drawbacks related to Grammarly use. The questionnaire drawbacks or not.
results regarding the perceived drawbacks of using Grammarly
in writing class were more intriguing. Some of the students Discussion
generally viewed that despite the positive results of Grammarly’s This study aimed to explore the perceived usefulness and
perceived usefulness, they still experienced technical errors in drawbacks of Grammarly as an AWE tool among first-,
using the tool (X = 3.00, . The feedback was not always perceived second-, and third-year Indonesian undergraduate EFL
useful by some students (X = 2.75, . Also, some suggestions students. Regarding the first research question, “Is there any
were seen not in line with what they expected (X = 2.86, , or difference between the students across cohorts regarding their
some explanations were not understandable enough (X = 2.59, perceptions about the writing process?” there was no difference
. Apart from that, the results revealed that most students still in terms of perceptions of Grammarly use among students
perceived Grammarly as an easy-to-use and helpful AWE tool of different cohorts. The students under study perceived that
to obtain valuable feedback and explanation regarding the proofreading service is needed to produce good writing and
errors to refer to them in revising their works. that one of the uses of Grammarly is for proofreading services.
The ANOVA test results of the questionnaire items are The fact that there is no significant difference in mean across
showcased in Table 8. The results showed that the significance the three student cohorts indicates that length of learning
value of questionnaire item 12 was higher than α = 0.05 English in higher education settings may not influence the
(.669), representing no difference between the students across increased perceived self-efficacy in writing in English. It
cohorts in their perceptions about the clarity of Grammarly means that several students are already confident enough in
explanation. The significance value of questionnaire items writing in English without any assistance even a proofreading

Pegem Journal of Education and Instruction, ISSN 2146-0655 133

 Automated Writing Evaluation (AWE) in Higher Education:

service. They admitted the need for automated feedback the usefulness of Grammarly. The student recommends the
because they still dealt with several writing deficiencies, users to better learn from themselves, not relying much on the
such as lack of confidence and abilities to express their ideas software since it only corrects common writing errors. This
in a well-structured manner. They felt that their knowledge may counter the role of repetition of exposure toward increased
of English grammar and vocabulary seems not sufficient familiarity with the utilization of the software. It contrasts with
enough to write in English properly. However, all student Parra and Calero’s (2019) study that the students’ perception
cohorts do not considerably agree that the feedback given of the AWE tools, Grammark and Grammarly, is positive
by the lecturers is difficult to understand. This may imply toward the development of their writing skills to some extent.
that albeit understandable, the lecturers’ feedback seems One of the reasons of such discrepancy might be associated
not influential enough to encourage the students to improve with the students’ diverse learning styles or strategies in the
writing skills because the sixth-semester students still have a present study. The unfavored perception of Grammarly use
tendency for not being confident at their writing and grammar may also be connected with the value of interactions between
skills. Such circumstance again results in the need for help the students and the lecturers. Since the feedback given by
from a proofreading service, instead of relying merely on the Grammarly is based on the writing being analyzed, there is no
lecturers’ feedback. The reason for this can be associated with further explanation for the students of why the results can be
the clarity, depth, and comprehensiveness of feedback given so. This confirms Zhang and Hyland (2018) that opportunities
by the lecturers or the techniques employed to convey the for open interactions are considerably helpful for students
feedback to the students. to enhance their second language acquisition and writing
Regarding the second research question, “Is there any quality. Considering the diverse learning styles or strategies,
difference between the students across cohorts regarding the teachers should take into account various techniques to
their perceptions about the use of Grammarly in the writing deliver the feedback as well.
class?” the present study also showed that Grammarly use is The perceived less comprehensive correction provided
perceived useful because it provides practical explanations and by the software may be caused by the student’s preferences
suggestions for improving the writing among the students. The on certain types of corrective feedback. Focused and
one-way ANOVA test results demonstrated that the hypothesis comprehensive written corrective feedback needs to be
H0: There is no difference of perceptions between and within revisited because both strategies of giving feedback to students
first-, second-, and third-year student groups regarding the use still receive ample attention among scholars (see Lee, 2020
of Grammarly in the writing classroom was accepted regarding for a more comprehensive review on this issue). She further
the students’ perceptions of the usefulness of Grammarly use. recommends the possibility of combining focused and
The students’ responses further explain that the common comprehensive WCF. On the one hand, the time allocation in
categories of errors provided by Grammarly encompassed the classroom can indeed influence the teachers to focus on
spellings and punctuations, subject-verb agreement, and major issues in the writing. On the other hand, the students’
structure, as justified by Daniels and Leslie (2013). In other needs may contradict to the teachers’ focused WCF given. The
words, the students across cohorts generally view Grammarly present study accords with Lee’s (2020) final thought that in an
as useful automated writing correction tool because they can authentic writing classroom, addressing what really matters
get explicit results of the grammar errors contained in their according to the students’ needs in an authentic writing
writing. The findings corroborated Salteh and Sadeghi (2010) classroom is essential.
that the students preferred explicit error corrections. Grammarly gave the explanations and corrective feedback
However, differences were identified in which one of the that the students can use as self-paced learning resources. This
students from the 2nd-semester student cohort feels shocked finding confirms Dodgson et al. (2016) that using English
because it is the first time to use Grammarly. Quite similarly, Language Learning Websites as learning materials helps
a student from the 4th-semester student cohort views such students cope with indirect corrective feedback because they
AI-based AWE tool not considerably helpful in terms of need references to the learning process’s corrective feedback.
identifying plagiarism rate. Contrastingly, the students from Qassemzadeh and Soleimani (2016) also argue that the
the 6th-semester student cohort perceive Grammarly as a feedback from AWE tools can significantly affect learning
helpful tool to understand the basic writing foundation, passive forms. The feedback has a statistically significant
albeit not mentioning the issue of plagiarism rate. This influence on the retention of passive forms from Grammarly
discrepancy may indicate the role of repetition of exposure to and teachers among Iranian EFL students.
get the students more familiar with the use of Grammarly to Another benefit perceived by the students is the easy-to-
assist their writing process independently. However, another operate feature of Grammarly. Cavaleri and Dianati (2016)
intriguing perception is that one student from the same found in their study that more than 80% of the participating
6th-semester student cohort does not considerably agree with students give a positive evaluation of the tool. The reason is

134 Pegem Journal of Education and Instruction, ISSN 2146-0655

 Automated Writing Evaluation (AWE) in Higher Education:

that Grammarly can be used as either a stand-alone website Moreover, the evaluation results are sometimes perceived
or an add-on in several information-processing tools such as as not satisfactory by some of the students. One example is
Word, Search Engine, and educational websites. Grammarly noticed in Nova’s (2018) study, where Grammarly, to date, is
is realistic, which has colors that make the students recognize not capable of catering for varieties of English, only American
the errors in their writing. The automated written corrective English spellings and punctuations. It might be caused by the
feedback also helped them understand their mistakes. In short, complex nature of writing and the variety of styles in using
it was beneficial to decide what to revise and what to write written English, which cannot be comprehensively evaluated
from the feedback. Like an AI-based AWE tool, Grammarly by AI tools. This domain is beyond forms and semantics
was considered easy-to-operate because the students could so that it needs the teacher’s role to handle this weakness.
use it anytime and anywhere. Hence, most of the three groups Hence, interaction and discussion between students and their
accepted Grammarly use to assist their learning and writing teachers can foster a festive learning atmosphere, contributing
processes. to meaningful students’ writing class engagement, in line
Nova (2018) justified the perceived ease of use of Grammarly with the study by Isnawati et al. (2019). Although students’
that the students can get immediate access to and monitor the engagement with written corrective feedback has been
feedback in their accounts to save more time to self-edit and explored quite extensively (see Zhang, 2017; Zheng & Yu, 2018),
self-revise their works. In sum, Grammarly use in a writing this issue remains underexplored in the context of the current
class for higher education is perceived quite well because it development of AWE such as Grammarly.
helps the students get immediate and comprehensive feedback,
eases the access to the explanations of the errors, improves C o n c lu s i o n
knowledge, particularly about grammar, and provides some The data sets have delineated the Indonesian undergraduate
suggestions to revise the papers. The development of AWE tools EFL students’ perceptions of Grammarly use in writing
has allowed the students to understand what they want to learn classes. The findings reveal that the students across the year
and acquire the knowledge they need so that they know their levels agree regarding their perceptions of Grammarly as a
potentials and weaknesses in writing and in fulfilling their useful AWE tool because they are already aware of the need
own learning needs independently; in other words, the use to obtain proofreading services. Non-conformities, however,
of Grammarly as revealed in this study might stimulate EFL are still identified across the student cohorts or among the
students in writing classes to become autonomous. students within the same cohort. This study posits that the
However, several drawbacks of Grammarly are still length of experience in and exposure of using Grammarly
identified in the present study. Several students still seems not the only factor, which influences the students’
experienced technical errors in using Grammarly. This might perceptions of the usefulness of Grammarly. Apart from
be caused by the infrequent use of the writing class tool and all the merits of Grammarly as an AI-based AWE tool, the
the absence of learner training in using the technology. The drawbacks remain on the surface of discussion. The infrequent
questionnaire results also reveal that Grammarly’s feedback use of Grammarly, as viewed from the student cohorts, may
does not always help some students because the feedback might affect their perceptions about whether the feedback given by
be challenging to digest, not comprehensive enough, or not Grammarly is always helpful.
conform to their expectations. Previous research (Ghufron &
Rosyida, 2018; Nova, 2018; Parra & Calero, 2019) resonates with
Suggestion
the present study. Although Grammarly has been developed
to detect run-on sentences, dangling expressions, sentence The fact that Grammarly is not capable of detecting and
fragments, and difficult-to-understand sentences, such an evaluating unorganized content or ideas and of accommodating
AWE tool can still detect the organization, structure, or flow English varieties in this more globalized era raises a classical
of the ideas or content. However, H1: There is a difference of question on whether AWE can replace teachers’ roles to
perception between and within second-, fourth-, sixth-semester maintain interaction and enhance critical thinking in the
student groups regarding the use of Grammarly in the writing writing class. This issue needs to be further investigated.
classroom is accepted for the questionnaire item 10 (Sig.
value .017 < α=0.05). The second- and third-year students L imi tat i o n
disagree with the statement that Grammarly’s feedback is Considering the limitations of the present study, more rigorous
not always helpful, which differs from the first-year students. research can be taken into account to reveal the influence of
This difference might be caused by the intensity level in using students’ multidimensional attributes on their preferences
Grammarly in writing class so that the first exposure of and acceptance of using Grammarly in writing classes. More
Grammarly use to the first-year students may influence their qualitatively, future research can investigate teachers’ processes
views on its usefulness. and strategies in balancing their roles and those of Grammarly

Pegem Journal of Education and Instruction, ISSN 2146-0655 135

 Automated Writing Evaluation (AWE) in Higher Education:

in developing students’ knowledge of writing and grammar as r e s e a r c h e s . J PA I R I n s t i t u t i o n a l R e s e a r c h , 1(1) ,

well as raising their writing awareness. 97–109.
Kern, R., & Warschauer, M. (2000). Introduction. In M. Warschauer
References & R. Kern (Eds.), Network-based language teaching: Concepts
Azar, B. (2007). Grammar-based teaching: A practitioner’s and practice (pp. 1–19). London: Cambridge University Press.
perspective. TESL-EJ, 11(2), 1–12. Lee, I. (2020). Utility of focused/comprehensive written corrective
Bai, L., & Hu, G. (2017). In the face of fallible AWE feedback: How feedback research for authentic L2 writing classrooms. Journal
do students respond? Educational Psychology, 37(1), 67–81. of Second Language Writing, 49, 1-7. https://doi.org/10.1016/j.
https://doi.org/10.1080/01443410.2016.122327. jslw.2020.100734
Caveleri, M., & Dianati, S. (2016). You want me to recheck your Liao, H. (2015). Using automated writing evaluation to reduce
grammar? The usefulness of an online grammar checker as grammar errors in writing. ELT Journal, 70(3), 308–319.
perceived by students. Journal of Academic Language and Lim, F. V., & Phua, J. (2019). Teaching writing with language feedback
Learning. 10(1), 223–236. http://www.journal.aall.org.au/index.
technology. Computers and Composition, 54, 102518. https://
php/jall/article/view/393/246
doi.org/10.1016/jcompcom.2019.102518
Chen, C. F. E., & Cheng, W. Y. E. (2008). Beyond the design of
automated writing evaluation: Pedagogical practices and Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the role of
perceived learning effectiveness in EFL writing classes. automated writing evaluation (AWE) feedback in ESL writing
Language Learning & Technology, 12(2), 94-112. Retrieved instruction. Journal of Second Language Writing, 27, 1–18.
from https://www.researchgate.net/publication/45681611 Nova, M. (2018). Utilizing Grammarly in evaluating academic writing
Cotos, E. (2014). Genre-based automated writing evaluation for L2 A narrative research on EFL students’ experience. Premise:
research writing: From design to evaluation and enhancement. Journal of English Education and Applied Linguistics, 7(1),
New York, NY: Palgrave Macmillan. 80-96.
Council of Europe. (2001). Common European Framework of O’Neill, R., & Russell, A. (2019). Stop! Grammar time: University
Reference for Languages: Learning, teaching, assessment. New students’ perceptions of the automated feedback program
York, NY: Cambridge University Press.
Grammarly. Australasian Journal of Educational Technology,
Daniels, P., & Leslie, D. (2013). @ CUE Grammar Software Ready for
35(1), 42-56. https://doi.org/10.14742/ajet.3795
EFL Writers ? 391–401
Darayani, N. A., Karyuatry, L. L., & Rizqan, M. D. A. (2018). Oxford, R. (1990). Language learning strategies: What every teacher
Grammarly as a tool to improve students’ writing quality (Free should know. Rowley, MA: Newbury House.
online proofreader across the boundaries). Jurnal Sains Sosial Ranalli, J. (2018). Automated written corrective feedback: How well
dan Humaniora, 2(1), 83-89. can students make use of it? Computer Assisted Language
Dodgson, A. N., Tariq, B., Alauyah, M., & Yusof, M. (2016). The Learning. DOI: 10.1080/09588221.2018.1428994
secondary school students’ usage of English learning websites Rezaei, A.R., & Lovorn, M. (2010). Reliability and validity of rubrics
to self-correct writing errors. Asian TEFL, 1(11), 2503–2569. for assessment through writing. Assessing Writing, 15, 18–39.
http://doi.org/10.21462/asiantefl.v1i1.3 Salteh, M. A., & Sadeghi, K. (2010). Teachers’ and students’ attitudes
El-Ebyary, K., & Windeatt, S. (2010). The impact of computer-based toward error correction in L2 writing. The Journal of Asia
feedback on students’ written work. International Journal of TEFL, 12(3), 1-31. http://dx.doi.org/10.18823/asiatefl.2015.12.
English Studies, 10(2), 121–142. 3.1.1
Elliot, S. M., & Mikulas, C. (2004, April). The impact of MY access!™ Shermis, M. D., Mzumara, H. R., Olson, J., & Harrington S. (2001).
use on student writing performance: A technology overview Online grading of student essays: PEG goes on the World
Wide Web. Assessment & Evaluation in Higher Education,
and four studies. Paper presented at the Annual Meeting of the
26(3), 247–259.
American Educational Research Association, San Diego, CA.
Stevenson, M., & Phakiti, A. (2019). Automated feedback and second
Ghufron, M. A., & Rosyida, F. (2018). The role of Grammarly in language writing. In K. Hyland, & F. Hyland (Eds.). Feedback
assessing English as a foreign language (EFL) writing. Lingua in second language writing: Contexts and issues (pp. 125–142).
Cultura, 12(4), 395-403. (2nd ed.). New York, NY: Cambridge University Press.
Grammarly. (2020). About Grammarly. Retrieved from https:// Qassemzadeh, A., & Soleimani, H. (2016). The impact of feedback
support.grammarly.com/hc/en-us/categories/115000018611- provision by Grammarly software and teachers on learning
About-Grammarly passive structures by Iranian EFL learners. Theory and Practice
Isnawati, I., Sulistyo, G. H., Widiati, U., & Suryati, N. (2019). Impacts in Language Studies, 6(9), 1884. http://doi.org/10.17507/
of teacher-written corrective feedback with teacher-student tpls.0609.23
conference on students’ revision. International Journal of Ventayen, R. J. M., Orlanda-ventayen, C. C. (2018). Graduate students’
perspective on the usability of Grammarly® in one ASEAN state
Instruction, 12(1), 669-684.
university. Asian ESP Journal, 14(7.2), 9-30.
Japos, G. (2013). Effectiveness of coaching interventions using
Wang, Y-J., Shang, H-F., & Briody, P. (2013). Exploring the impact
Grammarly of using automated writing evaluation in English as a foreign
sof t wa re a nd plagia rism detection sof t wa re in reducing language university students’ writing. Computer Assisted
grammatical errors and plagiarism of undergraduate Language Learning, 26(3), 234–257.

136 Pegem Journal of Education and Instruction, ISSN 2146-0655

 Automated Writing Evaluation (AWE) in Higher Education:

Ware, P. D., & Warschauer, M. (2006). Electronic feedback Yang, Y.F. (2010). Students’ reflection on online self-correction and
and second language writing. In K. Hyland & F. Hyland peer review to improve writing. Computers & Education, 55,
(Eds.), Feedback in second language writing: Context and 1202–1210.
issues (pp. 104–122). New York: Cambridge University Zhang, Z. (2017). Student Engagement with computer-generated
Press. feedback: A case study. ELT Journal, 71(3), 317–328.
Wilson, J., & Andrada, G. N. (2016). Using automated feedback to Zhang, Z., & Hyland, K. (2018). Student engagement with the teacher
improve writing quality: opportunities and challenges. In and automated feedback on L2 writing. Assessing Writing, 36,
Rosen, Y., Ferrara, S., Mosharraf, M. (eds). (2016). Handbook 90–102. https://doi.org/10.1016/j.asw.2018.02.004.
of Research on Technology Tools for Real-World Skill Zheng, Y., & Yu, S. (2018). Student engagement with teacher-written
Development (pp. 678-703). IGI Global: US. DOI: 10.4018/978- corrective feedback in EFL writing: A case study of Chinese
1-4666-9441-5.ch026 lower-proficiency students. Assessing Writing, 37, 13–24.

Pegem Journal of Education and Instruction, ISSN 2146-0655 137