Proceedings of the International Conference on Sustainable Communication Networks and Application (ICSCNA 2023)

IEEE Xplore Part Number: CFP23DW8-ART; ISBN: 979-8-3503-1398-7

Grammar Error Correction using Natural

Language Processing
2023 International Conference on Sustainable Communication Networks and Application (ICSCNA) | 979-8-3503-1398-7/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICSCNA58489.2023.10370283

M .Geetha1 K.Vijaya2 Dr Subramanian K M 3

Assistant Professor Assistant Professor, Professor
Department of Computer Science Department of Computer Science Department of Computer Science
and Engineering and Design and Engineering
M .Kumarasamy College of Kongu Engineering College Shadan College of Engineering and
Engineering, Karur. Erode Technology,
[email protected] [email protected] Hyderabad
[email protected].

Dr Rajesh Kanna P 4 Dr Rajeshkumar G 5 Karthika S6

Assistant Professor, Associate Professor, Assistant Professor
Department of Computer Science Department of Computer Science Department of Artificial
and Engineering and Engineering Intelligence and M achine Learning
Bannari Amman Institute of Bannari Amman Institute of Bannari Amman Institue of
Technology, Technology, Technology
Sathyamangalam. Sathyamangalam. Sathyamangalam
[email protected] [email protected] [email protected]

Abstract–In the digital era, most of the An illustration of one of these grammar checkers
communication is happening through online available on internet is Grammarly. Correction of
platforms. English plays a vital role for better typographical errors can raise the caliber of writing in
understanding. Good grammar helps in framing a chats, blogs, and emails. An Instrument model is
better sentence. This demands for the development trained to accept an incorrect sentence as input and
of an automatic Grammar Error Correction (GEC) return a sentence that is grammatically accurate in the
tool. Many research works have been initiated to GEC task, which may be thought of as a sequence-to-
make Grammar Error Correction possible by using sequence task [1].
various machine learning algorithms. Some of the The errors can be of different types. Such as
algorithms are Naive Bayes, BERT, Support Vector
Machine, and ULMFIT. Among these algorithms ● Apostrophe Usage
BERT gives a comparatively best accuracy rate. The ● Missing Comma
fundamental goal of this research work is to ● Mixing up similar words
compare these algorithms with respect to accuracy ● Pronoun Disagreement
Here, NLP is used for analyzing the grammatical ● Comparison
corrections in the sentence. Further, this study can
● Prepositions
be extended to overcome the drawbacks faced by
the BERT algorithm such as increase in time There are enormous applications to this issue,
consumption for training and cost. the explanation being composing is an extremely
familiar means to share thoughts and data. This could
Keywords–Natural Language Processing, Bidirectional assist the essayist with accelerating their work with
Encipher Representations from Instruments, Universal extremely negligible possibility of blunder. In
Language Model Fine-Tuning, Grammar Error addition, there could be numerous people who are not
Correction. familiar with a specific language. In this way, these
sorts of models ensure that language isn't an
obstruction in correspondence.
I. INTRODUCTION
A desktop program's ability is to understand both
Normally, the Grammatical Error Correction the regular and human language [2]. Natural
(GEC) systems are designed to fix errors in the text. Language Processing drives desktop programs that

979-8-3503-1398-7/23/$31.00 ©2023 IEEE 141

Authorized licensed use limited to: KLE Technological University. Downloaded on May 04,2024 at 15:50:04 UTC from IEEE Xplore. Restrictions apply.
Proceedings of the International Conference on Sustainable Communication Networks and Application (ICSCNA 2023)
IEEE Xplore Part Number: CFP23DW8-ART; ISBN: 979-8-3503-1398-7

translate text beginning with one language then ● Word sense disambiguation
into the next language, answer spoken calls for, ● Normal language surmising
and summarize huge volumes of words rapidly and
● Opinion characterization
even logically. There's a respectable open door you've
spoken with Natural Language Processing such as A. Working Model
Global Positioning Systems, electronic accomplices, Instrument is a system that learns the logical
talk-to- message correspondence programming, client connections between instructions or sub instructions
support artificial conversation, and other customer in a text and is made used by BERT [4]. Instrument's
solaces. Notwithstanding, NLP similarly expects a essential plan comprises two free components: an
filling part in large business game plans that help with encipher that peruses the message input and a decoder
streamlining business errands, increase laborer that creates a task expectation. Just the encipher
productivity, and enhance vital business processes. system is required in light of the fact that BERT's
Human speech is stacked up with ambiguities that point is to create a language model. In a publication
make it verifiably testing to create programming that published by Google, the precise operation of the
unequivocally chooses the arranged meaning of text Instrument is described. The Instrument encipher
or words of voice data. Homonyms, homophones, peruses the whole grouping of words on the double, as
joke, axioms, resemblances, complement and use opposed to directional models, which read the text
exemptions, assortments in sentence structure — these input consecutively (from anticlockwise to clockwise
several the peculiarities of languages of human that or clockwise to anticlockwise). Despite the fact that it
require some speculation to grasp, but that would be more precise to depict it as non-directional, it
programmers ought to help ordinary language is thus considered bidirectional. This element
consumed applications to see and see definitively empowers themodel to grasp the setting of a word [5].
from the begin, accepting those applications will be In the Fig.1, BERT planning stage, the model sorts
more important [3,16]. out some way to predict whether the ensuing
sentence/request in a couple will come after another in
II. BERT ALGORITHM
the primary report by getting sets of sentences as data.
During planning, a big part of the information sources
Bidirectional Encipher Portrayals from instrument,
are matches in which the ensuing sentence is the
or BERT, is another paper from Google intelligence
accompanying one in the principal message, and in the
Language specialists. The essential innovative extra portion of, the following request is a haphazardly
headway of BERT is the use of Instrument's
chosen sentence from the assortment. The essential
bidirectional preparation, a popular consideration
assumption is that the following expression won't
model, to language demonstrating. Conversely, prior beassociated with the first. [6]
research saw text successions from either a clockwise-
to-anticlockwise or a joined clockwise-to-endlessly Prior to entering the model, the information is
anticlockwise-to-clockwise preparing viewpoint. handled as follows to help the model in separating
between the two orders during preparing:
BERT is as of now being utilized at Google to
streamline the translate of user forge questions. BERT ● Typically, the first request has a [CLS] token
succeeds at a few abilities that make this possible, at the source, and each forthcoming request has a
including: An array to-succession based language age [SEP] token at the objective.
undertakings, for example, ● All token has a request inserting that assigns
● Quiz responding to request A or request B. Token inserting with a
vocabulary of 2 and request implanting's portion a
● Theoretical synopsis
comparative thought.
● Sentence expectation
● All token acknowledges a directional
● Informal reaction age implanting to mean its position in the request.
Regular language understanding undertakings, for example, The instrument paper presents the hypothesis and
viability of dictionary inserting[7]
● Polysemy and Reference (words that sound
or appear to be identical yet have various
implications) goal

979-8-3503-1398-7/23/$31.00 ©2023 IEEE 142

Authorized licensed use limited to: KLE Technological University. Downloaded on May 04,2024 at 15:50:04 UTC from IEEE Xplore. Restrictions apply.
Proceedings of the International Conference on Sustainable Communication Networks and Application (ICSCNA 2023)
IEEE Xplore Part Number: CFP23DW8-ART; ISBN: 979-8-3503-1398-7

Training Vs Validation
Loss
0.6
0.5
0.4
0.3
0.2
0.1
1 2 3 4
Stag
e

T rainin Validatio
g n
Fig.1. Working of BERT
Fig.2 Collation of Training and Validation in terms
The impending measures are taken in grouping to of Loss using BERT
coordinate whether the following assertion is, as a matter
of fact, connected with the fundamental: From the below TABLE I. Collation of training
and validation of BERT is shown with loss in
● The instrument model cycles the full info validation and its percentage of validation with time.
request.
● Utilizing a basic gathering, the [CLS] token's Stag Loss in Loss in % of Durati Durati
result is changed into a 21 molded point.
e Traini Validati Validat on of on of
● Use Softmax to coordinate the ng on ion Traini Time
IsNextSequence possibility. Attaine ng
d
To limit the consolidated misfortune capability of
the two strategies, Covered LM and Next 1 0.50 0.40 0.83 00:50 00:02
Sentence Expectation are advanced mutually while
preparing the BERT model. [8], [9].
2 0.30 0.42 0.85 00:50 00:02
B. Results of BERT
In the below Fig.2 the results of BERT is shown 3 0.19 0.49 0.85 00:50 00:02
with respect to training Vs validation loss. The result
has been dissected as the secret state vector of pre-
4 0.12 0.58 0.85 00:50 00:02
characterized secret size compared to every token in
succession. These concealed states from the last layer
of the BERT are then utilized for different NLP Table I. Collation of training and validation of bert
undertakings.
Loss % III. ULMFIT ALGORITHM

ULMFIT, also known as Universal Language

Model Fine-tuning, is a compositional and moving
learning method that can be used for NLP applications
[10]
There are some highlights that permit it to be a
widespread way to deal with NLP learning:
● It works across undertakings shifting in
report size, number, and name type
● It utilizes a solitary engineering and
preparing process

979-8-3503-1398-7/23/$31.00 ©2023 IEEE 143

Authorized licensed use limited to: KLE Technological University. Downloaded on May 04,2024 at 15:50:04 UTC from IEEE Xplore. Restrictions apply.
Proceedings of the International Conference on Sustainable Communication Networks and Application (ICSCNA 2023)
IEEE Xplore Part Number: CFP23DW8-ART; ISBN: 979-8-3503-1398-7

● It requires no custom component designing b) Progressive Defrosting

or preprocessing
There is a chance of catastrophic forgetting when
● It doesn't need extra in-area records or all layers are adjusted simultaneously. Thus, initially
names. all layers except the last one are frozen and the fine-
A transfer learning method called ULMFIT can be tuning is done for one epoch. The layers are unfrozen
useful for a variety of NLP tasks. It used to be the one at a time, and adjustments are made. This is
most advanced NLP technique, however BERT repeated till convergence.
[which was just dethroned by XLNet in text c) Bidirectional Modeling
categorization deposed it [11], [12]. There are a bunch
of regularization and improvement techniques for The performance is further enhanced by
word-based language displaying undertakings which assembling a forward and a backward LM-classifier
are exceptionally powerful. [13].
● Drop Connect: Rather than choosing
arbitrary Secret units to set its worth to
A. STAGES
nothing. The contributions from stowed away
units (L-1th layer) get by a secret unit (Lth
layer)) are haphazardly chosen
● Non-monotonically Set off Normal SGD:
Assuming the approval metric neglects to
improve for n cycles (n=5) Averaging of the
loads for n emphases are gotten back to
refresh the boundaries.
● Variable Length Backpropagation
Groupings: Rather than utilizing fixed
components to perform backpropagation of
fixed BPTT window. Seq_len is determined in
with the end goal that the beginning stage
doesn't necessarily fall on a particular word.
Fig.3 Stages of ULMFIT Algorithm
● Inserting Dropout: This is identical to
performing dropout on the installing grid at a Picking the base model:
word level, where the dropout is
The ideal source task was viewed as language
communicated across all the word vector's
demonstrating and was thought of as the practically
implanting.
equivalent to ImageNet for NLP undertakings. This is
● Decrease in Implanting Size: decrease in a direct result of the accompanying explanation: "It
complete boundaries for a language model is catches numerous features of language significant for
diminishing the word vector size. downstream errands, like long haul conditions,
various leveled relations, and opinion. Rather than
● Enactment Regularization: AR punishes undertakings like machine interpreting and
initiations that are fundamentally bigger than entailment, it gives information in close limitless
0 for of regularizing the organization. amounts for most spaces and languages. Additionally,
language demonstrating can be prepared to adjust to
a) Pooling Concats the specific one of a kind element of the objective
errand and language displaying is a part of different
Generally speaking, while classifying text, the key other NLP errands.
words represent only a small portion of the whole
document, especially if it is a lengthy one. In order to For the most part, a decent language model (LM)
prevent information loss, the max-pooled and mean- like the AWD-LSTM, is picked as the base model. It
pooled forms of the hidden state vector are is by and large expected that the more the base model,
concatenated. the more will be the presentation of the last model on
different NLP errands after calibrating. [14]
General-space LM pre-training

979-8-3503-1398-7/23/$31.00 ©2023 IEEE 144

Authorized licensed use limited to: KLE Technological University. Downloaded on May 04,2024 at 15:50:04 UTC from IEEE Xplore. Restrictions apply.
Proceedings of the International Conference on Sustainable Communication Networks and Application (ICSCNA 2023)
IEEE Xplore Part Number: CFP23DW8-ART; ISBN: 979-8-3503-1398-7

The preparing is to be finished on an enormous

corpus of for language which successfully gets the
principal properties and parts of language. This would
be something like the Picture Net corpus, in any case, Loss in Loss in % of Duration
for language. This stage must be performed just a Duration
Trainin Validatio Validatio of Time
Stage of
single time. The subsequent pre-trained model can be g n n
Training
reused for the following stages for all assignments. Attained

The preparing is done so show as of now grasps 0.60179

1 0.617884 0.679907 05:25 03:24
the overall properties of language and must be 6
changed a little to suit the particular errand. As a
matter of fact, it was found that preparing was 0.5805 0.58677 0.71729
2 05:51 02:59
particularly valuable for little datasets and medium- 24 3 0
sized datasets.
0.5288 0.58576 0.73481
3 06:13 04:14
73 5 3
Target task LM tweaking
0.4206 0.58503 0.72780
This stage is finished to make the model fit the 4 05:45 03:50
64 0 4
model to the particular objective undertaking. When a
pre-trained model is utilized, then, at that point, the Table II. Collation of Training and Validation of
combination at this stage is quicker. In this stage, ULMFIT
discriminative tweaking and skewed three-sided
learning rates are utilized for calibrating the language
model. [15] Comparative Analysis of Bert Vs Ulmfit
B. Result of ULMFIT In the below Fig.5 the BERT and ULMFIT
algorithm can be compared and analyzed for the best
In the below Fig.4 the results of ULMFIT is accuracy rate.
shown with respect to training Vs Validation Loss
Accuracy Rate
100
T raining Vs Validation Loss 90
0.7 80
70
0.6
Accuracy (%)

60
0.5 50
Loss (%)

0.4 40
30
0.3
20
0.2 10
0.1
0
1 2 3 4 BERT ULMFIT
Stage
Algorithms
Accuracy Not Accuracy
Training Validation

Fig.5 Analysis of Accuracy Rate

Fig.4 Collation of Training and Validation in terms of
IV. CONCLUSION
Loss using ULMFIT
This research study concludes by checking the
From the below TABLE II. Collation of training grammatical error using NLP. The algorithms BERT
and validation of ULMFIT is shown with loss in and ULMFIT were compared to give a good accuracy
validation and its percentage of validation with time. rate. Among these two algorithms, the BERT
algorithm concludes by giving a good accuracy rate
for grammar error correction. The percentage of the
BERT algorithm concludes by 86% accuracy rate. In

979-8-3503-1398-7/23/$31.00 ©2023 IEEE 145

Authorized licensed use limited to: KLE Technological University. Downloaded on May 04,2024 at 15:50:04 UTC from IEEE Xplore. Restrictions apply.
Proceedings of the International Conference on Sustainable Communication Networks and Application (ICSCNA 2023)
IEEE Xplore Part Number: CFP23DW8-ART; ISBN: 979-8-3503-1398-7

Future the work may be extended to reduce the time consumption and cost expenditure.

V.REFERENCES

1. https://deeplearninganalytics.org/nlp-building-a- grammatical China, November. Association for Computational Linguistics,

error-correction-model/ 2019.
2. Aditya Jain, Gandhar Kulkarni, Vraj Shah, “Natural Language 9. Chi Sun, Luyao Huang, and Xipeng Qiu, “Utilizing BERT for
Processing”, International Journal of Computer Sciences and aspect-based sentiment analysis via constructing auxiliary
Engineering Vol.6, Issue 1, January 2018. sentence”. In Proceedings of the Conference of the North
American Chapter of the Association for Computational
3. https://www.ibm.com/cloud/learn/natural-language Linguistics: Human Language T echnologies, Volume 1 (Long
processing#:~:text=Natural%20language%20processing%20(NL and Short Papers), pages 380–385, Minneapolis, Minnesota,
P)%20refers,same%20way%20hum an%20beings%20can. June. Association for Computational Linguistics, 2019.
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina 10. https://www.analyticsvidhya.com/blog/2018/11/tutorial-text-
T outanova “BERT : Pre-training of Deep Bidirectional classification-ulmfit-fastai-library/
T ransformers for Language Understanding”. In Proceedings of
Conference of the North American Chapter of the Association 11. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu,
for Computational Linguistics: Human Language Technologies, Yu Wang, Jianfeng Gao, Ming Zhou, and HsiaoWuen Hon.
Volume 1 (Long and Short Papers), pages 4171–4186, “Unified Language Model Pre-training for Natural Language
Minneapolis, Minnesota, June. Association for Computational Understanding and Generation” 2019.
Linguistics, 2019.
12. Christian Hadiwinoto, Hwee T ou Ng, and Wee Chung Gan
5. Yaru Hao, Li Dong, Furu Wei, and Ke Xu “Visualizing and “Improved word sense disambiguation using pre-trained
Understanding the Effectiveness of BERT ”. In Proceedings of contextualized word representations”. In Proceedings of the
the 2019 Conference on Empirical Methods in Natural Conference on Empirical Methods in Natural Language
Language Processing and the 9th International Joint Conference Processing and the 9th International Joint Conference on Natural
on Natural Language Processing (EMNLP -IJCNLP), pages Language Processing (EMNLP -IJCNLP), pages 5297–5306,
4141–4150, Hong Kong, China, November. Association for Hong Kong, China, November. Association for Computational
Computational Linguistics,2019. Linguistics, 2019.
6. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang, 13. Chuong B. Do and Andrew Y. Ng, “Transfer learning for text
“ GlossBERT : BERT for word sense disambiguation with gloss classification.” In Y. Weiss, B. Scholkopf, and ¨ J. C. Platt,
knowledge”. In Proceedings of the Conference on Empirical editors, Advances in Neural Information Processing Systems 18,
Methods in Natural Language Processing and the 9th pages 299–306. MIT Press, 2006.
International Joint Conference on Natural Language Processing
(EMNLP-IJCNLP), pages 3509–3514, Hong Kong, China, 14. Jeremy Howard, Sebastian Ruder, “Universal Language Model
November. Association for Computational Linguistics, 2019. Fine-tuning for Text Classification”, Proceedings of the 56th
Annual Meeting of the Association for Computational
7. Koroteev M.V., Financial University under the government of Linguistics (Long Papers), pages 328–339 Melbourne, Australia,
the Russian Federation, Moscow, Russia, “BERT: A Review of July 15 - 20, 2018.
Applications in Natural Language Processing and
Understanding”, March 2021. 15. A. D. K, D. V. S, R. G, M. P. D, P. Balasubrarnanie and S.
Hamsanandhini, "Machine Learning based Iris Recognition
8. Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam, Modern Voting System," 2023 International Conference on
“ Exploiting BERT for end-to-end aspect-based sentiment Innovative Data Communication Technologies and Application
analysis” In Proceedings of the 5th Workshop on Noisy User- (ICIDCA), Uttarakhand, India, 2023, pp. 1-5, doi:
generated T ext (W-NUT 2019), pages 34–41, Hong Kong, 10.1109/ICIDCA56705.2023.10099580.
16. Rajesh Kanna, P., Pandiaraja, P, "An efficient sentiment analysis
approach for product review using T urney algorithm." J.
Procedia Comput. Sci. 165, 356–362. 2019

979-8-3503-1398-7/23/$31.00 ©2023 IEEE 146

Authorized licensed use limited to: KLE Technological University. Downloaded on May 04,2024 at 15:50:04 UTC from IEEE Xplore. Restrictions apply.