Comparative Analysis (Published)
Comparative Analysis (Published)
Ayesha Shahid
PhD Scholar (ELT), Cyprus International University
[email protected]
Dr Khizar Hayat
PhD Education, Institute of Education & Research, University of the Punjab
[email protected]
Zahra Iqbal
MS Scholar (ELL), Cyprus International University
[email protected]
Iqra Jabeen
Assistant Professor, Department of English, University of Sargodha
[email protected]
Abstract
The current study aims at presenting a comparative analysis of the use of Artificial Intelligence
(AI) tool, i.e. ChatGPT against traditional teaching methods in a second language learning
classroom i.e. teaching of English as a Second Language (ESL). The adopting and adapting of
technology in language learning has embarked upon a great shift from traditional mode of
teaching / learning to the integration of AI tools, like ChatGPT offering new avenues for
language learning. The current study explores the efficiency, advantages, and limitations of
utilizing ChatGPT together with or in place of conventional and traditional teaching approaches
in ESL classrooms. By examining and comparing the two methodologies and their effectiveness
in an ESL classroom, the study seeks to bring into light the potential of AI-based tools in
completely transforming second language education. The study applies mixed method approach
for data collection and analysis. The data was collected using questionnaires and open-ended
interviews from students and teachers of Department of English and Computer Science,
University of Sargodha. For statistical analysis, SPSS was used. The findings of this study
contribute to the ongoing discourse on optimizing language teaching methods and inform
educators, policymakers, and stakeholders about the role and future prospects of AI in ESL
pedagogy.
Keywords: ChatGPT, Traditional Teaching Methods, Artificial Intelligence (AI), Pedagogy,
Language Education, Technology Integration
Introduction
In the backdrop of fast changing scenario of education, the integration of technology into
language teaching and learning has stimulated innovative advancements, redefining conventional
approaches to pedagogy (Ray, 2023). In this changing scenario, ChatGPT is at the vanguard,
where artificial intelligence (AI) has emerged as a potent instrument. An AI-powered
conversational model called ChatGPT has drawn a lot of interest since it has the potential to
completely change how people learn and teach languages (Caie, 2023; Belda-Medina & Calvo-
Ferrer, 2022; Ray, 2023). Nowadays, the progress we're making in artificial intelligence is seen
as a big change, but it brings up both worry and hope. We're hopeful because being smarter helps
us solve problems. But we're worried because the most popular type of AI, called machine
learning, might harm our understanding of science and morals. It could do this by including a
wrong idea of language and knowledge in our technology. (Chomsky, 2023)
This study explores how AI and language learning overlap, concentrating on teaching English as
a Second Language (ESL). The goal is to perform a thorough comparison of ChatGPT and
conventional teaching techniques in order to determine the benefits, drawbacks, and
effectiveness of incorporating this cutting-edge technology into ESL classrooms.
The cornerstone of language education has traditionally been the traditional teaching techniques,
which rely on tried-and-true pedagogical concepts, textbooks, and direct human connection. But
the emergence of AI, especially programmes like ChatGPT, offers a chance to improve and
expand the set of instructional resources (Kim, Chai & Kim, 2019). Deep learning algorithms are
used by ChatGPT to have discussions that are natural and appropriate given the circumstances.
Exploration into its feasibility as a supplemental or alternative tool in language training has been
sparked by its potential to offer real-time feedback, encourage discourse, and emulate language
usage (Huang, Hew & Fryer, 2022)
This study embarks on a comparative journey, juxtaposing the utilization of ChatGPT with
traditional teaching methods. The focus areas encompass a spectrum of critical language
acquisition domains, including writing skills, conversational proficiency, vocabulary acquisition,
and student engagement. Through a meticulous examination of these aspects, we seek to
decipher the potential strengths and weaknesses of each approach, providing insights into their
applicability within the realm of ESL pedagogy.
Moreover, ethical considerations are paramount in evaluating the integration of ChatGPT. As AI
becomes increasingly intertwined with education, addressing ethical implications concerning
privacy, bias, and responsible utilization is imperative. This research strives to foster a nuanced
understanding of the ethical landscape, ensuring that the implementation of AI aligns with the
fundamental values of equitable and inclusive education.
By navigating the realms of AI technology and traditional teaching paradigms, this study aims to
contribute substantively to the discourse surrounding innovative language education. As the
digital era unfolds, understanding the symbiosis of AI and pedagogy is not only timely but
essential for educators, policymakers, and stakeholders in optimizing language learning
strategies and nurturing the linguistic competencies of future generations.
Research Objectives
Following are the specific objectives of the research:
to evaluate and compare the impact of ChatGPT and traditional teaching methods on ESL
participants' language skills
to investigate and compare the levels of student engagement
to explore the extent to which ChatGPT and traditional teaching methods can be
customized and adapted to meet the diverse learning needs
Research Questions
The research aims at finding the answers to the following specific questions:
How does the use of ChatGPT influence ESL participants' language skills compared
to traditional teaching methods?
How does the use of ChatGPT affect student engagement and active participation in
ESL classrooms as compared to traditional teaching methods?
To what extent can ChatGPT be customized to cater to the diverse learning needs and
proficiency levels of ESL participants?
Literature Review
The landscape of language education has witnessed a significant shift in recent years due to
advancements in artificial intelligence (AI) technologies. In particular, the emergence of
ChatGPT, a cutting-edge AI-powered conversational model, has sparked growing interest in its
potential to augment traditional teaching methods in the domain of English as a Second
Language (ESL) instruction.
The deep learning architecture used by ChatGPT, created by OpenAI, enables it to have
conversations that mimic those of a human and produce text based on the input it receives.
Studies examining the use of ChatGPT in language learning have emphasised its capacity to
offer real-time feedback on writing tasks, increasing vocabulary, grammar, and syntax learning.
Additionally, ChatGPT provides a conversational method of learning, imitating interactive
discussion and assisting in the improvement of conversational abilities (McMinn, 2023, Sharma,
2023b).
In a recent study by Sharma (2023a), in an ESL classroom, ChatGPT was implemented to
enhance language learning. The results showed that ChatGPT has the potential to promote
interactive learning experiences since participants showed increased levels of engagement during
writing and conversational activities.
Understanding ChatGPT's effectiveness and potential benefits requires comparing it to
conventional teaching techniques. Traditional ESL instruction frequently makes use of
textbooks, class discussions, and one-on-one contacts between teachers and students. However,
incorporating AI-driven tools like ChatGPT introduces a dynamic and interactive dimension to
language learning.
Research by Tlili et al (2023) conducted a comparative analysis of ChatGPT and traditional
methods for writing improvement in an ESL context. The study found that ChatGPT
significantly enhanced the writing skills of participants, providing instant feedback and
personalized guidance, surpassing the capabilities of traditional grammar correction tools.
As AI technologies become integral to language education, ethical considerations regarding their
use gain prominence. Scholars emphasize the importance of addressing privacy concerns, biases,
and responsible AI utilization (Zhou, et al, 2023, Ray, 2023). Privacy and data security issues
related to the interaction between participants and AI models, such as ChatGPT, are subjects of
ongoing investigation to ensure responsible and secure integration within educational
environments. Just like other AI systems, ChatGPT might show bias in its responses. These
biases happen due to various reasons, like the specific algorithms used for machine learning and
the data utilized for training and improving the system. Even though people help label data based
on ChatGPT's instructions during training, it's important to note that these human labelers may
not represent a wide range of opinions and viewpoints. This lack of representation can introduce
biases into the data (Zhou et al, 2023, Saini, 2023).
There have been multiple cases where ChatGPT has demonstrated racial and gender biases. This
AI chatbot operates on learning models that haven't been specifically trained to remove biases. It
essentially follows its algorithm and provides information based on the data it was given, without
the ability to recognize or correct biases. The data used to train ChatGPT is both outdated and
limited. ChatGPT was trained on a dataset of approximately 570 GB, which is roughly 300
billion words. However, this dataset isn't sufficient to address queries on a wide range of topics
from various perspectives. Moreover, this data hasn't been updated since 2021, failing to keep up
with progressive changes. (Saini, 2023)
Customization and adaptability of educational tools are vital to address the diverse learning
needs of ESL students. ChatGPT offers customization features that can be tailored to specific
learning goals, learner proficiency levels, and preferences. Additionally, ChatGPT's adaptability
to various ESL teaching contexts and its potential to personalize learning experiences have been
highlighted as valuable attributes (Zhou et al, 2023; Rudolph, 2023)
With a focus on its potential advantages and the requirement for a comparison against
conventional teaching techniques, this evaluation emphasises how the landscape of language
education is changing as a result of the integration of ChatGPT. For educators and academics in
the field of ESL instruction in a multicultural environment, the capacity to customise and adapt
ChatGPT to meet various learning demands continues to be a crucial area of investigation,
particularly in this fast changing modern world (Rettberg, 2022).
Methodology
Research Design
The research design for this study is a mixed-methods approach that incorporates both
quantitative and qualitative methods. Descriptive quantitative analysis helps in quantifying the
data and making it look more authentic. Qualitative analysis on the other hand brings in the data
collected through observation and interaction. Through a comparative lens, the integration of
ChatGPT and conventional teaching techniques in English as a Second Language (ESL)
instruction is assessed to determine their individual effects on language learning outcomes,
student engagement, ethical issues, and adaptability. Because of the thorough exploration of the
research objectives made possible by the research design, the phenomenon being studied is better
understood.
Participants
The participants in this study include ESL students, ESL teachers, and AI experts with
knowledge of natural language processing. The students and teachers were selected from the
Department of English, University of Sargodha, while the AI experts were chosen from the
Department of Computer Science of the same University. To ensure that student profiles,
language skill levels, and educational backgrounds are diverse, a purposeful sample technique is
used. The data was collected through questionnaires and interviews. Prior to selecting the
sample, pre-interviews were conducted to ascertain the educational and linguistic background of
the participants. An estimated 200 ESL students and 20 ESL teachers made up the sample. Also
included were 10 AI experts who can shed light on ChatGPT's technical details and possibilities.
Data Collection
Questionnaires
Questionnaires were given to students and teachers in order to get quantitative information about
their opinions and preferences on the use of ChatGPT in conjunction with more conventional
teaching techniques. Writing skills, conversational competence, vocabulary development, student
engagement, ethical issues, and customization/adaptability are covered in the questions, which
are organised to correspond with the research objectives.
Interviews
A smaller group of participants, which includes AI experts and ESL teachers, were interviewed
in-depth using a semi-structured format. In-depth qualitative insights about ChatGPT's
effectiveness, ethical issues, customizability options, and the use of technology in language
learning are sought after by these interviews.
Observations
The effectiveness of ChatGPT and conventional teaching techniques are evaluated through
classroom observations of student engagement and participation. The observations emphasise the
interaction between pupils, their total involvement, and behavioural elements.
Data Analysis
Quantitative Analysis
The quantitative data obtained from questionnaires (Likert Scale) were analysed using SPSS. T-
test was performed for significant analysis. This analysis provides a comparative assessment of
the impact of ChatGPT and traditional teaching methods on various language learning aspects.
Qualitative Analysis
The qualitative data from interviews and open-ended survey questions are analyzed using
thematic analysis. Themes related to the research objectives, including ethical considerations,
customization, learner engagement, and teacher perceptions, are identified and interpreted to
derive meaningful insights.
Results
The table below shows the minimum and maximum responses to the questions asked. It tells that
respondents are strongly agreed while maximum value shows that the participants are agreed.
The mean value of ‘ChatGPT developing writing skills’ is 1.68 while the standard deviation
(0.470) shows that the responses of the participants are not much different from the mean value.
The mean value of ‘ChatGPT developing speaking skills’ is 2.00 while the standard deviation
(0.857) shows that the responses of the participants are not much different from the mean value.
But the mean value of ‘ChatGPT building vocabulary’ is 4.34 while the standard deviation
(1.118) shows that the responses of the participants are not much different from the mean value.
Descriptive Statistics
The quantitative analysis revealed that ESL participants utilizing ChatGPT demonstrated a
statistically significant improvement in writing skills (p < 0.05) compared to those using
traditional teaching methods. The conversational proficiency of participants using ChatGPT also
showed notable enhancement (p < 0.01). However, vocabulary acquisition did exhibit a
statistically significant difference between the two groups but in favour of traditional method.
Almost 60% participants were of the view that ChatGPT did help them in understanding the
writing mechanics and enhancing their writing skills. They were cautious in their responses as
the use of ChatGPT is not welcomed openly by the academicians and its use is somehow
discouraged. Regarding conversational proficiency, the participants were still in favour of the use
of ChatGPT, though in a less percentage as compared to writing skills. Majority of the
participants (60%) were of the opinion that ChatGPT is a helpful tool in improving spoken
English. Quite an opposite view was collected when it came to vocabulary building. Most of the
participants (60%) were of the view that ChatGPT did not help them in improving their
vocabulary. They opined that they would opt for traditional method for building their
vocabulary. The figure below is a graphic representation of the results:
Observations indicated a higher level of student engagement and participation during ChatGPT-
assisted conversational exercises compared to traditional teaching methods. Participants were
more active in interactive dialogues facilitated by ChatGPT, demonstrating increased enthusiasm
and involvement. These observations were strengthened by the statistical data collected from the
participants and the teachers. Almost 70% participants were of the view that students get
involved in showing great concern in learning activities when they are using AI tools.
The participants expressed concerns about potential privacy risks associated with using
ChatGPT, especially regarding data storage and security. Ethical considerations, such as the
responsible use of AI tools and the need for transparency in data handling, were deemed critical
by both educators and AI specialists. It was concern of almost everybody, even the participants
themselves, that while using ChatGPT, the participants do not take into consideration the ethics
related to academic crimes, like plagiarism.
Comparison of ChatGPT vs Traditional Method
in an ESL Classroom
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
ChatGPT Traditional Method
Ethical Concerns
The qualitative analysis revealed that ChatGPT provided a higher degree of customization to
cater to diverse learning needs and preferences. Educators appreciated the adaptability of
ChatGPT to different proficiency levels and acknowledged its potential to offer tailored learning
experiences. Even the participants showed quite an inclination towards adopting AI tools to fulfil
their academic needs.
Discussion
The integration of ChatGPT, an AI-powered conversational model, into English as a Second
Language (ESL) instruction has been examined through a comparative lens with traditional
teaching methods. This discussion synthesizes the findings and addresses the research objectives,
providing insights into the potential of AI in language education.
The results suggest that ChatGPT offers significant improvements in writing skills and
conversational proficiency for ESL participants. The instant feedback and personalized guidance
provided by ChatGPT during writing exercises appear to be instrumental in enhancing
participants' writing abilities. Additionally, the interactive and conversational nature of ChatGPT
positively influences conversational proficiency, enabling more dynamic language practice.
However, the results regarding the use of ChatGPT in vocabulary acquisition raises questions
about the effectiveness of ChatGPT in this domain. Further research should explore strategies to
optimize ChatGPT's role in vocabulary expansion.
Observations indicated a higher level of engagement and participation among ESL participants
when utilizing ChatGPT. The interactive and dynamic nature of ChatGPT seems to captivate
participants' interest, encouraging active involvement in language learning activities. This
finding underscores the potential of AI-driven tools in creating an interactive and engaging
learning environment.
Privacy emerged as a noteworthy concern among participants, especially in the context of data
storage and security when using ChatGPT. Educators and AI specialists highlighted the necessity
for clear guidelines and protocols to ensure responsible AI use, addressing ethical considerations
regarding data privacy, biases, and transparency. Sam Altman, the CEO OpenAI (2022) tweeted:
“ChatGPT is incredibly limited, but good enough at some things to create a misleading
impression of greatness. It’s a mistake to be relying on it for anything important right now. It’s a
preview of progress; we have lots of work to do on robustness and truthfulness”.
ChatGPT demonstrated a high degree of customization, aligning with diverse learning needs and
preferences. Educators appreciated the adaptability of ChatGPT to different proficiency levels
and teaching contexts. The potential for personalized learning experiences offered by ChatGPT
presents a promising avenue for tailoring language instruction to individual learner requirements.
Conclusion
The comparative analysis of ChatGPT and traditional teaching methods in ESL instruction sheds
light on the transformative potential of AI in language education. ChatGPT showcased
significant enhancements in writing skills and conversational proficiency, underscoring its value
as a complementary tool in language learning. However, attention must be given to privacy
concerns and ethical considerations surrounding the use of AI in educational settings.
While this study provides valuable insights, future research should delve deeper into refining AI
models like ChatGPT to optimize vocabulary acquisition and address ethical implications
rigorously. The seamless integration of AI into ESL instruction, with careful consideration of
privacy and responsible AI use, holds promise for fostering an enriched and interactive language
learning experience.
The study highlights how important it is to make people more willing to use ChatGPT by
focusing on its ability to improve performance, gaining trust in the technology, and creating
conditions that make it easy and attractive to use. When the researchers looked at how different
things were related, they found strong positive links between wanting to try it, actually using it,
intending to use it, how users feel about it, how easy it is to use, and trusting AI applications.
These findings give useful ideas for companies and governments that want to make the most of
ChatGPT, especially for Bangladeshi knowledge workers. However, it's important to know that
this study had limits, especially because it focused on a specific group of people. Still, the
research helps us understand what affects how people start using ChatGPT and keep using it,
suggesting ways to make better decisions and be more productive using AI tools. In the future,
we should do a longer study to see how people's use of ChatGPT changes over time. Also, we
should look deeply into how the culture of the users affects how they start using and keep using
ChatGPT. This would help us understand how the culture shapes how people use new
technology.
As machine learning and AI progress, language models such as GPT are expected to enhance
significantly in power and sophistication. These advancements hold promise for enhancing
precision and productivity in language-related activities, potentially enabling novel applications.
Nevertheless, it's crucial to contemplate the ethical and societal impacts of deploying these
technologies and guarantee their responsible development and utilization.
In a nutshell, ChatGPT, when thoughtfully implemented and accompanied by ethical guidelines,
has the potential to revolutionize language education, enhancing learning outcomes and engaging
students in novel and interactive ways. Its integration signifies a pivotal step towards a
technologically enriched future in ESL pedagogy.
References
Belda-Medina, J., & Calvo-Ferrer, J. R. (2022). Using chatbots as AI conversational partners in
language learning. Applied Sciences, 12(17), 8427.
Cai, W. (2023, March 10). ChatGPT can be powerful tool for language learning. University
Affairs. https://www.universityaffairs.ca/career-advice/career-advice-article/chatgptcan-
be-powerful-tool-for-language-learning/
Chomsky, N. (2023, March 11). The false promise of ChatGPT. The New York Times.
https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html
Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning—are they really
useful? A systematic review of chatbot‐supported language learning. Journal of
Computer Assisted Learning, 38(1), 237–257. https://doi.org/10.1111/jcal.12610
Kim, N. Y., Cha, Y., & Kim, H. S. (2019). Future English learning: Chatbots and artificial
intelligence. Multimedia-Assisted Language Learning, 22(3), 32–53.
https://www.kci.go.kr/kciportal/ci/sereArticleSearch/ciSereArtiView.kci?sereArticleSe
archBean.artiId=ART002505056
McMinn, S. (2023) ChatGPT killed the classroom star: AI’s rise means it’s time to rethink
teaching and testing. South China Morning Post Online. Available at
https://www.scmp.com/comment/ opinion/article/3206436/chatgpt-killed-classroom-star-
ais-rise-means-its-time-rethink-teachingand-testing
Ray, P. P. (2023). ChatGPT: A comprehensive review on background, applications, key
challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-
Physical Systems, 3, (121-154). https://doi.org/10.1016/j.iotcps.2023.04.003.
Rettberg, J. W. (2022). ChatGPT is multilingual but monocultural, and it’s learning your values.
Blog entry. Available at https://jilltxt.net/right-now-chatgpt-is-multilingual-but-
monocultural-butits-learning-your-values
Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional
assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 1– 22.
https://doi.org/10.37074/jalt.2023.6.1.9
Saini, G. (2023). Ethical implications of ChatGPT: The Good, the Bad, the Ugly.
https://unstop.com/blog/ethical-implications-of-chatgpt
Sam Altman [@Sama]. (2022, Dec 11). ChatGPT is incredibly limited, but good enough at some
things to create a misleading impression of
greatness…. .https://twitter.com/sama/status/1601731295792414720?lang=en
Sharma, Y. (2023a, March 10). What ChatGPT means for linguistic diversity and language
learning. University World News. https://www.universityworldnews.com/post.php?Story
=20230223132734294
Shemesh, H. (2023b, March 10). The ultimate guide for using ChatGPT English learning. The
Accent's Way Magazine. https://hadarshemesh.com/magazine/chatgpt-for-
learningenglish/
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B.
(2023). What if the devil is my guardian angel: ChatGPT as a case study of using
chatbots in education? Smart Learning Environments, 10(1), 1–24.
https://doi.org/10.1186/s40561-023-00237-x
Zhou, J., Ke, P., Qiu, X., Huang, M., & Zhang, J. (2023). ChatGPT: Potential, prospects, and
limitations. Frontiers of Information Technology & Electronic Engineering. Advance
online publication. https://doi.org/10.1631/FITEE.2300089
Zhou, J., Müller, H., Holzinger, A., & Chen, F. (2023). Ethical ChatGPT: Concerns, challenges,
and commandments. Doi: arxiv-2305.10646