On the Use of ChatGPT to Support

Requirements Engineering Teaching

and Learning Process

Juan Pablo Carvallo(B) and Lenin Erazo-Garzón

Universidad del Azuay, Cuenca, Ecuador

{jpcarvallo,lerazo}@uazuay.edu.ec

Abstract. The use of artificial intelligence (AI) in education is an emerging trend

that offers multiple opportunities and challenges. In this article, we present an
exploratory study on the use of ChatGPT, an AI-based tool that allows generating
responses to questions posed by the user, as support for project-based learning
in the Requirements Engineering course. The objective of the study is to identify
the advantages and disadvantages of using ChatGPT in this context, as well as to
provide recommendations for its implementation and improvement. To this end,
a survey was conducted with 27 students who used ChatGPT in the course, and
their responses were analyzed using statistical and qualitative techniques. The
study results show that students perceived ChatGPT as a useful and novel tool
that facilitated the generation and understanding of concepts, models and tools
related to requirements engineering, as well as the simulation of interviews with
value chain actors. However, some limitations and aspects to improve using Chat-
GPT were also identified, such as the accuracy and consistency of the generated
responses, the adequate contextualization of information, and the ability to cus-
tomize and adjust the tool. The study concludes that ChatGPT has great potential
to support project-based learning in the Requirements Engineering course, but also
requires technical and pedagogical improvements to optimize its performance and
effectiveness.

Keywords: Artificial intelligence · ChatGPT · Project-based learning ·

Requirements engineering

1 Introduction

Requirements engineering is a discipline that deals with identifying, analyzing, specify-

ing, and validating the requirements or needs that a system or software must meet [1]. It
is essential for the success of any software development project as it allows for defining
the scope and objectives, as well as establishing quality and customer satisfaction cri-
teria [2]. However, learning requirements engineering is not an easy task as it involves
mastering complex, abstract theoretical concepts, as well as developing practical skills
to apply them in real contexts [3].

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023
S. Berrezueta (Ed.): LACLO 2023, LNET, pp. 328–342, 2023.
https://doi.org/10.1007/978-981-99-7353-8_25
 On the Use of ChatGPT 329

One methodology used to facilitate learning in requirements engineering is project-

based learning (PBL), which involves designing and developing a product or service that
addresses a real need or problem [4]. PBL has multiple benefits for students, such as
developing cognitive, social, and professional skills, fostering creativity and innovation,
and increasing motivation and commitment, among others [5–7]. However, PBL also
presents some challenges, such as difficulty accessing reliable and up-to-date information
sources, lack of guidance and feedback during the process, and scarcity of resources and
time to carry out activities, among others [8].
In this context, an opportunity arises to use an artificial intelligence (AI) tool that
can support and improve PBL in the Requirements Engineering course (REC). The
field of AI has experienced significant development and advancement in recent years
thanks to increased computational capacity and the availability of large amounts of
data [9]. AI has multiple applications and benefits in various fields, including education
[10]. In this sense, various AI-based tools have been developed to support and improve
teaching-learning processes for both teachers and students [11].
In this article, we present an exploratory study on the use of ChatGPT, an AI-based
tool that allows generating responses to questions posed by users, to support PBL in a
REC. ChatGPT is an application based on GPT-3 (Generative Pre-Trained Transformer
3), a natural language model based on deep neural networks, which has demonstrated
a remarkable ability to generate coherent and diverse text on a wide variety of topics
and domains [12]. This study aims to identify the advantages and disadvantages of
using ChatGPT in this context and provide recommendations for its implementation and
improvement. To this end, we conducted a survey with 25 students who used ChatGPT
in the course and analyzed their responses using statistical and qualitative techniques.
The paper is structured with six sections, in addition to the introduction: Sect. 2
presents the related work, Sect. 3 introduces the case study and methodology, Sect. 4
shows the results of the study, Sect. 5 discusses the results, Sect. 6 discusses the threats
to validity, and finally, Sect. 7 presents some conclusions and future work.

2 Related Work
The application of Artificial Intelligence in Education (AIEd) has garnered significant
research attention in recent years, leading to a growing body of literature on the topic.
These studies have explored the use of AIEd tools in various educational contexts,
investigating their advantages, challenges, and limitations.
In the field of natural language processing (NLP), Jurafsky and Martin (2009) intro-
duced NLP and its practical applications, including its relevance to education [13].
McTear et al. (2016) discussed the design and implementation of AI-based conversa-
tional interfaces, such as chatbots and virtual assistants, in different fields, including
education [14]. Johnson and Valente (2008) developed Tactical Language and Culture
Training Systems (TLCTS), an AI-based system designed to teach foreign languages
and cultures to military students [15]. These studies demonstrate the broad applications
of AI and NLP in educational contexts.
Furthermore, Crompton and Burke (2023) conducted a study examining the applica-
tion of AI in Higher Education (HE) from 2016 to 2022 [16]. The research revealed a rapid
330 J. P. Carvallo and L. Erazo-Garzón

increase in AIEd studies published in HE, spanning across six continents. The most com-
mon department affiliation was Education (28%), followed by Computer Science (20%).
The study identified five primary usage codes for AIEd in HE: Assessment/Evaluation,
Predicting, AI Assistant, Intelligent Tutoring System (ITS), and Managing Student.
The study also emphasized the need for empirical exploration of new tools like
ChatGPT, which became publicly available in late 2022. The authors suggested that
ChatGPT could align with several use codes found in the study, providing support to
students in writing papers, while also helping instructors assess students’ work and
assisting with writing tasks [16].
Hulick (2023) highlighted the potential impact of ChatGPT and similar AI technolo-
gies on education. While these tools can benefit students by offering assistance with
assignments, concerns regarding cheating and the potential for misinformation have
been raised [17]. To address these challenges, the UCLA Center for the Advancement
of Teaching (2023) developed a comprehensive guidance document for instructors on
effectively incorporating generative AI, including ChatGPT, into their teaching practices,
with an emphasis on ethical considerations and pedagogical strategies [18].
Although several studies [19–21] have explored the use of IA, including ChatGPT,
to support foreign language learning and highlighted the significant impact of the use of
these technologies on learning outcomes, research has also examined the specific appli-
cation of AI-based tools in engineering education. Goldberg (2012) provided insights
into the design of competent genetic algorithms, an AI technique applicable to solving
complex problems across various domains, including engineering [22]. Qadir (2022)
reflected on the promises and challenges of integrating ChatGPT and other generative
AI tools in engineering education, highlighting the opportunities for enhancing learn-
ing, creativity, and collaboration while acknowledging the risks associated with misuse
and improper application [23]. Nikolic et al. (2023) conducted a multidisciplinary and
multi-institutional study to assess the impact of ChatGPT on the integrity of engineering
education assessment [24]. They investigated how ChatGPT could both enhance and
compromise the assessment process, depending on the context and task design.
These studies lay a strong foundation for exploring the use of ChatGPT in a require-
ment engineering course. They demonstrate the potential of AIEd tools like ChatGPT in
enhancing educational practices and supporting students and instructors in various ways.
However, they also highlight the need for further research to fully comprehend and lever-
age the capabilities of these tools in specific educational contexts, such as requirements
engineering.
While the aforementioned works provide valuable insights, it is important to
acknowledge their limitations. Although studies have examined the benefits of Chat-
GPT and AI-based tools in education, further research is needed to fully understand
their pedagogical impact and address potential ethical concerns. These studies often
focus on specific domains, such as language education or engineering, and may not
encompass the full range of educational contexts in which ChatGPT can be applied.
Additionally, it is crucial to evaluate the long-term effects of using AI tools in education
and develop robust frameworks for assessing their effectiveness and impact on student
learning outcomes.
 On the Use of ChatGPT 331

Building on the existing body of research, our study aims to contribute to the field of
engineering education in several ways. Firstly, our research evaluates the use of ChatGPT
as a support tool in a university-level engineering requirements course in Ecuador. By
examining students’ perceptions, experiences, and learning outcomes when using Chat-
GPT to simulate a fictitious company and its components, we aim to provide insights
into the practical application of AI in the specific context of engineering requirements.
Moreover, our study aims to identify the advantages, disadvantages, limitations, and
areas for improvement in using ChatGPT in this educational setting. By gaining a deeper
understanding of the potential benefits and challenges, we can develop strategies to
enhance the integration of AI tools in engineering education while mitigating associated
risks. This includes considering how to foster critical thinking and academic integrity
among students while harnessing the creative and collaborative potential of these tools.
Overall, our research expands upon the existing knowledge base by focusing on the
application of ChatGPT in engineering education and examining its specific impact in
the context of a requirements course. By addressing the limitations of previous studies
and providing new insights into the use of AI tools, we aim to contribute to the broader
discussions on the effective integration of AI in higher education, with implications for
pedagogy, student engagement, and the future of engineering education.

3 Case Study
3.1 Context
Universidad del Azuay, located in Cuenca, Ecuador, was established in 1970 and has
since become a recognized institution of higher education. The university is known
for its commitment to providing quality education and fostering academic excellence
in various disciplines. The Computer Science program at Universidad del Azuay is
designed to equip students with a comprehensive understanding of computer science
principles and practical application of technology.
Within the Computer Science curriculum, the REC holds significant importance.
This course focuses on the essential process of gathering, specifying, and validating
software requirements. Students learn the necessary skills to effectively communicate
with stakeholders and translate their needs into actionable requirements. By emphasizing
this critical aspect of software development, the university prepares students to excel in
the field.
Traditionally, the course required students to contact real companies and conduct
interviews to obtain firsthand information for requirements elicitation. This approach
aimed to bridge the gap between theoretical concepts and real-world applications, allow-
ing students to experience the challenges and complexities of working with stakeholders
in industry settings.
However, with recent curriculum changes, the requirement’s engineering course was
moved to the second semester, posing unique challenges. Second-semester students typ-
ically have limited technical knowledge compared to those in higher semesters, making
it even more difficult to engage with external companies and their employees. Acquiring
agreements with companies and securing their employees’ time and cooperation became
increasingly challenging, impacting the quality of the students’ learning experience.
332 J. P. Carvallo and L. Erazo-Garzón

To overcome these challenges and maintain the practical and experiential nature
of the REC, an innovative approach was introduced: The integration of ChatGPT as a
support tool aimed to enhance the teaching-learning process and provide students with
an experience close to industrial reality.
The introduction of ChatGPT as an aid in the teaching-learning process of require-
ments engineering is expected to yield several benefits. Firstly, it addresses the challenges
of engaging with real companies and their employees, especially for students in the sec-
ond semester with limited technical knowledge. Secondly, to provide a simulated but
realistic experience that allows students to develop their understanding of requirements
engineering concepts and enhance their practical application skills. Finally, by using
ChatGPT as a support tool, students are expected to create comprehensive models that
accurately depict the organization’s context, functional decomposition, and descriptions
of business and system use cases.

3.2 The Requirements Engineering Course in Universidad del Azuay

The REC in Universidad del Azuay, is based in the farmwork proposed by Klaus Pohl
[25], which fundaments the requirements engineering process in a well-established Sys-
tem Context. System context defines the scope and limits of the system, whose require-
ments are refined with three core activities, Elicitation, Negotiation and Documentation.
Also, three kinds of artefacts are used to model requirements: Goals, Scenarios and
Solution-oriented requirement models. Management and Validation are cross-sectional
activities to be performed along the entire requirements lifecycle.
To materialize the framework and guide the process in a systematic way, the
DHARMA method [26] is applied. This method aims to the definition of enterprise
architectures using the i* notation. The theoretical bases to support the method in the
analysis of enterprise context, structure and strategy, are two concepts defined by Porter
[27]: 1) the model of the market forces, used to reason about potential available strate-
gies and how to make them profitable, by analyzing existing dependencies with external
actors within five market forces, and 2) Value chain, which includes primary and sup-
port activities helpful to identify internal actors and dependencies in the scope of the
organization. The DHARMA method consists in four activities, as shown in Fig. 1:
• Activity 1. Modelling the enterprise context. The organization and its strategy are
carefully analyzed to identify its role inside the context, allowing the definition of
Context Actors (CA) and Organizational Areas (OA). At the end of this activity, i*
SD models are built and used to support reasoning and represent results.
• Activity 2. Modelling the environment of the system. In this activity, a system-
to-be is placed into the organization and its impact over the elements in the CM is
analyzed. The strategic dependencies of OAs and CAs are inspected to determine
which of them may be totally or partially satisfied by the system. The result of this
activity is also an i* SD model representing the dependencies that the system can
satisfy in relation to the different CAs or OAs.
• Activity 3. Decomposition of system goals. Dependencies included in the CM are
analyzed and decomposed into a hierarchy of intentional elements required to satisfy
them. These elements depict the services that the system must provide (functional
 On the Use of ChatGPT 333

requirements) as well as restrictions on them (non-functional requirements). An i*

SR diagram for the system is built.
• Activity 4. Identification of system architecture. This activity includes the identi-
fication of System Actors (SA), which play a role in the system and represent atomic
software domains. Intentional elements identified in Activity 3 are analyzed and
semantically grouped. Each aggrupation reveals the services that are expected to be
covered by SA.
During the course, students are required to perform several activities with the support
of the AI, namely, to describe a fictional company, define is organizational structure and
Porter´s models of five forces and value chain, identify CA and OA base on them and to
interview the AI in the role of representatives of OA, to identify interactions (intentional
elements) with CAs and also with other OAs. At the end, students are required to compose
Use Cases to describe the behavior of intentional elements in the i* SR model (seen as
scenarios). Finally, UML Use Case models are built to show the relation among them.
In order to evaluate the use of ChatGPT in the course, a survey was designed. The
survey encompassed seven questions aimed to assess students’ perceptions, advantages,
disadvantages, lessons learned, problems, limitations, and suggestions regarding the use
of ChatGPT in the course. We obtained 27 responses from students in the course, which
was taught during the first semester of 2023. The questions were answered using different
formats, such as multiple choice, Likert scale, and open-ended.

Fig. 1. The DHARMA Method

334 J. P. Carvallo and L. Erazo-Garzón

4 Analysis of the Results

The statistical analysis of the survey results is presented in this section. Invitations to
answer the questionnaire were sent to all students of the two parallels of the course. A
total of 27 students participated and sent back their answers. The demographic data of
the participants are shown in Fig. 2.

Fig. 2. Demographics of participants: ages (X-axis) and genders.

Question 1: What was your general perception about using ChatGPT as a support tool
in the Requirements Engineering course?
This question was answered using a five-level Likert scale, where 1 means very
negative, 2 means negative, 3 means neutral, 4 means positive, and 5 means very positive.
The frequency and percentage of each option are shown in Fig. 3. The results indicate
that most students had a positive or very positive perception of using ChatGPT in the
course (88.8%), with no negative responses.
Question 2: What do you consider were the main advantages of using ChatGPT in the
course?
This question was answered using a multiple-choice format, where the students could
select more than one option from a list of six possible advantages. The frequency and
percentage of each option are shown in Table 1. The results indicate that almost all of the
students (96.3%) agreed that “ChatGPT facilitated the generation of the simulation of
a fictitious company”, which was the main advantage of using the tool in the course. In
addition, “Allowing to perform simulated interviews with the AI”, was another advantage
pointed out by almost three-fourths of the students (70.4%). Other advantages that were
selected by half or more of the students were: “Helping create the organizational chart”
and “the model of Porter’s five forces”.
 On the Use of ChatGPT 335

Fig. 3. Distribution of responses regarding the perception of using ChatGPT in the course.

Table 1. Frequency and percentage of each option in question 2.

Option Frequency Percentage

It facilitated the generation of the simulation of a fictitious company 26 96.3%
It helped in the creation of the organizational chart of the fictitious 15 55.6%
company
It facilitated the generation of the value chain of the fictitious 12 44.4%
company
It helped in the creation of the model of Porter’s five forces for the 12 44.4%
fictitious company
It allowed to perform simulated interviews with the AI in the role of 19 70.4%
each actor in the value chain

Question 3: Did you identify any disadvantages or limitations in using ChatGPT as a

support tool?
This question was also answered using a multiple-choice format. The frequency and
percentage of each option are shown in Table 2. The results indicate that the most common
disadvantages or limitations identified by the students were “Limitations in understand-
ing and answering specific questions precisely”, with 17 participants (63.0%), followed
by “Difficulty in adequately contextualizing the information generated by ChatGPT”,
with 14 participants (51.9%). Four students did not identify any disadvantages or lim-
itations. In turn, five participants wrote additional responses, such as “Sometimes it
forgot what we were talking about”, “Sometimes it forgot about the type of company
and changed the actors”, “Too vague answers”, or “It cannot generate graphs”.
Question 4: What lessons or relevant knowledge did you obtain in the use of ChatGPT?
This question was answered using an open-ended format, where the students could
write their own responses. The lessons learned can be grouped into three main themes:
Firstly, the importance of pre-training ChatGPT and providing it with contextual
knowledge is highlighted to obtain accurate and contextualized responses. Furthermore,
the need to properly apply ChatGPT is emphasized, including asking specific questions,
336 J. P. Carvallo and L. Erazo-Garzón

Table 2. Frequency and percentage of each option in question 3.

Option Frequency Percentage

Limitations in understanding and answering specific questions 17 63.0%
precisely
Difficulty in adequately contextualizing the information generated by 14 51.9%
ChatGPT
Not identify any disadvantages or limitations 4 14.8%
Other: 5 18.5%
• Sometimes the chat would forget that we were talking about a
company and generate other responses and had to remind it of the
main topic
• Limitation of knowledge: ChatGPT3 only has knowledge up to
2021
• Initially, it was not possible to share the same chat with two people
(we had to share the same email to access the fictitious company’s
chat)
• Sometimes it forgets about the type of company and changes the
actors
• Responses are too ambiguous
• Problems with generating coherent and logical responses
• It cannot generate graphs

repeating previous responses, and verifying the quality of the obtained answers. Lastly,
the benefits of using ChatGPT in different contexts, such as interview simulations, creat-
ing fictional companies, and problem-solving support, are acknowledged. By following
these lessons, one can fully harness the potential of ChatGPT in the REC.
Question 5: According to your experience, what are the main benefits of using ChatGPT
in a requirement engineering course?
The responses were given as multiple choices, with six possible options and one for
others. The results showed that the most selected option was “Better ability to simulate
a fictitious company and its components”, with 22 participants (81.5%), followed by
“A greater understanding of the concepts related to requirements engineering”, with
19 participants (70.4%). The least selected option was “Simulation of documents and
contents”, with 5 participants (18.5%). No participant wrote an additional response. The
frequency and percentage of each option are shown in Table 3.
Question 6: Did you face any problems or difficulties when using ChatGPT in the
course?
The responses were given as multiple choices, with four possible options and one
option for others. The results showed that the most selected option was “Limitations in
generating desired results”, with 16 participants (59.3%), followed by “Difficulties in
obtaining clear and precise answers”, with 15 participants (55.6%). The least selected
option was “No problem or difficulty faced”, with 3 participants (11.1%). Two partici-
pants wrote additional responses: “Sometimes it did not generate answers and I had to
 On the Use of ChatGPT 337

refresh the page” and “It does not generate graphs”. The frequency and percentage of
each option are shown in Table 4.
Question 7: Do you have any suggestions or recommendations for improving the use
of ChatGPT in the course?
Responses were given as multiple choices, with four possible options and one option
for others. Results show that the most selected option was “Expand the knowledge
base of ChatGPT to address a greater variety of topics”, with 19 participants (70.4%),
followed by “Provide a greater capacity for customization and adjustment of ChatGPT”,
with 13 participants (48.1%). The least selected option was “No limitation or aspect
to improve identified”, with 1 participant (3.7%). No participant wrote an additional
response. Frequency and percentages of options are shown in Table 5.

Table 3. Frequency and percentage of each option in question 5.

Option Frequency Percentage

A greater understanding of the concepts related to requirements 19 70.4%
engineering
Better ability to simulate a fictitious company and its components 22 81.5%
Acquisition of skills in the practical application of models and 9 33.3%
tools used
Ease to simulate interviews with managers of areas 11 40.7%
Detailed answers at an appropriate level 6 22.2%
Simulation of documents and contents 5 18.5%

Table 4. Frequency and percentage of each option in question 6.

Option Frequency Percentage

Difficulties in obtaining clear and precise answers 15 55.6%
Problems maintaining a coherent and logical conversation with 7 25.9%
ChatGPT
Limitations in generating desired results 16 59.3%
No problem or difficulty faced 3 11.1%
Other: 2 7.41%
• Sometimes it did not generate answers and I had to refresh the
page
• It does not generate graphs
338 J. P. Carvallo and L. Erazo-Garzón

5 Discussion
The study conducted at Universidad del Azuay in Ecuador evaluated the use of ChatGPT
as a support tool in a university-level engineering requirements course. The research
aimed to provide insights into the practical application of AI in the specific context
of engineering requirements. The study also sought to identify the advantages, disad-
vantages, limitations, and areas for improvement in using ChatGPT in this educational
setting.

Table 5. Frequency and percentage of each option in question 7.

Option Frequency Percentage

Improve the accuracy and consistency of the responses generated 10 37.0%
Expand the knowledge base of ChatGPT to address a greater variety 19 70.4%
of topics
Provide a greater capacity for customization and adjustment of 13 48.1%
ChatGPT
No limitation or aspect to improve identified 1 3.7%

The results of the study were largely positive. Most students had a positive or very
positive perception of using ChatGPT in the course, with no negative responses. The
main advantage identified by the students was that ChatGPT facilitated the generation
of the simulation of a fictitious company. This was seen as a significant benefit, as it
allowed students to gain practical experience without the challenges of engaging with
real companies.
However, the study also identified some limitations and areas for improvement. Some
students noted that ChatGPT sometimes had difficulty providing precise and contextu-
alized responses to specific questions. Additionally, there were instances where the AI
would forget the context of the conversation and generate unrelated responses. The main
themes that emerged from the analysis of the responses are:
• The need for more training and guidance on using ChatGPT effectively, including
best practices, tips, and examples.
• The importance of providing more context and background information to ChatGPT
before asking questions or generating information related to the fictitious company
and the requirements engineering concepts.
• The suggestion to use ChatGPT in combination with other tools or resources, such as
textbooks, articles, videos, etc., to complement and verify the information generated
by ChatGPT.
• The recommendation to update ChatGPT’s knowledge base regularly to include more
recent information and developments in the field of requirements engineering.
• The possibility of integrating ChatGPT with other tools or platforms used in the
course, such as learning management systems, collaboration tools, etc., to facilitate
its use and improve its effectiveness.
 On the Use of ChatGPT 339

In light of these findings, the following recommendations can be made to improve

future courses:
1. Enhance AI Contextual Understanding: Future iterations of the course could include
more structured guidelines on how to interact with the AI. For instance, students
could be taught to provide more context in their questions or to use specific keywords
that the AI is more likely to understand. For example, instead of asking “What is
this?”, students could be encouraged to ask “What is this concept in the context of
software engineering?”. This would help the AI provide more accurate and relevant
responses. On the AI development side, we expect that improvements will be made to
the AI’s ability to understand and maintain context throughout a conversation. This
could involve refining the AI’s algorithms or training the AI on more diverse and
context-rich datasets.
2. Improve Result Generation: This could be addressed by providing more specific
instructions or parameters to the AI. For example, if students are looking for a specific
type of information, they could be taught to request this more effectively from the AI.
For instance, if a student wants to know about usability requirements for a mobile app
for a specific type of users, they could ask “Can you provide the usability requirements
for a mobile app to be used by special or different abilities people?”, instead of a
more general question like “Tell me about the usability requirements for a mobile
app”, This would help the AI generate more relevant and useful responses.
3. Address Contextual Forgetting: This could be mitigated by incorporating reminders
or cues into the conversation to help the AI maintain context. For example, students
could be taught to refer back to the main topic of the conversation regularly or to
reiterate key points. On the AI development side, improvements could be made to the
AI’s ability to remember and refer back to previous parts of the conversation. This
could involve enhancing the AI’s memory capabilities or refining its algorithms to
better handle the context.
4. Enhance Graphical Capabilities: If the graphical representation of data is an impor-
tant part of the course, alternative tools or methods could be introduced to supplement
the AI’s capabilities in this area. For example, students could be taught to use data
visualization tools like Tableau or Power BI to create graphs and charts. Alterna-
tively, the AI could be integrated with a tool that can generate graphs based on the
data provided. However, when it comes to the use of requirements modeling frame-
works, some attention has to be taken, since seamless integration with goal or object
based graphical modeling tools may introduce ethical risks due to students’ lack of
dedication to more manual modeling tasks.
5. Provide More Guidance: This could be addressed by providing more guidance on
how to ask specific and detailed questions to get more precise responses. For example,
students could be taught to break down complex questions into simpler, more specific
questions. This would make it easier for the AI to understand and answer each part
of the question.
6. Develop guidelines for specific activities: Based on the lessons learned in this study,
it is recommended to propose comprehensive guidelines to teach students to collect,
specify and validate software requirements with the support of AI-based tools (e.g.,
ChatGPT), simulating scenarios very close to reality. In this specific course, guidelines
340 J. P. Carvallo and L. Erazo-Garzón

may include procedures to: i) elaborate the mission, vision, policies, and strategic
objectives of a fictitious company; i) identify the main stakeholders in the market
(Porter’s competitive forces) and strategic partners to deliver its value proposition; ii)
develop its functional organic structure; iii) design structured interviews with chatbot
representatives from various roles within the company; and, iv) elicit, document and
validate business and system requirements with the support of high-level models
(context, goals, use cases).
After this experience, we are convinced that integrating AI tools like ChatGPT
in engineering education has shown promising results. However, it is crucial to con-
tinue exploring strategies to enhance their integration while mitigating associated risks.
This includes fostering critical thinking and academic integrity among students while
harnessing the creative and collaborative potential of these tools.

6 Threats to Validity

When interpreting the results of the study, it is essential to consider several threats to
validity that may impact the reliability and generalizability of the findings. One imme-
diate threat to validity is related to the students’ prior knowledge and experience with
ChatGPT. Since students did not receive formal training on using this tool, they had to
learn alongside the tasks required to complete the course. While more formal training
might have yielded more relevant results and improved task efficiency, this initial expe-
rience also aimed at exploring students’ ability to investigate and learn about new tool
paradigms.
However, despite the lack of formal training, the results of the course exceeded
expectations in terms of the level of detail and quality of deliverables produced by
students, surpassing previous editions of the course where students interacted with real
companies. This improvement may be attributed to the limited time and information
provided by real companies and their employees compared to the support and detailed
information available through ChatGPT.
It is important to acknowledge that the survey conducted in the study only included
27 students who used ChatGPT in a specific REC. Consequently, the generalizability of
the study’s results to other courses and educational contexts may be limited. However,
it is important to note that the students who took part in the survey belonged to two
distinct groups, both simultaneously attending the same subject during the semester but
with different schedules. Both groups included 18 students; 17 from the first group and
10 from the second answered the survey. While this may enhance the reliability of the
results to some extent, we acknowledge that it is required to conduct further studies with
additional cohorts to validate and extend the promising findings observed in this study.

7 Concluding Remarks

By incorporating ChatGPT as a tool, students were able to overcome the challenges of

engaging with real companies and their employees. This approach provided a simulated
but realistic experience, allowing students to develop their understanding of requirements
 On the Use of ChatGPT 341

engineering concepts and enhance their practical application skills. The use of ChatGPT
facilitated the creation of a fictional company, analysis based on industry models, and
interviews with simulated employees, enabling students to construct comprehensive
models that accurately depicted the organization’s context, functional decomposition,
and descriptions of business and system use cases. This case study demonstrates the suc-
cessful integration of ChatGPT as a valuable support tool in the requirements engineering
education process at Universidad del Azuay.
However, the study also highlights some limitations of using AI in this context,
including difficulties in understanding and answering specific questions, generating
desired results, maintaining context, and producing graphical outputs. These limitations
present opportunities for both pedagogical and technological improvements.
The recommendations provided aim to address these limitations. However, further
studies are required in order to make them more comprehensive for students.
The integration of AI tools in higher education, particularly in engineering courses,
is a promising avenue that can significantly enhance the learning experience. However, it
is crucial to continuously evaluate and refine these integrations to ensure they effectively
meet the learning objectives and cater to the needs of the students. The insights from
this study contribute to the broader discussions on the effective integration of AI in
higher education, with implications for pedagogy, student engagement, and the future
of engineering education.
As future work, it is proposed to develop a guide to accompany the project-based
learning process in requirements engineering with the support of AI tools. This guide
will be used in future RECs, together with the incorporation of other AI tools.

References
1. Sommerville, I., Sawyer P.: Requirements Engineering: A Good Practice Guide. John Wiley &
Sons (1997)
2. Kotonya G., Sommerville I.: Requirements Engineering: Processes and Techniques. John
Wiley & Sons (1998)
3. Nuseibeh, B., Easterbrook, S.: Requirements engineering: a roadmap. In: Proceedings of the
Conference on the Future of Software Engineering, pp. 35–46 (2000)
4. Blumenfeld, P.C., Soloway, E., Marx, R.W., Krajcik, J.S., Guzdial, M., Palincsar, A.: Moti-
vating project-based learning: sustaining the doing, supporting the learning. Educ. Psychol.
26(3–4), 369–398 (1991)
5. Thomas, J.W.: A Review of Research on Project-Based Learning. Autodesk Foundation, San
Rafael (2000)
6. Krajcik, J.S., Blumenfeld, P.C., Marx, R.W., Bass, K.M., Fredricks, J., Soloway, E.: Inquiry
in project-based science classrooms: initial attempts by middle school students. J. Learn. Sci.
7(3–4), 313–350 (1998)
7. Hmelo-Silver, C.E.: Problem-based learning: what and how do students learn? Educ. Psychol.
Rev. 16(3), 235–266 (2004)
8. Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach, 4th edn. Pearson (2020)
9. Luckin, R., Holmes, W., Griffiths, M., Forcier, L.B.: Intelligence Unleashed: An Argument
for AI in Education. Pearson, London (2016)
10. Nkambou, R., Bourdeau, J., Mizoguchi, R. (eds.): Advances in Intelligent Tutoring Systems.
Springer Berlin Heidelberg, Berlin, Heidelberg (2010)
342 J. P. Carvallo and L. Erazo-Garzón

11. Tom, B., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33,
1877–1901 (2020)
12. Roumeliotis, K.I., Tselikas, N.D.: ChatGPT and open-AI models: a preliminary review. Future
Internet 15, 192 (2023). https://doi.org/10.3390/fi15060192
13. Jurafsky, D., Martin, J.H.: Speech and Language Processing: An Introduction to Natural
Language Processing, Computational Linguistics, and Speech Recognition. Pearson/Prentice
Hall Upper Saddle River (2009)
14. McTear, M., Callejas, Z., Griol, D.: The Conversational Interface. Springer International
Publishing, Cham (2016). https://doi.org/10.1007/978-3-319-32967-3
15. Johnson, W.L., Valente, A.: Tactical language and culture training systems: Using artificial
intelligence to teach foreign languages and cultures. In: Proceedings of the 20th national
conference on Innovative applications of artificial intelligence (IAAI’08), vol. 2, pp. 1632–
1639 (2008)
16. Crompton, H., Burke, D.: Artificial intelligence in higher education: the state of the field. Int.
J. Educ. Technol. High Educ. 20(1), 22 (2023). https://doi.org/10.1186/s41239-023-00392-8
17. Hulick, K.: How ChatGPT and similar AI will disrupt education. Science News. https://
www.sciencenews.org/article/chatgpt-ai-artificial-intelligence-education-cheating-accuracy
(2023)
18. UCLA Center for the Advancement of Teaching: Guidance for the Use of Generative AI.
https://teaching.ucla.edu/resources/ai_guidance/ (2023)
19. Han, J., Yoo, H., Kim, Y., Oh, A.: RECIPE: How to Integrate ChatGPT into EFL Writing
Education. https://arxiv.org/abs/2105.09938 (2023)
20. Yeh, S.-Y., et al.: A study of EFL students’ participation patterns and learning performance
in an English chatbot system. Comput. Educ. 151, 103862 (2020)
21. Bibauw, S., François, T., Desmet, P.: Dialogue-based CALL: a multilevel meta-analysis. Lang.
Learn. Technol. 24(1), 110–139 (2020)
22. Goldberg, D.E.: The design of innovation: Lessons from and for competent genetic algorithms.
Springer Science & Business Media (2012)
23. Qadir, J.: Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative
AI for Education. https://www.researchgate.net/publication/366712815_Engineering_Edu
cation_in_the_Era_of_ChatGPT_Promise_and_Pitfalls_of_Generative_AI_for_Education
(2022)
24. Nikolic, S., et al.: ChatGPT versus engineering education assessment: a multidisciplinary and
multi-institutional benchmarking and analysis of this generative artificial intelligence tool to
investigate assessment integrity. Eur. J. Eng. Educ. 48(4), 559–614 (2023). https://doi.org/10.
1080/03043797.2023.2213169
25. Pohl, K.: Requirements Engineering. Springer-Verlag, Berlin Heidelberg (2010)
26. Carvallo, J.P., Franch, X.: On the use of i* for architecting hybrid systems: a method and
an evaluation report. In: Persson, A., Stirna, J. (eds.) The Practice of Enterprise Modeling,
pp. 38–53. Springer Berlin Heidelberg, Berlin, Heidelberg (2009). https://doi.org/10.1007/
978-3-642-05352-8_5
27. Porter, M.: Competitive Strategy. Free Press. New York, NY, USA (1980)