See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/357044010

Educational chatbots for project-based learning: investigating learning

outcomes for a team-based design course

Article in RUSC Universities and Knowledge Society Journal · December 2021

DOI: 10.1186/s41239-021-00302-w

CITATIONS READS

60 1,800

1 author:

Jeya Amantha Kumar

Universiti Sains Malaysia
48 PUBLICATIONS 909 CITATIONS

SEE PROFILE

All content following this page was uploaded by Jeya Amantha Kumar on 15 December 2021.

The user has requested enhancement of the downloaded file.

Kumar Int J Educ Technol High Educ (2021) 18:65
https://doi.org/10.1186/s41239-021-00302-w

RESEARCH ARTICLE Open Access

Educational chatbots for project‑based

learning: investigating learning outcomes
for a team‑based design course
Jeya Amantha Kumar*

*Correspondence:
[email protected] Abstract
Centre for Instructional Educational chatbots (ECs) are chatbots designed for pedagogical purposes and are
Technology and Multimedia,
Universiti Sains Malaysia, viewed as an Internet of Things (IoT) interface that could revolutionize teaching and
Minden, Pulau Pinang, learning. These chatbots are strategized to provide personalized learning through the
Malaysia concept of a virtual assistant that replicates humanized conversation. Nevertheless, in
the education paradigm, ECs are still novel with challenges in facilitating, deploying,
designing, and integrating it as an effective pedagogical tool across multiple fields, and
one such area is project-based learning. Therefore, the present study investigates how
integrating ECs to facilitate team-based projects for a design course could influence
learning outcomes. Based on a mixed-method quasi-experimental approach, ECs were
found to improve learning performance and teamwork with a practical impact. Moreo-
ver, it was found that ECs facilitated collaboration among team members that indirectly
influenced their ability to perform as a team. Nevertheless, affective-motivational
learning outcomes such as perception of learning, need for cognition, motivation, and
creative self-efficacy were not influenced by ECs. Henceforth, this study aims to add to
the current body of knowledge on the design and development of EC by introducing a
new collective design strategy and its pedagogical and practical implications.
Keywords: Chatbot, Design education, Teamwork, Project-based learning,
Collaborative learning, Mobile learning, Telegram

Introduction
Chatbots are defined as computer programs that replicate human-like conversations by
using natural language structures (Garcia Brustenga et al., 2018; Pham et al., 2018) in
the form of text messages (websites or mobile applications), voice-based (Alexa or Siri),
or a combination of both (Pereira et al., 2019; Sandoval, 2018). These automated con-
versational agents (Riel, 2020) have been significantly used to replicate customer service
interaction (Holotescu, 2016) in various domains (Khan et al., 2019; Wang et al., 2021) to
an extent it has become a common trend (Wang et al., 2021). The use of chatbots are fur-
ther expanded due to the affordance, cost (Chocarro et al., 2021), development options
(Sreelakshmi et al., 2019; Wang et al., 2021), and adaption facilitated by social network

© The Author(s), 2021. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits
use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original
author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third
party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the mate-
rial. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or
exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://​
creat​iveco​mmons.​org/​licen​ses/​by/4.​0/.
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 2 of 28

and mobile instant messaging (MIM) applications (apps) (Brandtzaeg & Følstad, 2018;
Cunningham-Nelson et al., 2019) such as WhatsApp, Line, Facebook, and Telegrams.
Accordingly, chatbots popularized by social media and MIM applications have been
widely accepted (Rahman et al., 2018; Smutny & Schreiberova, 2020) and referred to as
mobile-based chatbots. These bots have been found to facilitates collaborative learn-
ing (Schmulian & Coetzee, 2019), multimodal communication (Haristiani et al., 2019),
scaffolding, real-time feedback (Gonda et al., 2019), personalized learning (Oke & Fer-
nandes, 2020; Verleger & Pembridge, 2019), scalability, interactivity (Dekker et al.,
2020) and fosters knowledge creation and dissipation effectively (Verleger & Pembridge,
2019). Nevertheless, given the possibilities of MIM in conceptualizing an ideal learning
environment, we often overlook if instructors are capable of engaging in high-demand
learning activities, especially around the clock (Kumar & Silva, 2020). Chatbots can
potentially be a solution to such a barrier (Schmulian & Coetzee, 2019), especially by
automatically supporting learning communication and interactions (Eeuwen, 2017; Gar-
cia Brustenga et al., 2018) for even a large number of students.
Nevertheless, Wang et al. (2021) claims while the application of chatbots in educa-
tion are novel, it is also impacted by scarcity. Smutny and Schreiberova (2020), Wang
et al. (2021), and Winkler and Söllner (2018) added that the current domain of research
in educational chatbots (EC) has been focusing on language learning (Vázquez-Cano
et al., 2021), economics, medical education, and programming courses. Henceforth,
it is undeniable that the role of EC, while not been widely explored outside these con-
texts (Schmulian & Coetzee, 2019; Smutny & Schreiberova, 2020) due to being in the
introductory stages (Chen et al., 2020), are also constrained with limited pedagogical
examples in the educational context (Stathakarou et al., 2020). Nevertheless, while this
absence is inevitable, it also provides a potential for exploring innovations in educational
technology across disciplines (Wang et al., 2021). Furthermore, according to Tegos et al.
(2020), investigation on integration and application of chatbots is still warranted in the
real-world educational settings. Therefore, the objective of this study is first to address
research gaps based on literature, application, and design and development strategies for
EC. Next, by situating the study based on these selected research gaps, the effectiveness
of EC is explored for team-based projects in a design course using a quasi-experimental
approach.

Literature review
Chatbots
The term “chatbot” was derived to represent two main attributes which are “chat” in lieu
of the conversational attributes and “bot” short for robot (Chocarro et al., 2021). Chat-
bots are automated programs designed to execute instructions based on specific inputs
(Colace et al., 2018) and provide feedback that replicates natural conversational style
(Ischen et al., 2020). According to Adamopoulou and Moussiades (2020), there are six
main chatbots parameters that determines design and development consideration:

i. knowledge domain—open and closed domains

ii. services—interpersonal, intrapersonal, and inter-agent chatbots
iii. goals—informative, chat-based, or task-based
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 3 of 28

iv. input processing and response generation—rule-based model, retrieval-based

model, and generative model
v. human aid
vi. build—open-source or closed platforms.

These parameters convey that a chatbot can fulfill numerous communication and
interaction functionalities based on needs, platforms, and technologies. Typically, they
are an exemplary use of artificial intelligence (AI) which conversely initiated various
state-of-the-art platforms for developing chatbots such as Google’s DialogFlow, IBM
Watson Conversation, Amazon Lex, Flow XO, and Chatterbot (Adamopoulou & Mous-
siades, 2020). However, while using AI is impressive, chatbots application is limited as
it primarily uses the concept of artificial narrow intelligence (ANI) (Holotescu, 2016).
Therefore, it can only perform a single task based on a programmed response, such as
examining inputs, providing information, and predicting subsequent moves. While lim-
ited, ANI is the only form of AI that humanity has achieved to date (Schmulian & Coet-
zee, 2019). Conversely, such limitation also enables a non-technical person to design and
develop chatbots without much knowledge of AI, machine learning, or neuro-linguistic
programming (Gonda et al., 2019). While this creates an “openness with IT” (Schlagwein
et al., 2017) across various disciplines, big-tech giants such as Google, Facebook, and
Microsoft also view chatbots as the next popular technology for the IoT era (Følstad &
Brandtzaeg, 2017). Henceforth, if chatbots are able to gain uptake, it will change how
people obtain information, communicate (Følstad et al., 2019), learn and gather informa-
tion (Wang et al., 2021); hence the introduction of chatbots for education.

Chatbots in education
Chatbots deployed through MIM applications are simplistic bots known as messenger
bots (Schmulian & Coetzee, 2019). These platforms, such as Facebook, WhatsApp, and
Telegram, have largely introduced chatbots to facilitate automatic around-the-clock
interaction and communication, primarily focusing on the service industries. Even
though MIM applications were not intended for pedagogical use, but due to affordance
and their undemanding role in facilitating communication, they have established them-
selves as a learning platform (Kumar et al., 2020; Pereira et al., 2019). Henceforth, as
teaching is an act of imparting knowledge through effective communication, the ubiqui-
tous format of a mobile-based chatbot could also potentially enhance the learning expe-
rience (Vázquez-Cano et al., (2021); thus, chatbots strategized for educational purposes
are described as educational chatbots.
Bii (2013) defined educational chatbots as chatbots conceived for explicit learning
objectives, whereas Riel (2020) defined it as a program that aids in achieving educational
and pedagogical goals but within the parameters of a traditional chatbot. Empirical stud-
ies have positioned ECs as a personalized teaching assistant or learning partner (Chen
et al., 2020; Garcia Brustenga et al., 2018) that provides scaffolding (Tutor Support)
through practice activities (Garcia Brustenga et al., 2018). They also support personal-
ized learning, multimodal content (Schmulian & Coetzee, 2019), and instant interac-
tion without time limits (Chocarro et al., 2021). All the same, numerous benefits have
been reported reflecting positive experiences (Ismail & Ade-Ibijola, 2019; Schmulian
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 4 of 28

& Coetzee, 2019) that improved learning confidence (Chen et al., 2020), motivation,
self-efficacy, learner control (Winkler & Söllner, 2018), engagement (Sreelakshmi et al.,
2019), knowledge retention (Cunningham-Nelson et al., 2019) and access of information
(Stathakarou et al., 2020). Furthermore, ECs were found to provide value and learning
choices (Yin et al., 2021), which in return is beneficial in customizing learning prefer-
ences (Tamayo et al., 2020).
Besides, as ECs promotes anytime anywhere learning strategies (Chen et al., 2020;
Ondas et al., 2019), it is individually scalable (Chocarro et al., 2021; Stathakarou et al.,
2020) to support learning management (Colace et al., 2018) and delivery of context-
sensitive information (Yin et al., 2021). Henceforth, encouraging participation (Tamayo
et al., (2020); Verleger & Pembridge, 2019) and disclosure (Brandtzaeg & Følstad, 2018;
Ischen et al., 2020; Wang et al., 2021) of personal aspects that were not possible in a tra-
ditional classroom or face to face interaction. Conversely, it may provide an opportunity
to promote mental health (Dekker et al., 2020) as it can be reflected as a ‘safe’ environ-
ment to make mistakes and learn (Winkler & Söllner, 2018). Furthermore, ECs can be
operated to answer FAQs automatically, manage online assessments (Colace et al., 2018;
Sandoval, 2018), and support peer-to-peer assessment (Pereira et al., 2019).
Moreover, according to Cunningham-Nelson et al. (2019), one of the key benefits of
EC is that it can support a large number of users simultaneously, which is undeniably
an added advantage as it reduces instructors’ workload. Colace et al. (2018) describe
ECs as instrumental when dealing with multiple students, especially testing behavior,
keeping track of progress, and assigning tasks. Furthermore, ECs were also found to
increase autonomous learning skills and tend to reduce the need for face-to-face inter-
action between instructors and students (Kumar & Silva, 2020; Yin et al., 2021). Con-
versely, this is an added advantage for online learning during the onset of the pandemic.
Likewise, ECs can also be used purely for administrative purposes, such as delivering
notices, reminders, notifications, and data management support (Chocarro et al., 2021).
Moreover, it can be a platform to provide standard information such as rubrics, learn-
ing resources, and contents (Cunningham-Nelson et al., 2019). According to Meyer von
Wolff et al (2020), chatbots are a suitable instructional tool for higher education and stu-
dent are acceptive towards its application.
Conversely, Garcia Brustenga et al. (2018) categorized ECs based on eight tasks in
the educational context as described in Table 1. Correspondingly, these tasks reflect
that ECs may be potentially beneficial in fulfilling the three learning domains by provid-
ing a platform for information retrieval, emotional and motivational support, and skills
development.
Albeit, from the instructor’s perspective, ECs could be intricate and demanding, espe-
cially when they do not know to code (Schmulian & Coetzee, 2019); automation of some
of these interactions could benefit educators in focusing on other pedagogical needs
(Gonda et al., 2019). Nevertheless, enhancing such skills is often time-consuming, and
teachers are usually not mentally prepared to take up a designer’s (Kim, 2021) or pro-
grammer’s role. The solution may be situated in developing code-free chatbots (Luo &
Gonda, 2019), especially via MIM (Smutny & Schreiberova, 2020).
By so, for EC development, it is imperative to ensure there are design principles
or models that can be adapted for pedagogical needs. At the same time, numerous
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 5 of 28

Table 1 Educational task of ECs

Task Description

Administrative and management EC is used to aid the onboarding of learning activities

FAQ Platform EC provides feedback on FAQs for administration or educational topics
Mentoring EC is used to monitor students learning outcomes cognitively and
affectively
Motivational EC provides emotional and motivational support
Practice specific skills and abilities EC is used as a practice buddy to learn a language, communication,
and programming
Simulations EC is used to simulate conditions that aid with rehabilitation, such as
in healthcare
Reflection and metacognitive strategies EC is used as a skillful classmate that aids learning
Student learning assessment ECs measure learning outcomes quickly and routinely

models have been applied in the educational context, such as CommonKADS (Cam-
eron et al., 2018), Goal-Oriented Requirements Engineering (GORE) (Arruda et al.,
2019), and retrieval-based and QANet models (Wu et al., 2020). Nevertheless, these
models reflect a coding approach that does not emphasize strategies or principles
focusing on achieving learning goals. While Garcia Brustenga et al. (2018), Gonda
et al. (2019), Kerly et al. (2007), Satow (2017), Smutny and Schreiberova (2020), and
Stathakarou et al. (2020) have highlighted some design guidelines for EC, imperatively
a concise model was required. Therefore, based on the suggestions of these empirical
studies, the researcher identified three main design attributes: reliability, pedagogy,
and experience (Table 2).
Nevertheless, it was observed that the communicative aspect was absent. Undeniably,
chatbots are communication tools that stimulate interpersonal communication (Ischen
et al., 2020; Wang et al., 2021); therefore, integrating interpersonal communication was
deemed essential. Interpersonal communication is defined as communication between
two individuals who have established a relationship (Devito, 2018), and such a relation-
ship is also significant through MIM to represent the communication between peers and
instructors (Chan et al., 2020). Furthermore, according to Han and Xu (2020), interper-
sonal communication moderates the relationship and perception that influences the use
of an online learning environment. According to Hobert and Berens (2020), while chat-
bot interaction could facilitate small talk that could influence learning, such capabilities
should not be overemphasize. Therefore, it was concluded that four fundamental attrib-
utes or strategies were deemed critical for EC design: Reliability, interpersonal commu-
nication, Pedagogy, and Experience (RiPE), which are explained in Table 3.
Nevertheless, ECs are not without flaws (Fryer et al., 2019). According to Kumar and
Silva (2020), acceptance, facilities, and skills are still are a significant challenge to stu-
dents and instructors. Similarly, designing and adapting chatbots into existing learn-
ing systems is often taxing (Luo & Gonda, 2019) as instructors sometimes have limited
competencies and strategic options in fulfilling EC pedagogical needs (Sandoval, 2018).
Moreover, the complexity of designing and capturing all scenarios of how a user might
engage with a chatbot also creates frustrations in interaction as expectations may not
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 6 of 28

Table 2 EC design strategies from empirical findings

Factors Contextualization Kerly et al. Gonda Smutny and Satow (2017) Stathakarou Garcia
of RiPE (2007) et al. Schreiberova et al. (2020) Brustenga
(2019) (2020) et al. (2018)

Reliability Easy access to a √ √ √

stable platform
Privacy √
Feedback for con- √
tinuous improve-
ment
Pedagogy Learning contents √ √ √ √ √
that are reliable,
precise, and specific
to the subject
Alignment with √ √ √ √
learning goals
Active learning that √ √
promotes reflection
and metacognition
Encourages com- √
munication and
collaboration in the
learning community
Personalize learning √ √
and feedbacks
Progress manage- √ √
ment
Experience Realistic humanized √ √ √
communication
with social cue
Affective interaction √ √ √ √
(greetings, humor,
anthropomorphism,
empathy)
Social media and √
MIM affordance

Table 3 Describing RiPE for educational chatbots

Factors Description

Reliability The chatbot should be easy to access through a stable and private platform where the learner
can depend on the chatbot to gain continuous feedback with confidence
Interpersonal The chatbot should establish a relationship between learner-learner and learner-instructor
communica- through activities that enable them to relate, share information, communicate, and/or collabo-
tion rate
Pedagogy The chatbot provides learning content and activities that align with the learning goals of the
course. Therefore, while facilitating a personalized learning platform, the chatbot should also
embody active learning and communication strategies that allow the instructor to monitor
learning progress
Experience The chatbot should be deployed on a preferred communication platform and reflect how
learners communicate in a natural online setting. Affective interaction such as greetings, humor,
emojis, and/or empathy should also be included to improve emotional engagement. Further-
more, the interaction should be based on small learning units strategized for micro-learning

always be met for both parties (Brandtzaeg & Følstad, 2018). Hence, while ECs as con-
versational agents may have been projected to substitute learning platforms in the future
(Følstad & Brandtzaeg, 2017), much is still to be explored from stakeholders’ viewpoint
in facilitating such intervention.
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 7 of 28

Research gaps in EC research

Three categories of research gaps were identified from empirical findings (i) learning
outcomes, (ii) design issues, and (iii) assessment and testing issues. Firstly, research gaps
concerning learning outcomes are such as measuring effectiveness (Schmulian & Coet-
zee, 2019), perception, social influence (Chaves & Gerosa, 2021), personality traits, affec-
tive outcomes (Ciechanowski et al., 2019; Winkler & Söllner, 2018), acceptance (Chen
et al., 2020; Chocarro et al., 2021), satisfaction (Stathakarou et al., 2020), interest (Fryer
et al., 2019), motivation, learning performance (Yin et al., 2021), mental health (Brandt-
zaeg & Følstad, 2018), engagement (Riel, 2020) and cognitive effort (Nguyen & Sidor-
ova, 2018). EC studies have primarily focused on language learning, programming, and
health courses, implying that EC application and the investigation of learning outcomes
have not been investigated in various educational domains and levels of education.
Next, as for design and implementation issues, a need to consider strategies that
align ECs application for teaching and learning (Haristiani et al., 2019; Sjöström et al.,
2018) mainly to supplement activities that can be used to replace face-to-face interac-
tions (Schmulian & Coetzee, 2019) has been implied. According to Schmulian and Coet-
zee (2019), there is still scarcity in mobile-based chatbot application in the educational
domain, and while ECs in MIM has been gaining momentum, it has not instigated stud-
ies to address its implementation. Furthermore, there are also limited studies in strate-
gies that can be used to improvise ECs role as an engaging pedagogical communication
agent (Chaves & Gerosa, 2021). Besides, it was stipulated that students’ expectations
and the current reality of simplistic bots may not be aligned as Miller (2016) claims that
ANI’s limitation has delimited chatbots towards a simplistic menu prompt interaction.
Lastly, in regards to assessment issues, measurement strategies for both intrinsic and
extrinsic learning outcomes (Sjöström et al., 2018) by applying experimental approaches
to evaluate user experience (Fryer et al., 2019; Ren et al., 2019) and psychophysiologi-
cal reactions (Ciechanowski et al., 2019) has been lacking. Nevertheless, Hobert (2019)
claims that the main issue with EC assessment is the narrow view used to evaluate out-
comes based on specific fields rather than a multidisciplinary approach. Moreover, eval-
uating the effectiveness of ECs is a complex process (Winkler & Söllner, 2018) as it is
unclear what are the characteristics that are important in designing a specific chatbot
(Chaves & Gerosa, 2021) and how the stakeholders will adapt to its application to sup-
port teaching and learning (Garcia Brustenga et al., 2018). Furthermore, there is a need
for understanding how users experience chatbots (Brandtzaeg & Følstad, 2018), espe-
cially when they are not familiar with such intervention (Smutny & Schreiberova, 2020).
Conversely, due to the novelty of ECs, the author has not found any studies pertaining to
ECs in design education, project-based learning, and focusing on teamwork outcomes.

Purpose of the study

This study aims to investigate the effects of ECs for an Instructional Design course that
applies team-based project towards learning outcomes, namely learning performance,
perception of learning, need for cognition, motivation, creative self-efficacy, and team-
work. Learning performance is defined as the students’ combined scores accumulated
from the project-based learning activities in this study. Next, perception of the learning
process is described as perceived benefits obtained from the course (Wei & Chou, 2020)
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 8 of 28

and the need for cognition as an individual’s tendency to participate and take pleasure
in cognitive activities (de Holanda Coelho et al., 2020). The need for cognition also indi-
cates positive acceptance towards problem-solving (Cacioppo et al., 1996), enjoyment
(Park et al., 2008), and it is critical for teamwork, as it fosters team performance and
information-processing motivation (Kearney et al., 2009). Henceforth, we speculated
that EC might influence the need for cognition as it aids in simplifying learning tasks
(Ciechanowski et al., 2019), especially for teamwork.
Subsequently, motivational beliefs are reflected by perceived self-efficacy and intrin-
sic values students have towards their cognitive engagement and academic performance
(Pintrich & de Groot, 1990). According to Pintrich et al. (1993), self-efficacy and intrin-
sic value strongly correlate with task value (Eccles & Wigfield, 2002), such as interest,
enjoyment, and usefulness. Furthermore, Walker and Greene (2009) explain that moti-
vational factors that facilitate learning are not always solely reliant on self-efficacy, and
Pintrich and de Groot (1990) claims that a combination of self-efficacy and intrinsic
value is better in explaining the extent to which students are willing to take on the learn-
ing task. Ensuing, the researcher also considered creative self-efficacy, defined as the stu-
dents’ belief in producing creative outcomes (Brockhus et al., 2014). Prior research has
not mentioned creativity as a learning outcome in EC studies. However, according to
Pan et al. (2020), there is a positive relationship between creativity and the need for cog-
nition as it also reflects individual innovation behavior. Likewise, it was deemed neces-
sary due to the nature of the project, which involves design. Lastly, teamwork perception
was defined as students’ perception of how well they performed as a team to achieve
their learning goals. According to Hadjielias et al. (2021), the cognitive state of teams
involved in digital innovations is usually affected by the task involved within the innova-
tion stages. Hence, the consideration of these variables is warranted.
Therefore, it was hypothesized that using ECs could improve learning outcomes, and
a quasi-experimental design comparing EC and traditional (CT) groups were facilitated,
as suggested by Wang et al. (2021), to answer the following research questions.

i. Does the EC group perform better than students who learn in a traditional class-
room setting?
ii. Do students who learn with EC have a better perception of learning, need for cog-
nition, motivational belief, and creative self-efficacy than students in a traditional
classroom setting?
iii. Does EC improve teamwork perception in comparison to students in a traditional
classroom setting?

Educational chatbot design, development, and deployment

According to Adamopoulou and Moussiades (2020), it is impossible to categorize chat-
bots due to their diversity; nevertheless, specific attributes can be predetermined to
guide design and development goals. For example, in this study, the rule-based approach
using the if-else technique (Khan et al., 2019) was applied to design the EC. The rule-
based chatbot only responds to the rules and keywords programmed (Sandoval, 2018),
and therefore designing EC needs anticipation on what the students may inquire about
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 9 of 28

Table 4 Description of ECs and objectives

EC Name of Bot Objective Respondents

1 Welcome Bot i.To gather student information such as full name, nickname, All
gender, student ID, email, and Telegram ID Number
ii.To update the user database
iii.To obtain permission to be contacted through Telegram
and to use their data based on the data protection act
2 Group Registration Bot To group students and update the user database All
3 Group leader registration Bot To register as a group leader or team member and update All
the user database
4 Project registration Bot To update project name and acronym Group leader
5 Assignment Bot To access and download learning contents All
6 Picaso Bot To upload contents through the bot All
7 Perception bot To gather feedback on the EC and acceptance All
8 Progress Bot To rate students’ self-perceived progress in their project, All
identify teamwork and project development issues
9 Report writing Bot FAQ to guide students on how to write the project report All
10 Peer to peer evaluation Bot Peer to peer feedback platform for presentation All

(Chete & Daudu, 2020). Furthermore, a designer should also consider chatbot’s capabili-
ties for natural language conversation and how it can aid instructors, especially in repeti-
tive and low cognitive level tasks such as answering FAQs (Garcia Brustenga et al., 2018).
As mentioned previously, the goal can be purely administrative (Chocarro et al., 2021) or
pedagogical (Sandoval, 2018).
Next, as for the design and development of the EC, Textit (https://​textit.​com/), an
interactive chatbots development platform, was utilized. Textit is a third-party software
developed by Nyaruka and UNICEF that offers chatbots building possibilities without
coding but using the concept of flows and deployment through various platforms such
as Facebook Messenger, Twitter, Telegram, and SMS. For the design of this EC, Telegram
was used due to data encryption security (de Oliveira et al., 2016), cloud storage, and the
privacy the student and instructor would have without using their personal social media
platforms. Telegram has been previously used in this context for retrieving learning con-
tents (Rahayu et al., 2018; Thirumalai et al., 2019), information and progress (Heryandi,
2020; Setiaji & Paputungan, 2018), learning assessment (Pereira, 2016), project-based
learning, teamwork (Conde et al., 2021) and peer to peer assessment (P2P) (Pereira et al.,
2019).
Subsequently, the chatbot named after the course code (QMT212) was designed as a
teaching assistant for an instructional design course. It was targeted to be used as a task-
oriented (Yin et al., 2021), content curating, and long-term EC (10 weeks) (Følstad et al.,
2019). Students worked in a group of five during the ten weeks, and the ECs’ interac-
tions were diversified to aid teamwork activities used to register group members, infor-
mation sharing, progress monitoring, and peer-to-peer feedback. According to Garcia
Brustenga et al. (2018), EC can be designed without educational intentionality where it
is used purely for administrative purposes to guide and support learning. Henceforth,
10 ECs (Table 4) were deployed throughout the semester, where EC1-EC4 was used for
administrative purposes as suggested by Chocarro et al. (2021), EC5-EC6 for assignment
(Sjöström et al., 2018), EC7 for user feedback (Kerly et al., 2007) and acceptance (Yin
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 10 of 28

et al., 2021), EC8 for monitoring teamwork progress (Colace et al., 2018), EC9 as a pro-
ject guide FAQ (Sandoval, 2018) and lastly EC10 for peer to peer assessment (Colace
et al., 2018; Pereira et al., 2019). The ECs were also developed based on micro-learn-
ing strategies to ensure that the students do not spend long hours with the EC, which
may cause cognitive fatigue (Yin et al., 2021). Furthermore, the goal of each EC was to
facilitate group work collaboration around a project-based activity where the students
are required to design and develop an e-learning tool, write a report, and present their
outcomes. Next, based on the new design principles synthesized by the researcher, RiPE
was contextualized as described in Table 5.
Example flow diagrams from Textit for the design and development of the chatbot are
represented in Fig. 1. The number of choices and possible outputs determine the com-
plexity of the chatbot where some chatbots may have simple interaction that requires
them to register their groups (Fig. 2) or much more complex interaction for peer-to-peer
assessment (Fig. 3). Example screenshots from Telegram are depicted in Fig. 4.

Methodology
Participants
The participants of this study were second-year Bachelor of Education (Teaching
English to Speakers of Other Languages (TESOL)) who are minoring in multime-
dia and currently enrolled in a higher learning institute in Malaysia. The 60 students
were grouped into two classes (30 students per class) as either traditional learning
class (control group-CT) or chatbot learning class (treatment group-EC). Out of the
60 participants, only 11 were male, 49 were female, and such distribution is typical
for this learning program. Both groups were exposed to the same learning contents,
class duration, and instructor, where the difference is only denoted towards different
class schedules, and only the treatment group was exposed to EC as an aid for teach-
ing and learning the course. Both groups provided written consent to participate in
the study and were given honorarium for participation. However, additional consent
was obtained from the EC group in regards of data protection act as the intervention
includes the use of social media application and this was obtained through EC1: Wel-
come Bot.

The course
The instructional design course aims to provide fundamental skills in designing
effective multimedia instructional materials and covers topics such as need analysis,
instructional analysis, learner analysis, context analysis, defining goals and objectives,
developing instructional strategy and materials, developing assessment methods, and
assessing them by conducting formative and summative assessments. The teaching
and learning in both classes are identical, wherein the students are required to design
and develop a multimedia-based instructional tool that is deemed their course pro-
ject. Students independently choose their group mates and work as a group to fulfill
their project tasks. Moreover, both classes were also managed through the institu-
tion’s learning management system to distribute notes, attendance, and submission of
assignments.
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 11 of 28

Table 5 Conceptualization of RiPE in the design of the EC

RiPE Contextualization in the EC

Reliability MIM using Telegram provides easy access and has added privacy features of the data shared.
EC1 was also designed to gather feedback on the privacy and confidentiality agreement using
students’ data and Telegram account details. Example:
"Next, we understand that using SMS is going to be expensive for you. Hence, we choose to use
Telegram as you will be able to use any WIFI system or data available. Telegram cloud services
and enhanced security system will also provide privacy and confidentiality"
"So, @results.nick_name, do you agree to use Telegram as a method for contacting you? All
your data will be private and confidential and be used only for this study. None of your personal
details will be made public in term of knowledge dissemination 1 = Yes, 2 = No"
Next, feedback for continuous improvement was established in EC7 on acceptance. Example
interaction:
“Well, that’s interesting…. what don’t you like about it?”
The EC also allows users to come back to any interaction by using keywords, example interac-
tion:
"Well, I guess you are busy so anytime you need my help just type "/report" and I will be here to
help you"
interpersonal The ECs were designed to mimic interpersonal communication. Example interactions are:
communica- “Ohh…that’s too bad. What do you think are the issues you are facing?”
tion “I would like to help you better. Do you see issues in teamwork?”
“Hi there @contact.first_name, I noticed you didn’t answer the questions yet. I (QMT212 Bot)
have been contacting you for a bit now. So, I would like to know how you feel about me and
your overall perception of using Telegrams Bots for learning.”
The communications were also personalized by using nicknames (@results.nick_name) and first
names (@contact.first_name)
Learning Learning contents were designed to be specific to the subject. Example interactions:
"Some example subtopic topics that should be added are:
i. Gagnes 9 Events of Instructions
ii. Instructional Strategies
iii. Formative Assessment
Would you like to have more information about these subtopics?"
Similarly, active learning promoting reflection, metacognition, and communication. Example:
“How do you feel about the prototype product your group has developed to date?”
The EC also enables students to receive notification, contents, and guidelines, examples are:
"Hi @contact.first_name, it is time to complete Assignment 1. This is a group work where you
are asked to describe your product’s initial phase (proposal). You will have to integrate what you
have studied to date into this proposal"
"That’s great. Next, these videos may be helpful in understanding how to complete the assign-
ment. The deadline for your submission will be on @ date.assignment1. Good Luck!!!"
The EC was also used as a platform for peer-to-peer assessment where the feedback was pri-
vately delivered through email to the respective individuals confidentially. Example interaction:
"Hi @contact.first_name, thank you for deciding to rate your friends. Your feedback is confiden-
tial and will be emailed personally to the group members for improvement"
Experience Realistic humanized communication with affective interaction through greeting using users
name and nickname, inspirational quotes and empathy were also designed such as:
"Find a group of people who challenge and inspire you, spend a lot of time with them, and it
will change your life."—Amy Poehler
“Thank you. I am sure you will do an excellent job @contact.first_name!!”
“Thank you @results.nick_name. As you know I am a bot, and I can’t identify if you are a Male/
Female by your name. Therefore, can you let me know your gender?
1: Male
2: Female”
"Interesting……..That looks like a beautiful artwork Picasso !!!!"
"Do you know that you can use Autodraw to design a unique logo for your products?"

Procedure
This study applies an interventional study using a quasi-experimental design
approach. Creswell (2012) explained that education-based research in most cases
requires intact groups, and thus creating artificial groups may disrupt classroom
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 12 of 28

Fig. 1 Textit flow diagrams

Fig. 2 Textit flow diagram for group registration

Fig. 3 Textit flow diagram for peer to peer evaluation

learning. Therefore, one group pretest–posttest design was applied for both groups
in measuring learning outcomes, except for learning performance and perception of
learning which only used the post-test design. The total intervention time was ten
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 13 of 28

Fig. 4 Telegram screenshots of the EC

weeks, as represented in Fig. 5. The EC is usually deployed for the treatment class one
day before the class except for EC6 and EC10, which were deployed during the class.
Such a strategy was used to ensure that the instructor could guide the students the
next day if there were any issues.

Measures
This study integrates five instruments which measure perception of learning (Silva
et al., 2017), perceived motivation belief using the Motivated Strategies for Learning
Questionnaire (MSLQ) (Pintrich & de Groot, 1990) and modified MSLQ (Silva et al.,
2017), need for cognition using the Need for Cognition Scale–6 (NCS-6) (de Holanda
Coelho et al., 2020), creative self-efficacy from the Creative Self-Efficacy (QCSE)
(Brockhus et al., 2014) and teamwork using a modified version of Team Assessment
Survey Questions (Linse, 2007). The teamwork survey had open-ended questions,
which are:
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 14 of 28

Treatment group Control group

(EC) N=30 (Tradional) N=30

Pre-test
Week 3 • Pre-class: Need for cognition and creative
self-efficacy.
• Post-class: motivational belief

EC1 Week 4

EC2 Week 5

EC3 Week 6 Pre-test - Team assessment

EC4 Week 7

EC5 Week 8

EC6 Week 9

EC7 Week 10

EC8 Week 11

EC9 Post-test
Week 12 • Pre-class: Need for cognition and creative
self-efficacy.
• Post-class: motivational belief

EC10 Week 13

Week 14 Post-test - Team assessment and percepon of learning

Learning Performance

Fig. 5 Study procedure

i. Give one specific example of something you learned from the team that you prob-
ably would not have learned on your own.
ii. Give one specific example of something other team members learned from you
that they probably would not have learned without you.
iii. What problems have you had interacting as a team so far?
iv. Suggest one specific, practical change the team could make that would help
improve everyone’s learning.

The instruments were rated based on the Likert scale ranging from 1 (strongly disa-
gree) to 5 (strongly agree) and administered using Google Forms for both groups. Where
else, learning performance was assessed based on the assessment of the project, which
includes report, product, presentation, and peer-to-peer assessment.
A series of one-way analyses of covariance (ANCOVA) was employed to evalu-
ate the difference between the EC and CT groups relating to the need for cognition,
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 15 of 28

motivational belief for learning, creative self-efficacy, and team assessment. As for
learning performance, and perception of learning, a t-test was used to identify the dif-
ference between the groups. The effect size was evaluated according to Hattie (2015),
where an average effect size (Cohen’s d) of 0.42 for an intervention using technologies
for college students is reflected to improve achievement (Hattie, 2017). Furthermore,
as the teamwork has open-ended questions, the difference between the groups was
evaluated qualitatively using Text analysis performed using the Voyant tool at https://​
voyant-​tools.​org/ (Sinclair & Rockwell, 2021). Voyant tools is an open-source online
tool for text analysis and visualization (Hetenyi et al., 2019), and in this study, the col-
locates graphs were used to represent keywords and terms that occur in close proxim-
ity representing a directed network graph.

Results
Learning performance for the course
The EC group (µ = 42.500, SD = 2.675) compared the CT group (µ = 39.933,
SD = 2.572) demonstrated significant difference at t (58) = 3.788, p = 0.000, d = 0.978;
hence indicating difference in learning achievement where the EC group outper-
formed the control group. The Cohen’s d value as described by Hattie (2017) indi-
cated that learning performance improved by the intervention.

Need for cognition

The initial Levine’s test and normality indicated that the homogeneity of variance
assumptions was met at F (1,58) = 0.077, p = 0.782. The adjusted means of µ = 3.416
for the EC group and µ = 3.422 for the CT group indicated that the post-test scores
were not significant at F (1, 57) = 0.002, p = 0.969, η2p = 0.000, d = 0.012); hence indi-
cating that student’s perception of enjoyment and tendency to engage in the course is
similar for both groups.

Motivational beliefs
The initial Levine’s test and normality indicated that the homogeneity of variance
assumptions was met at F (1,58) = 0.062, p = 0.804. The adjusted means of µ = 4.228
for the EC group and µ = 4.200 for the CT group indicated that the post-test scores
were not significant at F (1, 57) = 0.046, p = 0.832, η2p = 0.001, d = 0.056); hence
indicating that the student’s motivation to engage in the course are similar for both
groups.

Creative self‑efficacy
The initial Levine’s test and normality indicated that the homogeneity of variance
assumptions was met at F (1,58) = 0.808, p = 0.372. The adjusted means of µ = 3.566 for
the EC group and µ = 3.627 for the CT group indicated that the post-test scores were not
significant at F (1, 57) = 0.256, p = 0.615, η2p = 0.004, d = 0.133); hence indicating that
the student’s perception of creative self-efficacy was similar for both groups.
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 16 of 28

Table 6 Learning activities impacting project design and development

Groups Online feedback and Group activities Interaction with lecture
guidance

EC 30.00% (N = 9) 50.00% (N = 15) 20.00% (N = 6)

CT 0.10% (N = 3) 86.67% (N = 26) 0.06% (N = 2)

Table 7 Constructive feedback source

Groups Group mates Instructor Other students Others (example: industry
experts, Web programmer)

EC 56.67% (N = 17) 36.67% (N = 11) 0.06% (N = 2) 0

CT 50.00% (N = 15) 43.33% (N = 13) 0.06% (N = 2) 0

Perception of learning
The EC group (µ = 4.370, SD = 0.540) compared the CT group (µ = 4.244, SD = 0.479)
demonstrated no significant difference at t (58) = 0.956, p = 0.343, d = 0.247; hence
indicating no difference in how students perceived their learning process quantitively.
Nevertheless, we also questioned what impacted their learning (project design and
development) the most during the course, and the findings, as shown in Table 6, indi-
cated that both groups (EC = 50.00% and CT = 86.67%) found the group learning activity
as having the most impact. The control group was more partial towards the group activi-
ties than the EC group indicating online feedback and guidance (30.00%) and interac-
tion with the lecturer as an inequitable influence. It was also indicated in both groups
that constructive feedback was mostly obtained from fellow course mates (EC = 56.67%,
CT = 50.00%) and the instructor (EC = 36.67%, CT = 43.33%) (Table 7) while minimum
effort was made to get feedback outside the learning environment.

Team assessment
The initial Levine’s test and normality indicated that the homogeneity of variance
assumptions was met at F (1,58) = 3.088, p = 0.051. The adjusted means of µ = 4.518
for the experimental group and µ = 4.049 for the CT group indicated that the post-test
scores were significantly different at F (1, 57) = 5.950, p = 0.018, η2p = 0.095, d = 0.641;
hence indicating that there was a significant difference between groups in how they per-
formed in teams. The Cohen’s d value, as described by Hattie (2017), indicated that the
intervention improved teamwork.
Next, we questioned their perception of teamwork based on what they learned from
their teammates, what they felt others learn from them, the problem faced as a team,
and recommendations to improve their experience in the course. Based on the feedback,
themes such as teamwork, technology, learning management, emotional management,
creativity, and none were identified to categories the feedback. The descriptive data are
represented in Table 8 for both the groups and the trends reflecting the changes in feed-
back are described as follow:
 Kumar Int J Educ Technol High Educ

Table 8 Comparison between EC and CT teamwork perception

Learn from others Others learn from you Problems Recommendation
(2021) 18:65

PRE Post PRE Post PRE Post PRE Post

Themes A B A B A B A B A B A B A B A B

Team- 30.00% 40.00% 43.33% 43.33% 13.33% 16.67% 23.33% 16.67% 30.00% 36.67% 50.00% 66.67% 50.00% 90.00% 90.00% 53.33%
work (N = 9) (N = 12) (N = 13) (N = 13) (N = 4) (N = 5) (N = 7) (N = 5) (N = 9) (N = 11) (N = 15) (N = 20) (N = 15) (N = 27) (N = 27) (N = 16)
Technol- 26.67% 30.00% 26.67% 30.00% 3.33% 3.33% 23.33% 16.67% 6.67% 3.33% 6.67% – 16.67% 3.33% – 3.33%
ogy (N = 8) (N = 9) (N = 8) (N = 9) (N = 1) (N = 1) (N = 7) (N = 5) (N = 2) (N = 1) (N = 2) (N = 5) (N = 1) (N = 1)
Creativity 30.00% 6.67% 10.00% 3.33% 30.00% 10.00% 6.67% 6.67% 13.33% – – – 3.33% – – –
(N = 9) (N = 2) (N = 3) (N = 1) (N = 9) (N = 3) (N = 2) (N = 2) (N = 4) (N = 1)
Learning 10.00% 13.33% 6.67% 10.00% 13.33% 16.67% 23.33% 6.67% 36.67% 6.67% 16.67% 30.00% 3.33% 6.67% 23.33%
manage- (N = 3) (N = 4) (N = 2) (N = 3) (N = 4) (N = 5) (N = 7) (N = 2) (N = 11) (N = 2) (N = 5) (N = 9) (N = 1) (N = 2) (N = 7)
ment
Emotional 3.33% 10.00% 6.67% 13.33% 13.33% 10.00% 16.67% 16.67% – – 3.33% – – – – 10.00%
manage- (N = 1) (N = 3) (N = 2) (N = 4) (N = 4) (N = 3) (N = 5) (N = 5) (N = 1) (N = 3)
ment
Others – – – 3.33% 30.00% 46.67% 13.33% 6.67% 10.00% 23.33% 6.67% 16.67% – 3.33% 3.33% 10.00%
(N = 1) (N = 9) (N = 14) (N = 4) (N = 2) (N = 3) (N = 7) (N = 2) (N = 5) (N = 1) (N = 1) (N = 3)
Page 17 of 28
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 18 of 28

i. Respondent learned from teammates

This question reflects on providing feedback on one aspect they have learned from
their team that they probably would not have learned independently. Based on Fig. 6,
the illustration describes changes in each group (EC and CT) pre and post-interven-
tion. First, teamwork showed an increasing trend for EC, whereas CT showed slight
changes pre and post-intervention.
Next, using text analysis collocates graphs (Fig. 7) for EC post-intervention, a
change was observed indicating teamwork perception resonating from just learning
new ideas, communicating, and accepting opinions towards a need to cooperate as
a team to ensure they achieve their goal of developing the project. It was observed
that communicating merely was not the main priority anymore as cooperation
towards problem-solving is of utmost importance. Example feedbacks are such as,
“I learned teamwork and how to solve complicated problems” and “The project was
completed in a shorter period of time, compared to if I had done it by myself.” Next,
in both groups, creativity seems to have declined from being an essential aspect in
the project’s initial phase as it declines towards the end of the semester, whereas an
increment was noticed in giving more importance to emotional management when
handling matters of the project. Example feedback is such as “I learn to push myself

Fig. 6 Change in perception pre and post-intervention based on aspects learn from teammates

Fig. 7 Change in perception for the EC group based on aspects learn from teammate
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 19 of 28

Fig. 8 Change in perception pre and post-intervention based on aspects teammates learned from
respondents

more and commit to the project’s success.” Nevertheless, in both groups, all the
trends are almost similar.

ii. Teammates learned from the respondent.

This question reflects on an aspect the respondent believes that their team members
have learned from them. Initially, both groups reported being unaware of their contri-
bution by stating “nothing” or “I don’t know” which was classified as “other” (Fig. 8).
Nevertheless, intriguingly both groups showed a decline in such negative perception
post-intervention, which can be attributed to self-realization of their contribution in
task completion. Furthermore, different trends were observed between both groups for
teamwork, where the EC group showed more references to increased teamwork con-
tribution, where else the CT group remained unaffected post-intervention. In terms of
technology application, the respondents in both groups described how they were a valu-
able resource for teaching their peers about technology, where one respondent stated
that “My friends learn how to make an application from me.”

iii. Problem respondent faced as a team

Based on the analysis, it was found that the main issue faced in both groups were
related to teamwork (Fig. 9). The CT group reflected higher teamwork issues than the
EC group, and in both groups, these issues escalated during the learning process.
Based on analyzing the text, initially, the EC group found issues related to identifying
an appropriate time to have group discussions as some teammates were either absent or
unavailable (Fig. 10), where a respondent stated that “We can barely meet as a group.”
Post-intervention, the group found similar issues, highlighting a lack of communication
and availability due to insufficient time and being busy with their learning schedule. Exam-
ple respond, “We do not have enough time to meet up, and most of us have other work
to do.” As for the CT group pre-intervention, similar issues were observed as denoted for
the EC group, but communication issues were more prevalent as respondents mentioned
differences in opinions or void in feedback which affected how they solved problems col-
lectively (Fig. 11). Example feedback is “One of the members rarely responds in the group
discussion.” Post-intervention, the CT group claimed that the main issues besides com-
munication were non-contributing members and bias in task distribution. Examples are
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 20 of 28

Fig. 9 Graphical representation of issues faced as a team

Fig. 10 Change in perception for the EC group based on issues faced as a team

Fig. 11 Change in perception for the CT group based on issues faced as a team

“Some of my teammates were not really contributing” and “The task was not distributed
fairly.”

iv. Recommendations to improve teamwork

Two interesting trends were observed from Fig. 12, which are (a) EC group reflected
more need teamwork whereas the CT group showed otherwise (b) CT group empha-
sized learning management for teamwork whereas the EC group showed otherwise.
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 21 of 28

Fig. 12 Graphical representation of recommendations pre and post-intervention for both groups

Fig. 13 Changes in perception for the EC group based on recommendations for learning improvement as a
team

Fig. 14 Changes in perception for the CT group based on recommendations for learning improvement as a
team

When assessing the changes in the EC group (Fig. 13), transformations were observed
between pre and post-intervention, where students opined the need for more active col-
laboration in providing ideas and acceptance. One respondent from the treatment group
reflected that acceptance is vital for successful collaboration, stating that “Teamwork
and acceptance in a group are important.” Next, for the CT group (Fig. 14), the com-
plexity of defining teamwork pre-intervention, such as communicating, confidence, and
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 22 of 28

contribution of ideas, was transformed to reflect more need for commitment by stating,
“Make sure everyone is committed and available to contribute accordingly.”

Discussion
According to Winkler and Söllner (2018), ECs have the potential to improve learn-
ing outcomes due to their ability to personalize the learning experience. This study
aims to evaluate the difference in learning outcomes based on the impact of EC on a
project-based learning activity. The outcomes were compared quantitively and quali-
tatively to explore how the introduction of EC will influence learning performance,
need for cognition, motivational belief, creative self-efficacy, perception of learn-
ing, and teamwork. Based on the findings, EC has influenced learning performance
(d = 0.978) and teamwork (d = 0.641), and based on the Cohen’s d value being above
0.42, a significant impact on the outcome was deduced. However, other outcomes
such as the need for cognition, motivational belief, creative self-efficacy, and percep-
tion of learning did not reflect significant differences between both groups.
Firstly, Kearney et al. (2009) explained that in homogenous teams (as investigated
in this study), the need for cognition might have a limited amount of influence as
both groups are required to be innovative simultaneously in providing project solu-
tions. Lapina (2020) added that problem-based learning and solving complex prob-
lems could improve the need for cognition. Hence, when both classes had the same
team-based project task, the homogenous nature of the sampling may have attributed
to the similarities in the outcome that overshadowed the effect of the ECs. Equally, for
motivational belief, which is the central aspect needed to encourage strategic learning
behavior (Yen, 2018). A positive relation with cognitive engagement, performance,
and the use of metacognitive strategies (Pintrich & de Groot, 1990) is accredited to
the need to regulate and monitor learning (Yilmaz & Baydas, 2017), especially for
project-based learning activities (Sart, 2014). Therefore, in both groups, due to the
same learning task, these attributes are apparent for both groups as they were able
to complete their task (cognitive engagement), and to do so, they were required to
plan their task, schedule teamwork activities (metacognition), and design and develop
their product systematically.
Moreover, individual personality traits such as motivation have also been found to
influence creativity (van Knippenberg & Hirst, 2020) which indirectly influenced the
need for cognition (Pan et al., 2020). Nevertheless, these nonsignificant findings may
have some interesting contribution as it implies that project-based learning tends to
improve these personality-based learning outcomes. At the same time, the introduc-
tion of ECs did not create cognitive barriers that would have affected the cognition,
motivational and creative processes involved in project-based learning. Furthermore,
as there is a triangulated relationship between these outcomes, the author specu-
lates that these outcomes were justified, especially with the small sample size used, as
Rosenstein (2019) explained.
However, when EC is reflected as a human-like conversational agent (Ischen et al.,
2020) used as a digital assistant in managing and monitoring students (Brindha et al.,
2019), the question arises on how do we measure such implication and confirm its capa-
bilities in transforming learning? As a digital assistant, the EC was designed to aid in
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 23 of 28

managing the team-based project where it was intended to communicate with students
to inquire about challenges and provide support and guidance in completing their tasks.
According to Cunningham-Nelson et al. (2019), such a role improves academic perfor-
mance as students prioritize such needs. Conversely, for teamwork, technology-medi-
ated communication, such as in ECs, has been found to encourage interaction in team
projects (Colace et al., 2018) as they perceived the ECs as helping them to learn more,
even when they have communication issues (Fryer et al., 2019). Therefore, supporting
the outcome of this study that observed that the EC groups learning performance and
teamwork outcome had a more significant effect size than the CT group.
As for the qualitative findings, firstly, even though the perception of learning did
not show much variation statistically, the EC group showed additional weightage
that implicates group activities, online feedback, and interaction with the lecturer
as impactful. Interestingly, the percentage of students that found “interaction with
lecturer” and “online feedback and guidance” for the EC was higher than the control
group, and this may be reflected as a tendency to perceive the chatbot as an embodi-
ment of the lecturer. Furthermore, as for constructive feedback, the outcomes for
both groups were very similar as the critiques were mainly from the teammates and
the instructor, and the ECs were not designed to critique the project task.
Next, it was interesting to observe the differences and the similarities in both
groups for teamwork. In the EC group, there were changes in terms of how students
identified learning from other individual team members towards a collective per-
spective of learning from the team. Similarly, there was also more emphasis on how
they contributed as a team, especially in providing technical support. As for CT, not
much difference were observed pre and post-intervention for teamwork; however,
the post-intervention in both groups reflected a reduced need for creativity and
emphasizing the importance of managing their learning task cognitively and emo-
tionally as a team. Concurrently, it was evident that the self-realization of their value
as a contributing team member in both groups increased from pre-intervention to
post-intervention, which was higher for the CT group.
Furthermore, in regard to problems faced, it was observed that in the EC group,
the perception transformed from collaboration issues towards communicative
issues, whereas it was the opposite for the CT group. According to Kumar et al.
(2021), collaborative learning has a symbiotic relationship with communication skills
in project-based learning. This study identifies a need for more active collaboration
in the EC group and commitment for the CT group. Overall, it can be observed that
the group task performed through ECs contributed towards team building and col-
laboration, whereas for the CT group, the concept of individuality was more appar-
ent. Interestingly, no feedback from the EC group mentioned difficulties in using the
EC nor complexity in interacting with it. It was presumed that students welcomed
such interaction as it provided learning support and understood its significance.
Furthermore, the feedbacks also justified why other variables such as the need for
cognition, perception of learning, creativity, self-efficacy, and motivational belief did
not show significant differences. For instance, both groups portrayed high self-reali-
zation of their value as a team member at the end of the course, and it was deduced
that their motivational belief was influenced by higher self-efficacy and intrinsic value.
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 24 of 28

Next, in both groups, creativity was overshadowed by post-intervention teamwork sig-

nificance. Therefore, we conclude that ECs significantly impact learning performance
and teamwork, but affective-motivational improvement may be overshadowed by the
homogenous learning process for both groups. Furthermore, it can be perceived that
the main contribution of the ECs was creating a “team spirit” especially in completing
administrative tasks, interactions, and providing feedback on team progress, and such
interaction was fundamental in influencing their learning performance.

Theoretical and practical implication

This study report theoretical and practical contributions in the area of educational
chatbots. Firstly, given the novelty of chatbots in educational research, this study
enriched the current body of knowledge and literature in EC design characteristics
and impact on learning outcomes. Even though the findings are not practically sat-
isfactory with positive outcomes regarding the affective-motivational learning out-
comes, ECs as tutor support did facilitate teamwork and cognitive outcomes that
support project-based learning in design education. In view of that, it is worth noting
that the embodiment of ECs as a learning assistant does create openness in interac-
tion and interpersonal relationships among peers, especially if the task were designed
to facilitate these interactions.

Limitation and future studies

This study focuses on using chatbots as a learning assistant from an educational perspec-
tive by comparing the educational implications with a traditional classroom. Therefore, the
outcomes of this study reflected only on the pedagogical outcomes intended for design
education and project-based learning and not the interaction behaviors. Even though
empirical studies have stipulated the role of chatbots in facilitating learning as a com-
municative agent, nevertheless instructional designers should consider the underdevel-
oped role of an intelligent tutoring chatbot (Fryer et al., 2019) and question its limits in
an authentic learning environment. As users, the students may have different or higher
expectations of EC, which are potentially a spillover from use behavior from chatbots
from different service industries. Moreover, questions to ponder are the ethical implica-
tion of using EC, especially out of the learning scheduled time, and if such practices are
welcomed, warranted, and accepted by today’s learner as a much-needed learning strat-
egy. According to Garcia Brustenga et al. (2018), while ECs can perform some adminis-
trative tasks and appear more appealing with multimodal strategies, the author questions
how successful such strategies will be as a personalized learning environment without the
teacher as the EC’s instructional designer. Therefore, future studies should look into edu-
cators’ challenges, needs, and competencies and align them in fulfill EC facilitated learning
goals. Furthermore, there is much to be explored in understanding the complex dynamics
of human–computer interaction in realizing such a goal, especially educational goals that
are currently being influenced by the onset of the Covid-19 pandemic. Conversely, future
studies should look into different learning outcomes, social media use, personality, age,
culture, context, and use behavior to understand the use of chatbots for education.
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 25 of 28

Abbreviations
EC: Educational chatbots; CT: Control group; Ripe: Reliability, interpersonal communication, pedagogy, and experience;
GORE: Goal-oriented requirements engineering.

Acknowledgements
Not applicable.

Authors’ contributions
The author read and approved the final manuscript.

Funding
This study was funded under the Universiti Sains Malaysia Short Term Research Grant 304/PMEDIA/6315219.

Availability of data and materials

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Declarations
Competing interests
The author declares that there is no conflict of interest.

Ethical approval and consent to participate

Informed consent was obtained from all participants for being included in the study based on the approval of The
Human Research Ethics Committee of Universiti Sains Malaysia (JEPeM) Ref No: USM/JEPeM/18050247.

Received: 2 July 2021 Accepted: 23 September 2021

References
Adamopoulou, E., & Moussiades, L. (2020). An overview of chatbot technology. In: Maglogiannis, I., Iliadis, L. & Pimenidis,
E. (eds) Artificial intelligence applications and innovations. AIAI 2020. IFIP advances in information and communication
technology, vol 584 (pp. 373–383). Springer. https://​doi.​org/​10.​1007/​978-3-​030-​49186-4_​31.
Arruda, D., Marinho, M., Souza, E. & Wanderley, F. (2019) A Chatbot for goal-oriented requirements modeling. In: Misra S.
et al. (eds) Computational science and its applications—ICCSA 2019. ICCSA 2019. Lecture Notes in Computer Science, vol
11622 (pp. 506–519). Springer. https://​doi.​org/​10.​1007/​978-3-​030-​24305-0_​38.
Bii, P. (2013). Chatbot technology: A possible means of unlocking student potential to learn how to learn. Educational
Research, 4(2), 218–221.
Brandtzaeg, P. B., & Følstad, A. (2018). Chatbots: User changing needs and motivations. Interactions, 25(5), 38–43. https://​
doi.​org/​10.​1145/​32366​69.
Brindha, S., Dharan, K. R. D., Samraj, S. J. J., & Nirmal, L. M. (2019). AI based chatbot for education management. Interna-
tional Journal of Research in Engineering, Science and Management, 2(3), 2–4.
Brockhus, S., van der Kolk, T. E. C., Koeman, B., & Badke-Schaub, P. G. (2014). The influence of ambient green on creative
performance. Proceeding of International Design Conference (DESIGN 2014), Croatia, 437–444.
Cacioppo, J. T., Petty, R. E., Feinstein, J. A., & Jarvis, W. B. G. (1996). Dispositional differences in cognitive motivation: The life
and times of individuals varying in need for cognition. Psychological Bulletin, 119, 197–253.
Cameron, G., Cameron, D. M., Megaw, G., Bond, R. B., Mulvenna, M., O’Neill, S. B., Armour, C., & McTear, M. (2018). Back to
the future: Lessons from knowledge engineering methodologies for chatbot design and development. Proceedings
of British HCI 2018, 1–5. https://​doi.​org/​10.​14236/​ewic/​HCI20​18.​153.
Chan, T. J., Yong, W. K. Y., & Harmizi, A. (2020). Usage of whatsapp and interpersonal communication skills among private
university students. Journal of Arts & Social Sciences, 3(January), 15–25.
Chaves, A. P., & Gerosa, M. A. (2021). How should my chatbot interact? A survey on social characteristics in human–chat-
bot interaction design. International Journal of Human-Computer Interaction, 37(8), 729–758. https://​doi.​org/​10.​1080/​
10447​318.​2020.​18414​38.
Chen, H. L., Widarso, G. V., & Sutrisno, H. (2020). A chatbot for learning Chinese: Learning achievement and technology
acceptance. Journal of Educational Computing Research, 58(6), 1161–1189. https://​doi.​org/​10.​1177/​07356​33120​
929622.
Chete, F. O., & Daudu, G. O. (2020). An approach towards the development of a hybrid chatbot for handling students’
complaints. Journal of Electrical Engineering, Electronics, Control and Computer Science, 6(22), 29–38.
Chocarro, R., Cortiñas, M., & Marcos-Matás, G. (2021). Teachers’ attitudes towards chatbots in education: A technology
acceptance model approach considering the effect of social language, bot proactiveness, and users’ characteristics.
Educational Studies, 00(00), 1–19. https://​doi.​org/​10.​1080/​03055​698.​2020.​18504​26.
Ciechanowski, L., Przegalinska, A., Magnuski, M., & Gloor, P. (2019). In the shades of the uncanny valley: An experimental
study of human–chatbot interaction. Future Generation Computer Systems, 92, 539–548. https://​doi.​org/​10.​1016/j.​
future.​2018.​01.​055.
Colace, F., De Santo, M., Lombardi, M., Pascale, F., Pietrosanto, A., & Lemma, S. (2018). Chatbot for e-learning: A case of
study. International Journal of Mechanical Engineering and Robotics Research, 7(5), 528–533. https://​doi.​org/​10.​18178/​
ijmerr.​7.5.​528-​533.
Conde, M. Á., Rodríguez-Sedano, F. J., Hernández-García, Á., Gutiérrez-Fernández, A., & Guerrero-Higueras, Á. M. (2021).
Your teammate just sent you a new message! The effects of using Telegram on individual acquisition of teamwork
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 26 of 28

competence. International Journal of Interactive Multimedia and Artificial Intelligence, 6(6), 225. https://​doi.​org/​10.​
9781/​ijimai.​2021.​05.​007.
Cunningham-Nelson, S., Boles, W., Trouton, L., & Margerison, E. (2019). A review of chatbots in education: practical steps
forward. In 30th Annual conference for the australasian association for engineering education (AAEE 2019): Educators
becoming agents of change: innovate, integrate, motivate. Engineers Australia, 299–306.
Creswell, J. W. (2012). Educational Research : Planning, Conducting and Evaluating Quantitative and Qualitative Research (4th
ed.). Pearson Education.
de Holanda Coelho, G. L., Hanel, H. P., & Wolf, J. L. (2020). The very efficient assessment of need for cognition: Developing
a six-item version. Assessment, 27(8), 1870–1885. https://​doi.​org/​10.​1177/​10731​91118​793208.
de Oliveira, J. C., Santos, D. H., & Neto, M. P. (2016). Chatting with Arduino platform through Telegram Bot. 2016 IEEE Inter-
national Symposium on Consumer Electronics (ISCE), 131–132. https://​doi.​org/​10.​1109/​ISCE.​2016.​77974​06.
Dekker, I., De Jong, E. M., Schippers, M. C., De Bruijn-Smolders, M., Alexiou, A., & Giesbers, B. (2020). Optimizing students’
mental health and academic performance: AI-enhanced life crafting. Frontiers in Psychology, 11(June), 1–15. https://​
doi.​org/​10.​3389/​fpsyg.​2020.​01063.
Devito, J. (2018). The interpersonal communication book (15th ed.). Pearson Education Limited.
Eccles, J. S., & Wigfield, A. (2002). Motivational Beliefs, Values, and Goals. Annual Review of Psychology, 53(1), 109–132.
https://​doi.​org/​10.​1146/​annur​ev.​psych.​53.​100901.​135153.
Eeuwen, M. V. (2017). Mobile conversational commerce: messenger chatbots as the next interface between businesses and
consumers. Unpublished Master’s thesis. University of Twente.
Følstad, A., Skjuve, M., & Brandtzaeg, P. B. (2019). Different chatbots for different purposes: towards a typology of chatbots
to understand interaction design. In: Bodrunova S. et al. (eds) Internet Science. INSCI 2018. Lecture Notes in Computer
Science, vol 11551 (pp. 145–156). Springer. https://​doi.​org/​10.​1007/​978-3-​030-​17705-8_​13.
Følstad, A., & Brandtzaeg, P. B. (2017). Chatbots and the new world of HCI. Interactions, 24(4), 38–42. https://​doi.​org/​10.​
1145/​30855​58.
Fryer, L. K., Nakao, K., & Thompson, A. (2019). Chatbot learning partners: Connecting learning experiences, interest and
competence. Computers in Human Behavior, 93, 279–289. https://​doi.​org/​10.​1016/j.​chb.​2018.​12.​023.
Garcia Brustenga, G., Fuertes-Alpiste, M., & Molas-Castells, N. (2018). Briefing paper: Chatbots in education. eLearn Center,
Universitat Oberta de Catalunya. https://​doi.​org/​10.​7238/​elc.​chatb​ots.​2018.
Gonda, D. E., Luo, J., Wong, Y. L., & Lei, C. U. (2019). Evaluation of developing educational chatbots based on the seven
principles for good teaching. Proceedings of the 2018 IEEE international conference on teaching, assessment, and learn-
ing for engineering, TALE 2018, Australia, 446–453. IEEE. https://​doi.​org/​10.​1109/​TALE.​2018.​86151​75.
Hadjielias, E., Dada, O., Discua Cruz, A., Zekas, S., Christofi, M., & Sakka, G. (2021). How do digital innovation teams func-
tion? Understanding the team cognition-process nexus within the context of digital transformation. Journal of
Business Research, 122, 373–386. https://​doi.​org/​10.​1016/j.​jbusr​es.​2020.​08.​045.
Han, R., & Xu, J. (2020). A comparative study of the role of interpersonal communication, traditional media and social
media in pro-environmental behavior: A China-based study. International Journal of Environmental Research and
Public Health. https://​doi.​org/​10.​3390/​ijerp​h1706​1883.
Haristiani, N., Danuwijaya, A. A., Rifai, M. M., & Sarila, H. (2019). Gengobot: A chatbot-based grammar application on
mobile instant messaging as language learning medium. Journal of Engineering Science and Technology, 14(6),
3158–3173.
Hattie, J. (2017). Visible Learningplus 250+ influences on student achievement. In Visible learning plus. www.​visib​lelea​
rning​plus.​com/​conte​nt/​250-​influ​ences-​stude​nt-​achie​vement.
Hattie, J. (2015). The applicability of Visible Learning to higher education. Scholarship of Teaching and Learning in Psychol-
ogy, 1(1), 79–91. https://​doi.​org/​10.​1037/​stl00​00021.
Heryandi, A. (2020). Developing chatbot for academic record monitoring in higher education institution. IOP Conference
Series: Materials Science and Engineering. https://​doi.​org/​10.​1088/​1757-​899X/​879/1/​012049.
Hetenyi, G., Lengyel, A., & Szilasi, M. (2019). Quantitative analysis of qualitative data: Using voyant tools to investigate the
sales-marketing interface. Journal of Industrial Engineering and Management, 12(3), 393–404. https://​doi.​org/​10.​3926/​
jiem.​2929.
Hobert, S. (2019). How are you, chatbot? Evaluating chatbots in educational settings—Results of a literature review. In N.
Pinkwart & J. Konert (Eds.), DELFI 2019 (pp. 259–270). Gesellschaft für Informatik, Bonn. https://​doi.​org/​10.​18420/​delfi​
2019_​289.
Hobert S. & Berens F. (2020). Small talk conversations and the long-term use of chatbots in educational settings—experi-
ences from a field study. In: Følstad A. et al. (eds) Chatbot research and design. CONVERSATIONS 2019. Lecture Notes in
Computer Science, vol 11970 (pp. 260–272). Springer. https://​doi.​org/​10.​1007/​978-3-​030-​39540-7_​18.
Holotescu, C. (2016). MOOCBuddy: A chatbot for personalized learning with MOOCs. In: A. Iftene & J. Vanderdonckt (Eds.),
Proceedings of the 13th international conference on human-computer interaction RoCHI’2016, Romania, 91–94.
Ischen C., Araujo T., Voorveld H., van Noort G., Smit E. (2020) Privacy concerns in chatbot interactions. In: Følstad A. et al.
(eds) Chatbot research and design. CONVERSATIONS 2019. Lecture Notes in Computer Science, vol 11970 (pp. 34–48).
Springer. https://​doi.​org/​10.​1007/​978-3-​030-​39540-7_3.
Ismail, M., & Ade-Ibijola, A. (2019). Lecturer’s Apprentice: A chatbot for assisting novice programmers. Proceedings—2019
International multidisciplinary information technology and engineering conference, IMITEC 2019. South Africa, 1–8. IEEE.
https://​doi.​org/​10.​1109/​IMITE​C45504.​2019.​90158​57.
Kearney, E., Gebert, D., & Voelpel, S. (2009). When and how diversity benefits teams: The importance of team members’
need for cognition. Academy of Management Journal, 52(3), 581–598. https://​doi.​org/​10.​5465/​AMJ.​2009.​41331​431.
Kerly, A., Hall, P., & Bull, S. (2007). Bringing chatbots into education: Towards natural language negotiation of open learner
models. Knowledge-Based Systems, 20(2), 177–185. https://​doi.​org/​10.​1016/j.​knosys.​2006.​11.​014.
Khan, A., Ranka, S., Khakare, C., & Karve, S. (2019). NEEV: An education informational chatbot. International Research Journal
of Engineering and Technology, 6(4), 492–495.
Kim, M. S. (2021). A systematic review of the design work of STEM teachers. Research in Science & Technological Education,
39(2), 131–155. https://​doi.​org/​10.​1080/​02635​143.​2019.​16829​88.
Kumar Int J Educ Technol High Educ (2021) 18:65 Page 27 of 28

Kumar, J. A., & Silva, P. A. (2020). Work-in-progress: A preliminary study on students’ acceptance of chatbots for studio-
based learning. Proceedings of the 2020 IEEE Global Engineering Education Conference (EDUCON), Portugal, 1627–1631.
IEEE https://​doi.​org/​10.​1109/​EDUCO​N45650.​2020.​91251​83.
Kumar, J. A., Bervell, B., Annamalai, N., & Osman, S. (2020). behavioral intention to use mobile learning : Evaluating the role
of self-efficacy, subjective norm, and WhatsApp use habit. IEEE Access, 8, 208058–208074. https://​doi.​org/​10.​1109/​
ACCESS.​2020.​30379​25.
Kumar, J. A., Silva, P. A., & Prelath, R. (2021). Implementing studio-based learning for design education: A study on the
perception and challenges of Malaysian undergraduates. International Journal of Technology and Design Education,
31(3), 611–631. https://​doi.​org/​10.​1007/​s10798-​020-​09566-1.
Lapina, A. (2020). Does exposure to problem-based learning strategies increase postformal thought and need for cognition in
higher education students? A quasi-experimental study (Publication No. 28243240) Doctoral dissertation, Texas State
University-San Marcos. ProQuest Dissertations & Theses Global.
Linse, A. R. (2007). Team peer evaluation. In Schreyer Institute for Teaching Excellence. http://​www.​schre​yerin​stitu​te.​psu.​
edu/.
Luo, C. J., & Gonda, D. E. (2019). Code Free Bot: An easy way to jumpstart your chatbot! Proceeding of the 2019 IEEE Interna-
tional Conference on Engineering, Technology and Education (TALE 2019), Australia, 1–3, IEEE. https://​doi.​org/​10.​1109/​
TALE4​8000.​2019.​92260​16.
Meyer von Wolff, R., Nörtemann, J., Hobert, S., Schumann, M. (2020) Chatbots for the information acquisition at universi-
ties—A student’s view on the application area. In: Følstad A. et al. (eds) Chatbot research and design. CONVERSATIONS
2019. Lecture Notes in Computer Science, vol 11970 (pp. 231–244). Springer. https://​doi.​org/​10.​1007/​978-3-​030-​39540-
7_​16.
Miller, E. (2016). How chatbots will help education. Venturebeat. http://​ventu​rebeat.​com/​2016/​09/​29/​how-​chatb​ots-​will-​
help-​educa​tion/.
Nguyen, Q. N., & Sidorova, A. (2018). Understanding user interactions with a chatbot: A self-determination theory
approach. Proceedings of the Twenty-Fourth Americas Conference on Information Systems, United States of America,
1–5. Association for Information Systems (AIS).
Oke, A., & Fernandes, F. A. P. (2020). Innovations in teaching and learning: Exploring the perceptions of the education
sector on the 4th industrial revolution (4IR). Journal of Open Innovation: Technology, Market, and Complexity., 6(2), 31.
https://​doi.​org/​10.​3390/​JOITM​C6020​031.
Ondas, S., Pleva, M., & Hladek, D. (2019). How chatbots can be involved in the education process. Proccedings of the 17th
IEEE international conference on emerging eLearning technologies and applications ICETA 2019, Slovakia , 575–580.
https://​doi.​org/​10.​1109/​ICETA​48886.​2019.​90400​95.
Pan, Y., Shang, Y., & Malika, R. (2020). Enhancing creativity in organizations: The role of the need for cognition. Manage-
ment Decision. https://​doi.​org/​10.​1108/​MD-​04-​2019-​0516.
Park, H. S., Baker, C., & Lee, D. W. (2008). Need for cognition, task complexity, and job satisfaction. Journal of Management
in Engineering, 24(2), 111–117. https://​doi.​org/​10.​1061/​(asce)​0742-​597x(2008)​24:​2(111).
Pereira, J. (2016). Leveraging chatbots to improve self-guided learning through conversational quizzes. Proceedings of the
Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality—TEEM ’16, Spain, 911–918,
ACM. https://​doi.​org/​10.​1145/​30124​30.​30126​25.
Pereira, J., Fernández-Raga, M., Osuna-Acedo, S., Roura-Redondo, M., Almazán-López, O., & Buldón-Olalla, A. (2019). Pro-
moting learners’ voice productions using chatbots as a tool for improving the learning process in a MOOC. Technol-
ogy, Knowledge and Learning, 24(4), 545–565. https://​doi.​org/​10.​1007/​s10758-​019-​09414-9.
Pham, X. L., Pham, T., Nguyen, Q. M., Nguyen, T. H., & Cao, T. T. H. (2018). Chatbot as an intelligent personal assistant for
mobile language learning. Proceedings of the 2018 2nd international conference on education and e-Learning—ICEEL
2018, Indonesia, 16–21. ACM. https://​doi.​org/​10.​1145/​32910​78.​32911​15.
Pintrich, P. R., & de Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic per-
formance. Journal of Educational Psychology, 82(1), 33–40. https://​doi.​org/​10.​1037/​0022-​0663.​82.1.​33.
Pintrich, P. R., Smith, D. A. F., Garcia, T., & Mckeachie, W. J. (1993). Reliability and predictive validity of the motivated strate-
gies for learning questionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801–813. https://​doi.​org/​
10.​1177/​00131​64493​05300​3024.
Rahayu, Y. S., Wibawa, S. C., Yuliani, Y., Ratnasari, E., & Kusumadewi, S. (2018). The development of BOT API social media
Telegram about plant hormones using Black Box Testing. IOP Conference Series: Materials Science and Engineering.
https://​doi.​org/​10.​1088/​1757-​899X/​434/1/​012132.
Rahman, A. M., Al Mamun, A., & Islam, A. (2018). Programming challenges of chatbot: Current and future prospective. 5th
IEEE Region 10 Humanitarian Technology Conference 2017 (R10-HTC 2017), India, 75–78, IEEE. https://​doi.​org/​10.​1109/​
R10-​HTC.​2017.​82889​10.
Ren, R., Castro, J. W., Acuña, S. T., & De Lara, J. (2019). Evaluation techniques for chatbot usability: A systematic mapping
study. International Journal of Software Engineering and Knowledge Engineering, 29(11–12), 1673–1702. https://​doi.​
org/​10.​1142/​S0218​19401​94001​63.
Riel, J. (2020). Essential features and critical issues with educational chatbots: toward personalized learning via digital
agents. In: M. Khosrow-Pour (Ed.), Handbook of research on modern educational technologies, applications, and man-
agement (pp. 246–262). IGI Global. https://​doi.​org/​10.​1142/​S0218​19401​94001​63.
Rosenstein, L. D. (2019). Research design and analysis: A primer for the non-statistician. Wiley.
Sandoval, Z. V. (2018). Design and implementation of a chatbot in online higher education settings. Issues in Information
Systems, 19(4), 44–52. https://​doi.​org/​10.​48009/4_​iis_​2018_​44-​52.
Sart, G. (2014). The effects of the development of metacognition on project-based learning. Procedia—Social and Behav-
ioral Sciences, 152, 131–136. https://​doi.​org/​10.​1016/j.​sbspro.​2014.​09.​169.
Satow, L. (2017). Chatbots as teaching assistants: Introducing a model for learning facilitation by AI Bots. SAP Community.
https://​blogs.​sap.​com/​2017/​07/​12/​chatb​ots-​as-​teach​ing-​assis​tants-​intro​ducing-​a-​model-​for-​learn​ing-​facil​itati​
on-​by-​ai-​bots/.
 Kumar Int J Educ Technol High Educ (2021) 18:65 Page 28 of 28

Schlagwein, D., Conboy, K., Feller, J., Leimeister, J. M., & Morgan, L. (2017). “Openness” with and without information tech-
nology: A framework and a brief history. Journal of Information Technology, 32(4), 297–305. https://​doi.​org/​10.​1057/​
s41265-​017-​0049-3.
Schmulian, A., & Coetzee, S. A. (2019). The development of Messenger bots for teaching and learning and accounting
students’ experience of the use thereof. British Journal of Educational Technology, 50(5), 2751–2777. https://​doi.​org/​
10.​1111/​bjet.​12723.
Setiaji, H., & Paputungan, I. V. (2018). Design of Telegram Bots for campus information sharing. IOP Conference Series:
Materials Science and Engineering, 325, 1–6. https://​doi.​org/​10.​1088/​1757-​899X/​325/1/​012005.
Silva, P.A., Polo, B.J., Crosby, M.E. (2017). Adapting the studio based learning methodology to computer science educa-
tion. In: Fee S., Holland-Minkley A., Lombardi T. (eds) New directions for computing education (pp. 119–142). Springer.
https://​doi.​org/​10.​1007/​978-3-​319-​54226-3_8.
Sinclair, S., & Rockwell, G. (2021). Voyant tools (2.4). https://​voyant-​tools.​org/.
Sjöström, J., Aghaee, N., Dahlin, M., & Ågerfalk, P. J. (2018). Designing chatbots for higher education practice. Proceedings
of the 2018 AIS SIGED International Conference on Information Systems Education and Research.
Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger.
Computers and Education, 151(February), 103862. https://​doi.​org/​10.​1016/j.​compe​du.​2020.​103862.
Sreelakshmi, A. S., Abhinaya, S. B., Nair, A., & Jaya Nirmala, S. (2019). A question answering and quiz generation chatbot for
education. Grace Hopper Celebration India (GHCI), 2019, 1–6. https://​doi.​org/​10.​1109/​GHCI4​7972.​2019.​90718​32.
Stathakarou, N., Nifakos, S., Karlgren, K., Konstantinidis, S. T., Bamidis, P. D., Pattichis, C. S., & Davoody, N. (2020). Students’
perceptions on chatbots’ potential and design characteristics in healthcare education. In J. Mantas, A. Hasman, & M.
S. Househ (Eds.), The importance of health informatics in public health during a pandemic (Vol. 272, pp. 209–212). IOS
Press. https://​doi.​org/​10.​3233/​SHTI2​00531.
Tamayo, P. A., Herrero, A., Martín, J., Navarro, C., & Tránchez, J. M. (2020). Design of a chatbot as a distance learning assis-
tant. Open Praxis, 12(1), 145. https://​doi.​org/​10.​5944/​openp​raxis.​12.1.​1063.
Tegos, S., Demetriadis, S., Psathas, G. & Tsiatsos T. (2020) A configurable agent to advance peers’ productive dialogue in
MOOCs. In: Følstad A. et al. (eds) Chatbot research and design. CONVERSATIONS 2019. Lecture Notes in Computer Sci-
ence, vol 11970 (pp. 245–259). Springer. https://​doi.​org/​10.​1007/​978-3-​030-​39540-7_​17.
Thirumalai, B., Ramanathan, A., Charania, A., & Stump, G. (2019). Designing for technology-enabled reflective practice:
teachers’ voices on participating in a connected learning practice. In R. Setty, R. Iyenger, M. A. Witenstein, E. J. Byker,
& H. Kidwai (Eds.), Teaching and teacher education: south asian perspectives (pp. 243–273). Palgrave Macmillan. https://​
doi.​org/​10.​1016/​S0742-​051X(01)​00046-4.
van Knippenberg, D., & Hirst, G. (2020). A motivational lens model of person × situation interactions in employee creativ-
ity. Journal of Applied Psychology, 105(10), 1129–1144. https://​doi.​org/​10.​1037/​apl00​00486.
Vázquez-Cano, E., Mengual-Andrés, S., & López-Meneses, E. (2021). Chatbot to improve learning punctuation in Spanish
and to enhance open and flexible learning environments. International Journal of Educational Technology in Higher
Education, 18(1), 33. https://​doi.​org/​10.​1186/​s41239-​021-​00269-8.
Verleger, M., & Pembridge, J. (2019). A pilot study integrating an AI-driven chatbot in an introductory programming
course. Proceeding of the 2018 IEEE Frontiers in Education Conference (FIE), USA. IEEE. https://​doi.​org/​10.​1109/​FIE.​2018.​
86592​82.
Walker, C. O., & Greene, B. A. (2009). The relations between student motivational beliefs and cognitive engagement in
high school. Journal of Educational Research, 102(6), 463–472. https://​doi.​org/​10.​3200/​JOER.​102.6.​463-​472.
Wang, J., Hwang, G., & Chang, C. (2021). Directions of the 100 most cited chatbot-related human behavior research: A
review of academic publications. Computers and Education: Artificial Intelligence, 2, 1–12. https://​doi.​org/​10.​1016/j.​
caeai.​2021.​100023.
Wei, H. C., & Chou, C. (2020). Online learning performance and satisfaction: Do perceptions and readiness matter? Dis-
tance Education, 41(1), 48–69. https://​doi.​org/​10.​1080/​01587​919.​2020.​17247​68.
Winkler, R., & Söllner, M. (2018). Unleashing the potential of chatbots in education: A state-of-the-art analysis. In Academy
of Management Annual Meeting (AOM). https://​www.​alexa​ndria.​unisg.​ch/​254848/​1/​JML_​699.​pdf.
Wu, E. H. K., Lin, C. H., Ou, Y. Y., Liu, C. Z., Wang, W. K., & Chao, C. Y. (2020). Advantages and constraints of a hybrid model
K-12 E-Learning assistant chatbot. IEEE Access, 8, 77788–77801. https://​doi.​org/​10.​1109/​ACCESS.​2020.​29882​52.
Yen, A. M. N. L. (2018). The influence of self-regulation processes on metacognition in a virtual learning environment.
Educational Studies, 46(1), 1–17. https://​doi.​org/​10.​1080/​03055​698.​2018.​15166​28.
Yilmaz, R. M., & Baydas, O. (2017). An examination of undergraduates’ metacognitive strategies in pre-class asynchronous
activity in a flipped classroom. Educational Technology Research and Development, 65(6), 1547–1567. https://​doi.​org/​
10.​1007/​s11423-​017-​9534-1.
Yin, J., Goh, T. T., Yang, B., & Xiaobin, Y. (2021). Conversation technology with micro-learning: The impact of chatbot-based
learning on students’ learning motivation and performance. Journal of Educational Computing Research, 59(1),
154–177. https://​doi.​org/​10.​1177/​07356​33120​952067.

Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

View publication stats