applied

sciences
Article
The Innovative Use of Intelligent Chatbot for Sustainable
Health Education Admission Process: Learnt Lessons and
Good Practices
Sorin Claudiu Man 1,† , Oliviu Matei 2,† , Tudor Faragau 3,† , Laura Andreica 4, *,† and Dinu Daraba 5,†

1 Mother and Child Department, University of Medicine and Pharmacy “Iuliu Hat, ieganu”,
400347 Cluj-Napoca, Romania
2 Electric, Electronic and Computer Engineering Department, Technical University of Cluj-Napoca,
400114 Cluj-Napoca, Romania
3 European Projects Department, HOLISUN, 430397 Baia Mare, Romania
4 Research and Development Department, HOLISUN, 430397 Baia Mare, Romania
5 Engineering and Technology Management Department, Technical University of Cluj-Napoca,
400114 Cluj-Napoca, Romania
* Correspondence: [email protected]
† These authors contributed equally to this work.

Abstract: This article presents the methodology of creation of an innovative used by intelligent
chatbots which support the admission process in universities. The lifecycle of the ontology, unlike the
classical lifecycles, has six stages: conceptualization, formalization, development, testing, production
and maintenance. This leads to sustainability of the chatbot, called Ana, which has been implemented
at the “Iuliu Hatieganu” University of Medicine and Pharmacy from Cluj-Napoca during the admis-
sion session throughout July–September 2022, for international candidates. The sustainability of the
chatbot comes from the continuous maintenance and updates of the ontology, based on candidates’
interraction with the system and updates of the admission procedures. Over time, the chatbot is able
to answer the questions according to the present situation of the admission and the real needs of the
candidates. Ana had a huge impact, succeeding to resolve a number of 5173 applicants requests, and
only 809 messages was transferred to the human operators, statistics which show a high cost-benefit
Citation: Man, S.C.; Matei, O.;
improvement in terms of reducing the travel expenses for the candidates and also for the university.
Faragau, T.; Andreica, L.; Daraba, D.
The article also summarizes the good practices in developing and use of such an intelligent chatbot.
The Innovative Use of Intelligent
Chatbot for Sustainable Health
Education Admission Process. Learnt
Keywords: intelligent; chatbot; innovation; ontology; methodology; university; admission; sustainability
Lessons and Good Practices. Appl.
Sci. 2023, 13, 2415. https://doi.org/
10.3390/app13042415
1. Introduction
Academic Editor: Xianpeng Wang
In the yearly university admission period, universities are overwhelmed by the huge
Received: 2 December 2022 numbers of candidates visiting the admission departments of each university, with ques-
Revised: 18 January 2023 tions about the university, admission requirements and related documents [1]. This poses a
Accepted: 19 January 2023 challenge for most admission teams, as the staff involved in the admission usually partake
Published: 13 February 2023
in one to one conversations with candidates. This is not a cost effective method of commu-
nication in such situations, and in some cases physically visiting the admission centers,
waiting in queues in order to get answers from the admission team might be seen too costly
and time consuming for candidates in order to received the required answers and consider
Copyright: © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
applying for a specific university.
This article is an open access article
1.1. Background
distributed under the terms and
conditions of the Creative Commons Our study will focus on international candidates and the digitization of the traditional
Attribution (CC BY) license (https:// international university admission process, which provides a specific set of challenges
creativecommons.org/licenses/by/ to candidates, such as sending the application form and admission documents through
4.0/). national and international postal services to the admission center, which can be costly,

Appl. Sci. 2023, 13, 2415. https://doi.org/10.3390/app13042415 https://www.mdpi.com/journal/applsci

Appl. Sci. 2023, 13, 2415 2 of 24

stressful, and in some cases bear the risk of losing the admission documents through
postal services, or the risk that candidates cannot timely sent the necessary or additional
documents required by the admission staff in the imposed deadline, due to slow postal
operators, strikes, customs, etc.
Moreover communication efforts in the pre-admission and admission period in univer-
sities are usually conducted traditionally, which requires the availability of a large number
of university administrative staff in order to communicate on an individual basis, with a
large number of candidates in a short period of time [2]. Communication methods for uni-
versity admissions has evolved from personal physical discussion between candidates and
the admission academic staff, to emails and call centers, as competition between universities
pushes them to accept multiple forms of communication, in order to attract candidates.
Simultaneously with the increased competition between universities in order to at-
tract candidates, seen with the acceptance of multiple forms of communication, the recent
COVID-19 pandemic has accelerated the worldwide transition from the traditional physical
admission to the modern web based online admission systems [3], such as Socrates, devel-
oped by Holisun (http://www.holisun.com/en/, accessed on 15 January 2023), in order to
respond to the social distance rules imposed by the pandemic. As in the admission period
of 2019–2020 most of the university admissions were online, an overwhelming number of
candidates started using online forms of communication with the admission staff, those
online forms of communication provides a large database with the questions addressed to
the admission staff, most times those questions being repetitive ones.
Even if most universities are using Q&A public documents addressing the most fre-
quently asked questions, the majority of candidates still prefer using personal forms of
communication, such as writing emails to the administrative staff in order to get answers
to their inquiries, as the majority of candidates seem to not be interested in reading Q&A
documents. This creates a problem for universities across the world, as answering those
repetitive questions by the academic staff requires significant labor hours and the avail-
ability of a large admission staff, but now is possible to digitize the most frequent Q&A,
associated with the admission process, by defining those questions and answers as a natural
language processing problem [4], that can be used as a system of dialogues between candidates
and a Q&A database through an automated chat box, used primarily to provide an interface
of communication between candidates and the Q&A database, this technology is most
widely and commonly known as “chatbot” [5].

1.2. Aim of the Research

This paper looks into the way chatbots are starting to be used in a multicultural
and bilingual environment by Romanian universities, the first one being the University of
Medicine and Pharmacy “Iuliu Hatieganu” from Cluj-Napoca (UMF Cluj). This leads to cost
reduction and high sustainability in terms of resources allocated to the whole admission
process (human resources and time).
The aim of the paper is to explore how Ana, as the chatbot is called, can innovate the
university admission support process by answering questions from international candidates
that otherwise would ask those questions at the admission office or by other means of one
to one communication methods that entails large costs in terms of labor hours from the
admission staff. The chatbot’s name Ana was chosen by the UMF Cluj Admission Staff
following the well established practice of giving popular and internationally recognized
human names for chatbots in order to make their interaction with humans feel more natural,
as a human to human conversation does.

1.3. Research Questions

The research questions the article answers are:
• RQ1. What is an appropriate structure of the ontology of the chatbot?;
• RQ2. How and to what extent a chatbot can contribute to the candidate support during the
admission period?;
Appl. Sci. 2023, 13, 2415 3 of 24

• RQ3. What are good practices in desinging such a chatbot?;

• RQ4. What is the innovation and added value of such a chatbot to the admission process?;
• RQ5. What is the acceptance of such a chatbot and what is its (perceived) impact?;
• RQ6. What actions are needed for making the chatbot sustainable over more admission
sessions?

1.4. Novel Contribution

As an element of novelty, the Ana chatbot was designed and developed to be used in
English and French conversations with international candidates applying for the University
of Medicine and Pharmacy “Iuliu Hatieganu” from Cluj-Napoca, one of the largest medical
universities in Romania. Ana, developed initially for UMF Cluj, is the first chatbot to be
used in Romanian universities for admission purposes and the chosen chatbot name does
not relate to other chatbots on the market that might use the same popular human name
of “Ana”.
As one of the most documented challenges of any institution interested in maximizing
chatbots acceptance by the end user is developing social intelligence features [6] through
simulating humanlike communication characteristics in order to make the interaction feel
more natural [7], the challenge and novelty in our study, if compared with the works
of [8] is not only developing bilingual chatbots and adapting their database to the youth
social language [9] but to see if chatbots can be effectively used is a highly multicultural
environment, which implies overcoming communication issues associated with 3332 in-
ternational candidates from UMF Cluj who applied from 76 different countries and 5
continents (Europe, Africa, Asia, North America, South America).
In this sense our study will also provide an insight into how a dedicated university
chatbot performs not only in a highly multicultural environment, but also on how it
performs on candidates with a human science profile.

1.5. Organization of the Article

This paper is organized as follows: Section 1 Introduction introduces the main aspects
related to developing and using chatbots in higher education, with focus on the interna-
tional university admission process. Section 2 Related Work presents other chatbots and
ontologies used for admissions at universities. Section 3.1 Context provides the context in
which the Ana chatbot has been developed, namely for the admission process of “Iuliu Hat-
ieganu” University of Medicine and Pharmacy, Cluj Napoca, Romania. Further, Section 3
Methodology presents the methodology for developing the ontology of Ana. Section 3.3
Software Architecture describes the Software Architecture of the Chatbot. Section 3.4 Im-
proved lifecycle of the chatbot refers to all the lifecycle steps improved within the chatbot.
The results of using Ana are reported in Section 4 Results and challenges related to the
chatbot lifecycle. Then we draw some good practices and recommendations in Section 5
Recommendation, and finally the conclusions and possible future developments are stated
in Section 6 Conclusions.

2. Related Work
The general idea of using artificial intelligence (A.I.) techniques in order to simulate
human conversation can be traced back to 1950’s, when the famous Alan Turing, best
known for its Cryptanalyst merits during World War II [10], developed the famous Turing
Test [11], in which we can determine if the entity with whom we are having a conversation
is a real personal or a computer program.

2.1. First Intelligent Chatbots

The first chatbot called ELIZA [12] was developed in 1960’s, and used keyword
matching algorithms in order to match user input, but at this time the technology was in its
early stages, without any significant practical uses in higher education.
Appl. Sci. 2023, 13, 2415 4 of 24

With the arrival of the graphical user interfaces and the development of natural language
algorithms, a new generation of more complex chatbots were developed, with applications
in the public, private and higher education areas, such as MegaHall [13], CONVERSE [14],
ELIZABETH and HEXBOT [15], ALICE [16] and Siri [17], developed by the Apple Corpo-
ration, that alongside Alexa from Amazon, Cortana from Microsoft and Google Assistant
from Google are using advanced software, data mining and machine learning to create the
most popular and advanced speech conversation chatbots in the world. The chatbot market
is rapidly developing and is predicted to reach 1,23 billion dollars by 2025, according
to [18].
In order to reduce labor costs, chatbots using natural language processing and machine
learning algorithms are developed in order to assure a chat like interface for common
questions addressed by candidates and are designed to provide answers similarly to how
humans do, but automatically and on a much larger scale, without any human intervention.

2.2. Chatbots in Higher Education

Chatbots in higher education vary in complexity, communication between a candidate
and the chatbot can be in text, audio, pictures and video formats. Chatbots can be installed
on most online communication channels, such as the most well known messaging and
social media platforms, but can also function on websites and various software applications.
Chatbots are becoming increasingly popular not only in the online admission process,
where according to MainStay, formerly known as AdmitHub [19] can generally answer
more than 65% of candidate and student questions instantly, 24/7. Advanced chatbots
can also be used by universities in order to reduce summer melt [20], increasing access to
financial aid and improving student engagement.
Chatbots are developed by large and smaller universities alike, such as Georgia State
University (USA) that has one of the most advanced university chatbots in the world [20],
and other universities which are using less advanced chatbots such as Staffordshire Univer-
sity (UK) [21], Telkom University (Indonesia) [5] and even Jamhuriya University (Soma-
lia) [22], from one of the poorest countries from the world.

2.3. Conversational Chatbots

Yet, back in 2011 Al-Zubaide and Issa [23] developed OntBot, a chatbot which uses
appropriate mapping technique to transform ontologies and knowledge into relational
database and then use that knowledge to drive its chats. Vegesna et al. [24] propose a
chatbot that handles queries from users for an E-commerce website. The Ontology template
is developed using Protégé which stores the knowledge acquired from the website APIs.
Nazir et al. [23] presented a novel incremental approach for building a chatbot for
fashion brands based on semantic web, by organizing a dataset of 5000 question and
answers of top10 brands in the fashion domain, which covers the information about new
arrivals, sales, packages, discounts, exchange/return policies, etc. Ranoliya et al. [25]
describes the design of a chatbot, which provides an efficient and accurate answer for any
query based on the dataset of FAQs using Artificial Intelligence Markup Language (AIML)
and Latent Semantic Analysis (LSA).
Agus Santoso et al. [26] propose the development of Chatbot, called Dinus Intelligent
Assistance (DINA), which acts as a conversation agent that can play a role of as student
candidate service. The source of the knowledge base is taken from Universitas Dian
Nuswantoro (UDINUS) guest book. It contains of questions and answers about UDINUS
admission services. Testing of this system is done by entering questions. From 166 intents,
the author tested it using ten random sample questions. Among them, it got eight tested
questions answered correctly.
Appl. Sci. 2023, 13, 2415 5 of 24

Colace et al. [27] presents the realization of a chatbot prototype in the educational
domain, which was developed a system to provide support to university students on some
courses. The initial purpose has focused on the design of the specific architecture, model to
manage communication and furnish the right answers to the student. Dimitriadis shows
in [28] that Pounce Chatbot used within Georgia University increased the enrolment with
3.9% and reduced the summer melt of the students with 21.4%.

3. Research and Development Methodology

3.1. Context
“Iuliu Hatieganu” University of Medicine and Pharmacy (UMF) is the continuation of
the Romanian Faculty of Medicine founded in 1919, as part of the University of “Upper
Dacia”. The “Iuliu Hat, ieganu” University has over 6000 students, 2400 resident and over
1100 teachers and researchers divided between its 3 faculties: Medicine, Dental Medicine
and Pharmacy.
As 600 of the total 1266 admission vacancies, or 47% are allocated to international
candidates, there is a business continuity need to assure that the number of interna-
tional candidates does not drop, as 76% of the total annual tuition fees collected by
the university for students in the first year of study, meaning 3,600,000 euros out of
the total of 4,697,065 euros collected were from international students. The need to
assure better online and interpersonal forms of communication with international can-
didates first emerged during the 2020 international admission, when due to interna-
tional travel restrictions and social distancing heath regulations, UMF Cluj implemented
Socrates (https://holisun.com/produse/socrate-admitere-universitati, accessed on 15 Jan-
uary 2023), an online admission platform, in order to assure the continuation of the interna-
tional admission session in the COVID-19 pandemic period of 2020.
The Ana chatbot implemented in the 2022 international admission of UMF Cluj is
the first chatbot implemented by a Romanian university for admission purposes and
is based on the AIDA.AI natural language algorithms and machine learning methods
developed by Holisun, with the purpose of creating and training the Ana chatbot using
data consisting of the most frequent Q&A’s pairs, with questions and answers that have a
tree-type complexities, with multiple branches of answers, depending on the context of
the discussion carried out by the candidates with the Ana chatbot, developed to be used in
English and French chat conversations.

3.2. The Admission Process

The whole admission process has four steps, as depicted in Figure 1. The first step of
the whole process is the one in which prospects (visitors of the admission platform) can
apply for educational offers of the universities. In the second phase, the applications are
evaluated based on certain assessment criteria. After the evaluation phase, the candidates
are ranked. The admitted ones have to register for becoming students.

Figure 1. University admission process.

Appl. Sci. 2023, 13, 2415 6 of 24

Ana is meant to be used only for the university application stage (see Figure 1), as
that is the step where candidates have the most significant number of questions and need
the most support and tutoring.

3.3. Software Architecture

The software architecture of Ana chatbot, detailed in Figure 2, comprises of several
modules:
• Domain Abstract Ontology: Is the schema of the general ontology with domain
independent concepts, able to be instantiated for each particular authority. This
ontology refers to public information, such as deadlines, student opportunities and
educational offer.
• Specific Ontology: represents concepts which belong or are particular to a specific
candidate, be it country origin, last courses taken. The ontology is structured in three
sub-ontologies, namely:
– Public information: refers to information for public use, such as number of available
seats, specialisation etc.
– Adaptive information is still general information, but provided to the candidates
based on their profile, e.g., different countries issue different documents related to
graduation (Abitur, Bacalaureate, GMAT etc.), depending on bilateral agreements,
foreigners have access to education based on various fees. This adaptive feature
is highly recommended by inclusive communication guidelines [29].
– Personal information is closely related to the candidate and contains a significant
amount of personal data which falls under GDPR regulations: data related to
birth and identity, grades, medical certificates and educational options. Therefore
strict access policies will be in place, both technically as well as administratively.
• AI Engine: includes the algorithms used in processing the knowledge acquisition,
and modeling correct answers for the candidates. This module makes the connection
between the two parties of the conversation, respectively feeds the candidate with
very specific, personalized, and accurate information from the ontology. Moreover,
the information is explained, so that the candidate has no doubt about the accuracy
and conformity of the knowledge which is provided. This module is also able to learn
based on user interaction and feedback so that the chatbot improves its knowledge,
responsiveness, and usability with each conversation.
• Knowledge Acquisition: collects information from different sources such databases,
cloud, documents, other software, and local data.
Of course, the Ana chatbot security is paramount as any university must protect the
privacy of the candidates, respectively the personal data and preferences. That has to be
done not only for GDPR compliance but also because people need to trust the Ana chatbot
and the University of Medicine and Pharmacy "Iuliu Hatieganu" as well. The security
is structured on three levels: (1) data security which regards the safety and reliability of
the data, database and data flow; (2) ontology security which focuses on the consistency
accuracy and quality of the ontology; (3) third party security which assures a proper, correct
and responsible integration of the Ana chatbot with other applications.
Appl. Sci. 2023, 13, 2415 7 of 24

Figure 2. Ana Chatbot Architecture.

3.4. Improved Lifecycle of the Chatbot

In this section we detail and expand the lifecycle briefly described in Section 3 along
with the results for each stage.
The chatbot ontology lifecycle follows closely the classical ontological development
lifecycle [30]. However, we include two more steps, namely deployment into production
and maintenance because the chatbots are meant to be intensively used in production
environments and have a life of their own. The six steps of the chatbot lifecycle are (see
also Figure 3):
• conceptualization aims at identifying and defining the objects, concepts and their
relationships referred by the ontology. It produces an agreed upon meaning for a
concept for the purposes of research.
• formalization tries to bring the previously identified concepts to a canonical form,
eventually structured on several abstraction levels.
• development implies the creation of the ontology using the formal concepts.
• testing assures that the ontology is sound and robust. The step is mainly technical,
focused on the quantity of knowledge, rather than on its quality and is performed by
ontology experts.
• deployment into production is the phase in which the chatbot interacts with the
prospects and provides them with the expected, pertinent and correct answers.
• maintenance is required because a chatbot, like any other software can be considered
a live agent which needs to be continuously updated as the information regarding
admission process changes or the process itself evolves.

Figure 3. The improved chatbot lifecycle.

However, the chatbot has to be very adaptive, therefore the lifecycle is agile [31], with
continuous updates based on the immediate needs of the users, rather than frozen in a
rigid structure, such as waterfall [32].
Appl. Sci. 2023, 13, 2415 8 of 24

3.4.1. Conceptualization
The conceptualization starts before developing the ontology of the chatbot, based
on previous questions of the users. They are to be collated and then canonicalized, as
there are many variations of the same question or the answers are slightly similar. And,
finally, the statements are to be very personalized, appealing to the user and stated from
her perspective.
The corpus of questions comes partially from the questions raised by the prospects
in the previous admission session and partially from the clerks managing the process. All
the specifications are structured in a table manner, as shown in Tables 1 and 2. For each
question (hop, node), we defined:
• node ID, as xxx.yyy.zzz, where xxx means a main node, xxx.yyy means a sub-node,
xxx.yyy.zzz means a sub-sub-bode and so on. This is displayed in column ID.
• the content (information, action), shown in column Content.
• keywords used for accessing the respective info when the user types in her question.
These are listed in column Keywords. The keywords concept is designed to admit
possible typing errors like skipping a letter, or inversion of two letters. For this
situation we used the * symbol which allows the keyword to have as many errors as
we indicate in the beginning.
• related nodes, represented by their ID’s in column Related nodes.
Table 1. The questions known by the chatbot (I).

ID Content Keywords Related Nodes

Welcome to the International Students
100
Department
200 My name is Ana, how can I help you?
300 Please agree with GDPR policy gdpr
300.100 If you did not agree with GDPR policy
400 Main menu start
400.100 Connect to an operator operator
400.100.100 Operator available
400.100.200 Operator unavailable working, hours, work, hour
400.200 Admission platform
400.200.100 Video presentation admission*, video*, tutorial* 600
400.200.200 How many pages can I upload? pdf, page, pages 400.200.100, 600
400.200.300 What does “pending payment” mean? Pending, Payment*, Pay 400.200.100, 600
400.300 Clarification regarding the application file
400.300.100 Multiple application files file*, option*, application* 400.400.300, 600
Documents issued in a non-European non-EU, non-UE, non-European*,
400.300.200 400.300.100, 600
country Hague, Apostille
400.300.300 The medical certificate Medical, Specialist*, Healt* 400.300.400, 600
400.300.400 Psychological examination Psychological 400.300.300, 600
400.300.500 Expired passport Passport, pasport, Valid, Expir* 600
Admission, education, taxes and
400.400
prerequisites
400.400.100 What is your question related to language Lang*
400.400.100.100 In what language are the courses taught cours*, Corses, Corse 400.400.100, 600
400.400.100.300,
How can applicants demonstrate their
400.400.100.200 proficienc*, certif*, prficiency 400.400.100.400,
language proficiency?
600
Appl. Sci. 2023, 13, 2415 9 of 24

Table 2. The questions known by the chatbot (II).

ID Content Keywords Related Nodes

If I studied in English/French, do I still
400.400.100.300 600
need a Language Proficiency Certificate?
Can I pass a language proficiency
400.400.100.400 examination at your university, before exam*, exmination 600
applying?
place*, seat seats, medicin*, pharm*,
400.400.200 How many places are available? farma*, dental, dent, dentistry, 400.400.300, 600
dentist
What are the tuition fees and costs of the
400.400.300 tuition*, fee, fees, cost, costs 600
application process?
method*, metodology, ethodology,
400.400.400 How can I apply? 400.200.100, 600
apply, appl, aply
What is the deadline for submitting the deadline, date, dates, dead, Result*,
400.400.500 600
application files and when are the results? reslts
If I was not admitted in the Early
400.400.600 Early, erly 400.400.500, 600
Admission, is my file still in competition?
Unfortunately, I do not understand your
500 400
question.
Close the chat. Go to the main options.
600 thank* 400, 400.100
Connect to an operator

3.4.2. Formalisation
We want the candidates to have the best user experience while using Ana chatbot. This
means a feeling of interacting with a human and optimal access to knowledge, which is
achieved in four ways. Firstly, the chatbot is called Ana, name ranked the 16th in the top
of the most popular human names [33]. The name is feminine as there is a gender bias
expectation in such applications, as reported by Feine et al. in [34] and McDonnell and
Baxter in [35]. Secondly, once in a while, the chatbot simulates the typing delay of a human
operator, which means that the chatbot waits a couple of seconds before prompting its
reply. Thirdly, for a complete user experience the chatbot interacts in several modes such
as buttons to be selected by the user and free text. And fourthly the ontology is structured
in a tree manner with optimal access to knowledge. The tree is built as suggested by
Singer et al. in [36]. For each level of the tree, the information gain is computed according
to the formula:
IG (S, a) = H (S) − H (S| a) (1)
where IG (S, a) is the information for the dataset S for the variable a for a random variable,
H (S) is the entropy for the dataset before any change (described above) and H (S| a) is the
conditional entropy for the dataset given the variable a. In other words, the information
gain is the difference in entropy of the information of the whole ontology minus the entropy
of the sub-ontologies split by the attribute a [37].
The entropy is defined as [38]:
n
H (S) = − ∑ P(si )logP(si ) (2)
i =1

where H (S) is the information entropy of the set S = s1 , s2 , . . . , sn . log is the logarithm, the
choice of base varying between different applications. Base 2 gives the unit of bits, while
base e gives the “natural units” nat, and base 10 gives a unit called “dits”.
The attribute with the highest discrimination power (which induces the largest infor-
mation gain) is set as node [36].
Appl. Sci. 2023, 13, 2415 10 of 24

For ethical and legal reasons, however the first replies of the chatbot are self-introductory
and presenting the terms and conditions of use.

3.4.3. Development
The structure of the chatbot ontology is depicted in Figure 4. The semantics of the
nodes are collated in Tables 1 and 2. There are 6n main nodes corresponding to the
main steps a conversation could have. From here on the nodes are represented by their
ID’s, as displaying the actual content would make the presentation, explanations and
graphs cumbersome.

Figure 4. The semantic diagram of the chatbot ontology.

The first three nodes (100, 200, 300) contain introductory sentences such as Welcome to
the International Students Department. My name is Ana, your virtual assistant. The transition
between these nodes (100, 200, 300) is automatic and the links are marked with the purple
arrow. The flow between 300 and 400 is conditioned by the acceptance of GDPR policy by
the user, otherwise the chat closed.
The node 400 represents the main menu containing the main options. From statement
100 to 400.*.*.*, the ontology is formalised as a tree (see Section 3.4.2). All the leaves of this
tree (400.*.*.*) should provide answers to the user, but we also take into account that the
answer is not the proper one. Therefore from those leaves, the user is provided with three
options, marked as node 600:
• to go to the main menu (400), by typing start;
• to contact an operator (400.100), by typing operator;
• to close the chat, by typing thank you.
If a text input by the user is not understood (does not match sufficiently any of
the ontological nodes), the bot outputs the text Unfortunately, I do not understand your
question and sends an email to the knowledge expert, so that she can improve the ontology
continuously based on real needs of the candidates.

3.4.4. Testing
The trust in chatbots is very important from a social perspective [39], therefore the
tests were extensive and addressed the ontology as well as the business logic. The stage of
the lifecycle has been performed by a very competent team consisting of all people involved
in admission process over the years.
Appl. Sci. 2023, 13, 2415 11 of 24

As the chatbot is to be used by people with various cultural and educational back-
ground, a special attention was given to the structure of the ontology and the way it is
displayed, therefore we performed five sets of usability tests, aiming at:
• The users’ perception of human-like chatbot;
• The proper way to interact with a user;
• The depth of the ontological tree;
• The optimal number of available options;
• The number of interconnections between; ontological nodes;
The usability tests have been performed on 28 people (admission staff and students)
using the Cognitive Walkthrough Method [40], which is an analytic inspection used to
evaluate prototypes from the user’s perspective. The testers take the role of the user
and “walk through” the process of using the product. The analysis uses storyboards and
expert evaluation is based on the information obtained in the conceptualization phase (see
Section 3.4.1). All 28 people involved in testing have either social or medical background,
so none of them has strong IT-related skills, other than the usual ones.
The usability related to a specific aspect of the chatbot was measured by asking the
subjects to rate it on a scale from 1 to 100. The charts display the average perceived usability
on the vertical axis, where 0 means no usability and 100% means full ergonomy with respect
to that specific aspect.
The information retrieval time was determined based on the logs of the chatbot and
was measured as the total interaction time of the user with the chatbot divided by the
number of information items searched and found by the users, based on their declaration.

The Users’ Perception of Human-like Chatbot

Figure 5 describes the impact which the interaction with the chatbot is having on the
human perception, based on the rhythm of typing the automatic answers. Therefore as
the chart reflects, if the answer came in less than 1 s, the perception of the conversation
did not reach the desired objective, to be considered as human as possible. And also a
typing response over 5 s, as long the conversations are practical, without having descriptive
predefined answers. The vertical axis displays the percentage of users that considered the
reaction of the chatbot as being the one of a human.

Figure 5. The users’ perception of human-like chatbot.

Modes of Interaction of the User with the Chatbot

The user can input information:
• by typing in the question;
• by choosing an option from a list of predefined ones;
• mixed—typing or/and choosing one of the predefined options provided by the chatbot.
The perceived interaction usability for each of the three scenarios is depicted in Figure 6.
As expected, a mixed interaction is preferred to simple clicking and is way better than the
Appl. Sci. 2023, 13, 2415 12 of 24

old-school typing-in. The usability was graded on a scale from 1 to 100 by the users and
Figure 6 displays the average grades for each possible type of interaction.

Figure 6. Perceived Interaction Usability.

Depth of the Ontological Tree

The ontology is structured mainly as a tree (called ontological tree), as explained in
Section 3.4.3. Definitely, there are cross-relationships between various nodes, which is the
reason why the Figure 4 displays a graph, in which the ontological tree is depicted by gray
arrows. However, the information is served to the user mainly in a tree-based manner.
Test and experiments aim at determining the optimal depth of the ontological tree, so
that users feel comfortable of using it and find all the needed information in the shortest
time. Therefore we asked the users to grade the usability of the platform and measured the
time needed to reach the desired information, while varying the depth of the ontological
tree from three to ten.
Figure 7 depicts the correlation between the depth of the ontology and the usability,
respectively the correlation between the depth of the ontology and the average retrieval
time. The X axis represents the depth of the ontology (from 3 to 10). The Y axes represent
the usability of the chatbot as perceived by the user (image on the left), respectively the
relative average time (in seconds) needed to retrieve the information (image on the right).

Figure 7. The depth of the ontological tree.

The usability ranges from 56% corresponding to a depth of 3, to 86% corresponding to

a depth of 7. That means that the level 6–7 is the optimal depth to which an ontology is
recommended to be developed. The average retrieval time behaves in the opposite way,
which means that a too lower depth of the ontology determine high levels of retrieval, also
a greater depth will lead to the same effect, due to the fact that information become more
and more precise and specific, and also the quantity of knowledge is very large.

Optimal Number of Available Options

Figure 8 depicts the correlation between the number of available options from which
a user can choose and the usability percentage. The X axis represents the depth of the
Appl. Sci. 2023, 13, 2415 13 of 24

ontology (from 3 to 8). The Y axes represent the usability of the chatbot as perceived by the
user (image on the left), respectively the relative average time (in seconds) needed to retrieve
the information (image on the right). The usability is having a higher percentage when
the options to be chosen are fewer, which indicates that the accessibility to information
reaches out to all of the four principles from a UX perspective: perceivable, operable,
understandable, and robust [41].

Figure 8. Optimal number of available options.

The optimal number of available options should not exceed six, due to UX principles
and the dimension of the screen utilized, because the visibility on the chatbot popup is
limited, and this can lead to leakage of important information. The value of 6 options
is a trade-off between the perceived usability (left chart, where usability is measured in
percentages) and the information retrieval time (right chart, where the time is measured
in seconds).

Number of Interconnections between Ontological Nodes

The interconnections between the ontological nodes are depicted in Figure 4 as green
arrows. The optimal number of interconnections in this implementation is correlated to the
number found by Yadav et al. in [42]. We started at a number of 32 nodes and the depth of
the tree was 4. The correlated nodes from almost every leaf, increased the number of total
backlinks to 32, meaning an average number of backlinks of 1.52 for each leaf.
It is important to define relations between nodes or topics, which may be needed
together for a complete understanding by the user or which are complementary. It is hard
to define key phrases to differentiate the specific information, from different persons, who
in most cases don’t know what they are searching for, or often happens that the way they
ask a question is unclear. To help and improve the conversation and the area of topics
discussed, the chatbot offer hints in relevant and connected area of discussion, which may
be helpful for the user to find the path to the specific information needed. This way, the
chatbot will avoid having a circular conversation, which may lead to the same repetitive
questions and answers. The diverse pool of questions will cover all the relevant information
needed by user, from a specific topic.

3.4.5. Deployment into Production

As mentioned in Section 3, Ana was used during the application stage of the online
admission process. It was trained initially on the questions in Tables 1 and 2.
As shown in the software architecture depicted in Figure 2, the chatbot is able to
integrate and collect information from third-party sources. For easier maintenance, we
integrated the chatbot with:
• Admission platform (https://admissions.umfcluj.ro/frontend/auth/login, accessed
on 15 January 2023);
• The database with specializations and number of available seats;
• The authentication details and granted rights of the admission committee.
To achieve openness and trust, by respecting the FAIR principles (findability, acces-
sibility, interoperability and reusability), as well as GDPR regulation, imposed by the
academic sector, the Ana chatbot obeys GDPR guidelines and ethical implications of han-
Appl. Sci. 2023, 13, 2415 14 of 24

dling sensitive data. On top of security measures such as authorization, authentication, and
end-to-end encryption of communication channels, the following measures will be taken to
preserve the candidate’s anonymity and are considered compliant with the privacy and
security guidelines. Therefore the chatbot, but especially its integration with third-party
data sources has been monitored against the top 10 OWASP security threats [43].

3.4.6. Maintenance
The chatbot is dynamic and is to be adapted to the context in which it operates and
provides answers. Therefore its maintenance is crucial, more important than in most other
cases of intelligent software systems [44]. Moreover, the testing phase was not exhaustive,
therefore prolonged somehow into the production stage, hence the maintenance has also a
second role, to fix possible inadvertence.
The maintenance of the chatbot requires a knowledge expert. The continuous develop-
ment and maintenance of the ontology relies on:
• each stage of the admission process brings up new required questions and needs and
leaves out caducous information;
• each question typed by the user and unanswered by the chatbot might be a new fact
to be asserted to the ontology;
• other questions coming by other means (e.g., email, phone) are to be formalized and
inserted into the ontological tree.
The feedback was requested at the end of each chat as a likable or dislikeable experi-
ence, as this is a classical way of collecting feedback from users online, as seen in Figure 9.
Dislikes numbers decrease over time, while the number of likes increases at the same time.
There is a little spike in dislikes in week 6 when the number of candidates was the highest,
and human operators were not able to answer all the candidate’s questions received over
the phone, email, and WhatsApp.

Figure 9. Like to dislike ratio over the whole admission period.

A significant amount of maintenance was allocated for adjusting the chatbot to the
needs and questions of candidates during the admission sessions. The obsolescence of the
information, such as admission deadlines, fees, places available for registration, etc., has
been taken into account, due to the fact that it needs to be updated every year based on
the admission methodology of the university, as well as in accordance with the decisions
of the Ministry of Education. The number of available seats changes from one admission
session to another, also there are questions that do not apply during the admission and are
not relevant, but also there are questions that have the same answer throughout the time.
Appl. Sci. 2023, 13, 2415 15 of 24

4. Results and Challenges Related to the Chatbot Lifecycle

4.1. Results of the Ana Chatbot
The ontology started with 32 nodes, out of which 7 first-level nodes and 21 leaves. The
depth of the tree was 4. In total, there are 57 links to other nodes, averaging 1.84 for each
node. The number of backlinks from leaves is 32, meaning an average of 1.52 backlinks
from each leaf. The Ana chatbot was implemented in the 66 day out of 84 days of the
international online admission session held by UMF Cluj between 4 May and 26 July 2022.
According to Table 3, in the 18 days, between 9 and 26 of July in which Ana worked
alongside the human support staff, we had seen a total of 1342 chats being opened by
candidates, on average with 75 chats per day, the most active day of the week being Monday
with 105 opened chats on average, and the last active day of the week being Sunday with
only 39 chats opened on average (See Figure 10).

Table 3. Total daily chats results between 9 and 26 of July.

Candidates Responses
Days Total
Message to From
Mess.
Total
Chats from Bot Operator Bot Operator
mess.
candid.
Monday*(3) 314 1543 5569 1347 196 3928 98
Tuesday*(2) 161 780 2823 649 131 1946 97
Wednesday*(2) 189 827 3141 708 119 2125 189
Thursday*(2) 172 877 3093 653 224 2082 134
Friday*(3) 241 939 3721 818 121 2698 84
Saturday*(3) 149 565 2312 562 3 1744 3
Sunday*(3) 116 451 1782 436 15 1320 11
Total 1342 5982 22,441 5173 809 15,843 616
Average 192 855 3206 739 116 2263 88
Avg. per
75 332 1247 287 45 880 34
day

Figure 10. Average weekly opened chats.

The 75 average number of daily opened chats, or the total of 1342 open chats in the
18 day period corresponds to 40.27% of the number of international candidates (without
Romanian citizenship) who submitted their online admission at UMF Cluj, out of which
78.51% (2616) were from Europe, 13.09% (436) from Africa, 5.58% (194) from Asia, 2.49%
(83) from North America and only 0.09% (3) from South America, as seen in Figure 11, and
more detailed in Table 4.
Appl. Sci. 2023, 13, 2415 16 of 24

Table 4. International Candidates during the 2022 UMF Cluj International Admission.
No. Country % No. Country % No. Country %
1 France 61.40% 27 Cyprus 0.24% 53 Ukraine 0.06%
2 Germany 7.26% 28 Lebanon 0.21% 54 Georgia 0.06%
3 Morocco 6.66% 29 Pakistan 0.18% 55 Island 0.06%
4 Tunisia 3.87% 30 Austria 0.18% 56 Somalia 0.03%
5 Italy 1.74% 31 Iran 0.15% 57 Sudan 0.03%
6 Israel 1.65% 32 Luxembourg 0.15% 58 Benin 0.03%
7 U.A.E. 1.50% 33 Afghanistan 0.12% 59 Burkina Faso 0.03%
8 Switzerland 1.44% 34 Cameroon 0.12% 60 Cambodia 0.03%
9 Belgium 1.35% 35 Egypt 0.12% 61 Kenya 0.03%
10 Sweden 1.20% 36 Ghana 0.12% 62 Mauritania 0.03%
11 Guadeloupe 1.14% 37 Nigeria 0.12% 63 Niger 0.03%
12 Reunion 0.87% 38 Jordan 0.12% 64 Gabon 0.03%
13 Greece 0.78% 39 Kuwait 0.12% 64 Antigua & Barbuda 0.03%
14 Canada 0.69% 40 South Africa 0.09% 66 Haiti 0.03%
15 Saudi Arabia 0.66% 41 Dem. Rep. Congo 0.09% 67 East Timor 0.03%
16 United States 0.54% 42 Madagascar 0.09% 68 South Korea 0.03%
17 Qatar 0.51% 43 Brazil 0.09% 69 Indonesia 0.03%
18 United Kingdom 0.51% 44 Oman 0.09% 70 Malaysia 0.03%
19 Algeria 0.42% 45 Australia 0.09% 71 Poland 0.03%
20 Ireland 0.39% 46 Japan 0.09% 72 Hungary 0.03%
21 Spain 0.30% 47 Netherlands 0.09% 73 Albania 0.03%
22 Finland 0.30% 48 Senegal 0.06% 74 Croatia 0.03%
23 India 0.27% 49 Ethiopia 0.06% 75 Denmark 0.03%
24 Turkey 0.27% 50 Jamaica 0.06% 76 Monaco 0.03%
25 Norway 0.24% 51 Palestine 0.06%
26 Moldova 0.24% 52 Portugal 0.06%

Figure 11. The distribution of the candidates based on their continent of origin.

The usefulness of using Ana Chatbot as a bilingual (English and French) automated
support assistant in a multicultural and international environment (See Figure 11) is clearly
seen at UMF Cluj, as a large portion of international candidates from 76 countries used the
chatbot as their primary support assistant, with a small ratio of “Dislikes” (See Figure 9)
and a small number of redirected chats to human operators (See Figure 12).
Appl. Sci. 2023, 13, 2415 17 of 24

Figure 12. Messages sent by candidates to the bot and operators.

In the analyzed 18 days in which Ana worked at UMF alongside human operators, the
chatbot received a total of 5982 messages from candidates with a total of 16,459 responses
out of which 15,843 (96.3%) were from Ana, as such it can be argued that Ana implementa-
tion proved successful in providing a 24/7 online admission support and assuring that the
most basic and repetitive questions are addressed by the chatbot, and not by admission staff
at UMF Cluj, which received only 809 messages from candidates (13.52%), representing the
more complex questions that Ana couldn’t yet provide answers (See Table 3).
As depicted in Figures 12 and 13 we can see that with the exception of Wednesday,
responses from human operators to candidates inquires are less then 1, meaning that a
large number of questions addressed by candidates to operators are not answered in the
day they are addressed. This is due to the fact that UMF Cluj, like most universities, has
only a Monday–Friday, 8 working hours schedule for answering questions through its
admission staff, moreover as most candidates from the 2022 international admission were
on different time zones than Eastern European Summer Time (EEST), there are also cases
in which time zones between Romania and the candidates country are very different and
thus provides a challenge for personal communication between international university
candidates and the local admission staff.
In this instance, if candidates require the intervention of the academic staff on week-
ends or outside normal 9 AM–5 PM EEST working hours, they are told to send an email or
reconnect between 9 AM–5 PM EEST from Monday to Friday. This lack of time synchroniza-
tions also represents one of the benefits of using Ana instead of relying heavily on human
operators, as Ana not only works 24/7 but also adapts to previous more complex questions
addressed to human operators, as in time its machine learning capabilities are developing
alongside its Q&A database, which translates into a lesser reliance on human operators.

Figure 13. Messages sent by bot and operator to candidates.

Appl. Sci. 2023, 13, 2415 18 of 24

4.2. Sustainability of Ana Chatbot

The sustainability of the chatbot is given by continuous updates of the ontology
according to the present reality of the admission, as things change over time. In some sense,
the chatbot is alive and its ontology needs to be adjusted to the real life situation and that is
achieved in the maintenance stage of the lifecycle.
From the total of 1342 chats opened in the last 18 days of the 2022 UMF International
Admission Session, on average, as seen in Figure 14 each candidate generated 4.33 questions,
out of which 3.83 (86.45%) were addressed to Ana and only 13.54% of the questions were
addressed to human operators. On average we had 12.19 responses to 4.43 questions
addressed in each chat or 2.75 responses for each question. Ana gave 11.74 from all of
the total 12.19 responses per chat (96.30%). Only 3.7% out of the total answers sent to
candidates were from operators.

Figure 14. The number of messages from the candidates, from the bot, from the operator and the
number of chats to the operator and to the bot.

For most of the questions addressed by candidates in an average chat, 86.45% were
addressed to Ana, and only 13.54% of the questions were redirected to human operators.
Out of the total questions addressed to human operators per chat, only 76.66% of them
received a response (see Figure 14), as the admission staff at UMF responsible for chat
responses usually work from 9 AM–5 PM (EEST) and only from Monday to Friday, in this
sense chats opened by candidates that require human operators don’t receive a response
in the weekends and/or outside business working hours. During the admission session,
some users did not get the desired info directly from the chatbot. However, the number of
unanswered questions decreased over time, as depicted in Figure 15.

Figure 15. Number of messages without an answer from the chatbot or operator.

The English version of the ontology contains 945 words structured in 45 nodes as
shown in Section 3.4.3, which means an average of 21 words per message. As shown
by Laufer et al. [45], adults typically type at about 40 words per minute when writing
for enjoyment and 5 words per minute for in-depth essays or articles, with an average
Appl. Sci. 2023, 13, 2415 19 of 24

of 22.5 words per minute. This means that typing a message would require 0.93 min on
average. As summarized in Table 3, the chatbot posted 15,843 messages, meaning 246.45 h
saved in the 17 days between the 9th and the 26th of July, which account for 13.69 h
(equivalent of 1.71 man-days) gained daily in favour of other tasks.

4.3. Challenges Related to the Chatbot Lifecycle

Developing and implementing Ana chatbot was provocative from the beginning. Some
of the challenges that were raised were:
• Implementation issues due to a short deadline to finalize the chatbot;
• Formalisation of the admission concept was a challenge. In respect to the UX principle,
the information must be very clear, easy to access, and easy to be read, in such a way
that the result of the entire process is achieved.
• The multicultural identity of the chatbot raised brainstorming sessions for the team
involved, due to the fact that there were a lot of differences between candidates
countries regarding:
– identity documents;
– educational diplomas: Abitur, Bacalaureate, GMAT etc.;
– language barrier which leaded to miss nominated words;
– cultural habits.
• Developing Ana chatbot was not only for the purpose of researching the domain
of the ontologies, it was a real-life project, implemented and used in 2022 for the
admission process of the University of Medicine and Pharmacy “Iuliu Hatieganu”
from Cluj-Napoca. Therefore, the main focus of the project was not on the research,
but on facilitating communication way between the university staff and candidates all
over the world. The aim of the ontology was to offer specific answers, in real-time, to
candidates’ questions, covering time zone differences.
• The testing period was limited regarding the number and the type of persons involved
in the process.

5. Learnt Lessons and Recommendations

Based on our real experience with UMF Cluj presented in this article, here are the
recommended good practices:
• It is important that the chatbot has a human appearance in terms of name, icon
(avatar), and behavior, meaning that, if the answers were delayed a couple of seconds,
as written by humans, the user’s perception on the chat increased with 4.6%.
• The name of the bot has to be localized with respect to the geographical location
of the users. Simple yet comprehensive names are preferred, based on world-wide
ranks [33,46] and gender stereotypes [39].
• There should be a right mix between the number of predefined options or answers a
user can choose from and the possibility to write open questions, freely. Too many or
exclusive predefined options give the feeling of constriction; on the other hand, only
free text makes the communication somehow exhausting for the user and difficult to
formalize in terms of ontology.
• The maximum depth of the tree should be no more than six or seven (see Sec-
tion “Depth of the Ontological Tree”, value correlated also with the research of Agar-
wal and Wadhwa [13].
• The number of predefined options within on reply should not exceed 6 (see
Section “Optimal Number of Available Options”).
• It is important to define relations between related nodes or topics, which may be
needed together for a complete understanding by the user or which are complementary
(see Section “Number of Interconnections between Ontological Nodes”).
Appl. Sci. 2023, 13, 2415 20 of 24

• For European countries, GDPR compliance is a must and the related conditions should
be served at the very beginning of the chat before the first action is required from
the user.

6. Conclusions
The article presents a chatbot used during the admission of foreign students at UMF
Cluj University in July 2022.
The chatbot, called Ana has run 1342 times and forwarded the user to the operator 216
times, which means coverage of 83.9%. Due to the high amount of administrative work
during the admission session, the knowledge expert succeeded to increased Ana’s ontology
by barely 18.5%. For the next session, the knowledge is expected to increase by at least
120%. Due to business-related limitations, Ana was used only during the last 37.5% of the
entire period. However, a whole admission session could be more conclusive and would
provide deeper insight into what a chatbot is, how it works, and the way it improves the
communication between the candidates and the administrative staff.

6.1. Answers to the Research Questions

Based on the experience with Ana chatbot, more specific research answers have
been drawn.
RQ1. What is an appropriate structure of the ontology of the chatbot?
Firstly, the proper structure of the chatbot ontology covers the main information a
candidate must know when applying to an university: from the faculties and the study
domains, to the taxes applied and in the end to the documents needed to complete the
enrolment process. Ana chatbot, can be scaled up to more universities, and to more
countries, using different language pack and with a minimum effort the responses can be
easily adapted due the fact that the core of the chatbot contains all these general aspects:
• faculties and the study domains
• taxes to be applied
• documents needed to complete the entire process
RQ2. How and to what extent a chatbot can contribute to the candidate support during the
admission period?
Secondly, a chatbot can cover a huge amount of questions based on the candidate’s
needs. It is an innovative support tool which handles properly repetitive questions came
from different areas of the world, that for traditional admission process involved a consid-
erable amount of money to cover all the expenses generated:
• for universities: salaries, indirect expenses, equipment usage;
• for candidates: the travel expenses, indirect expenses.

RQ3. What are good practices in desinging such a chatbot?

The good practices and the recommendations regarding the use of Ana chatbot are
summarized in Section 5.
RQ4. What is the innovation and added value of such a chatbot to the admission process?
The innovation brought is described further in Section 6.2.
RQ5. What is the acceptance of such a chatbot and what is its (perceived) impact?
We live in an era where the technology is all over the domains. The chatbot fits perfect
to the new generation, they are more accustom of using such tools. Based on our research,
the digital native generation has shown good acceptance of Ana.
RQ6. What actions are needed for making the chatbot sustainable over more admission sessions?
Appl. Sci. 2023, 13, 2415 21 of 24

Each admission session offers a huge amount of new information that can be converted
into new nodes in the ontology. This leads to a better performing chatbot, that is more
and more adapted to the needs of the candidates, and that leads to a higher response
rate, that reduce to a minimum all the costs with the admission process. Obviously, the
sustainability can be achieved by continuous maintenance of the chatbot and ontology
and that is why the lifecycle has two extra steps Section 3, unlike the classical lifecycles:
deployment into production and maintenance.

6.2. Novelty of the Ana Chatbot

In an era of digitized systems, Ana succeeded in sustaining the admission process for
the University of Medicine and Pharmacy “Iuliu Hatieganu” from Cluj-Napoca, in July
2022. The implementation of the Ana chatbot brings novelty aspects into the field by:
• developing an ontology that pinpoints the multiculturalism aspects, being the first
chatbot to document the acceptance and results of such technology with candidates
from 76 countries;
• expanding the digitization activities in universities, especially in domains that are not
directly related to the ICT;
• conceptualizing internal processes of a university in an innovative way;
• improving the ontological tree by collecting the data in a dynamic mode, and structur-
ing the information in three different categories: public, personal and adaptive. In that
way, all the candidates, staff, or third parties can co-create to improve the ontological
tree of the chatbot;
• increasing productivity, helping users to obtain timely and efficient assistance or
information. Therefore the time zone constriction it’s not anymore a limitation, the
candidate will obtain the answer in real-time.

6.3. Future Developments of Ana Chatbot

Building an ontology for the admission process storm up a diversity of ideas to be
implemented in the future. One of the most powerful skills nowadays is to know what to
do with data: organize-store, use-analyze, share, reuse-maintain, archive-destroy, create-
capture&collect [47];
A priority for Ana chatbot is the interoperability with other platforms used in the
admission and enrolment process.
From the beginning of the conversation, we want the chatbot to have an open dis-
cussion with the prospects, covering the mentoring area in guiding and leading users to
find out the specialization domains available at the university offers, and what fits them
best. And in the next step the conversation to continue with all the procedural aspects of
the enrolment.
Another future development will be multiplying the chatbot for all the faculties
inside the university and improving its knowledge with specific information regarding
the educational curriculum for each specialization, in that way the candidates may ask a
specific question regarding the domain she wants to study.
Another future goal is to expand the ontological tree with the result from previous
admission sessions and also to have the chatbot active all over the year for offering news,
information about the university etc.
One of the goals of implementing the chatbot in a university was to increase the
number of international students, therefore the accessibility aspects will be improved by
integrating the Ana chatbot with the most popular online messaging services, such as
Whats App and Facebook, in order to make the chatbot proactive and address questions
to candidates in relation to their admission application. Therefore the multicultural level
among students in university will increase.
An important aspect that needs to be considered is the constant improvement of the
security of the data processed among the conversations, applying methodologies such as
the ones proposed in the BIECO project (www.bieco.org, accessed on 15 January 2023) [48].
Appl. Sci. 2023, 13, 2415 22 of 24

Generating more statistics, reports, and future prediction based on the previous
admission session will help the university to improve its educational offer, to be more
connected to the candidate’s needs all over the world, and to co-design and co-create with
them the future of the university in the matter of creative learning in an ICT world.

Author Contributions: Project administration: S.C.M.; Methodology: O.M.; Software: L.A. and O.M.;
Vizualization: L.A. and T.F.; Validation: S.C.M. and D.D.; Writing: L.A., T.F. and O.M. All authors
have read and agreed to the published veriosn of the manuscript.
Funding: This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 101017057 (sub-grant number DIH4AI OC1
010-SCALE).
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: The data presented in this study are available on request from the cor-
responding author. The data belongs to the University of Medicine and Pharmacy “Iuliu Hatieganu”
from Cluj-Napoca, Romania (www.umfcluj.ro, accessed on 15 January 2023).
Acknowledgments: The research reported in this article has been done within HOLISUN (www.
holisun.com, accessed on 15 January 2023) company, during the admission session at the University of
Medicine and Pharmacy “Iuliu Hatieganu” from Cluj-Napoca, Romania (www.umfcluj.ro, accessed
on 15 January 2023).
Conflicts of Interest: The authors declare no conflict of interest.
Sample Availability: Ana chatbot is available on https://admissions.umfcluj.ro/, accessed on 15
January 2023 during the admission sessions.

Abbreviations
The following abbreviations are used in this manuscript:

API Application Programming Interface

AIML Artificial Intelligence Markup Language
DINA Dinus Intelligent Assistance
FAIR Findability, Accessibility, Interoperability, Reusability
FAQ Frequently Asked Questions
GDPR General Data Protection Regulation
GMAT Graduate Management Admission Test
ID Identification
LSA Latent Semantic Analysis
Q&A Questions and Answers
UMF Cluj University of Medicine and Pharmacy "Iuliu Hatieganu" from Cluj-Napoca
UDINUS Universitas Dian Nuswantoro
UX User Experience

References
1. Martínez-García, I.; Nielsen, T.; Alastor, E. Perceived stress and perceived lack of control of spanish education-degree university
students: Differences dependent on degree year, basis for admission and gender. Psychol. Rep. 2021, 125, 1824–1851. [CrossRef]
[PubMed]
2. Bakti, R.; Hartono, S. The Influence of Transformational Leadership and work Discipline on the Work Performance of Education
Service Employees. Multicult. Educ. 2022, 8, 109–125.
3. Tuhuteru, H.; Siwalette, R. System Design and Implementation of Online Admission System at XYZ University. Inspir. J. Teknol.
Inf. Dan Komun. 2022, 12, 105–117. [CrossRef]
4. Barus, S.P.; Surijati, E. Chatbot with Dialogflow for FAQ Services in Matana University Library. Int. J. Informatics Comput. 2022, 3,
62–70. [CrossRef]
5. Chandra, Y.W.; Suyanto, S. Indonesian chatbot of university admission using a question answering system based on sequence-to-
sequence model. Procedia Comput. Sci. 2019, 157, 367–374. [CrossRef]
Appl. Sci. 2023, 13, 2415 23 of 24

6. Mariacher, N.; Schlögl, S.; Monz, A. Investigating perceptions of social intelligence in simulated human-chatbot interactions. In
Progresses in Artificial Intelligence and Neural Systems; Springer: Cham, Switzerland, 2021; pp. 513–529.
7. Goot, M.J.; Hafkamp, L.; Dankfort, Z. Customer service chatbots: A qualitative interview study into the communication journey
of customers. In Proceedings of the International Workshop on Chatbot Research and Design, Online, 23–24 November 2020;
pp. 190–204.
8. Patel, N.P.; Parikh, D.R.; Patel, D.A.; Patel, R.R. Ai and web-based human-like interactive university chatbot (unibot). In Proceed-
ings of the 2019 3rd international conference on electronics, communication and aerospace technology (ICECA), Coimbatore,
India, 12–14 June 2019; pp. 148–150.
9. Chocarro, R.; Cortiñas, M.; Marcos-Matás, G. Teachers’ attitudes towards chatbots in education: A technology acceptance model
approach considering the effect of social language, bot proactiveness, and users’ characteristics. Educ. Stud. 2021, 1–19. [CrossRef]
10. Vlasiuk, R.; Petko, L. Alan Turing: A Founding Father of Computer Science, Artificial Intelligence and Modern Cognitive Science.
Ph.D. Thesis, Lviv Polytechnic National University, Lviv, Ukraine, 2022.
11. Saygin, A.P.; Cicekli, I.; Akman, V. Turing test: 50 years later. Minds Mach. 2020, 10, 463–518. [CrossRef]
12. Bassett, C. Apostasy in the temple of technology: ELIZA the more than mechanical therapist. In Anti-Computing; Manchester
University Press: Manchester, UK, 2022; pp. 168–185.
13. Agarwal, R.; Wadhwa, M. Review of state-of-the-art design techniques for chatbots. SN Comput. Sci. 2020, 1, 1–12. [CrossRef]
14. Xie, T.; Yang, X.; Lin, A.S.; Wu, F.; Hashimoto, K.; Qu, J.; Kang, Y.M.; Yin, W.; Wang, H.; Yavuz, S. Converse–A Tree-Based Modular
Task-Oriented Dialogue System. arXiv 2022, arXiv:2203.12187.
15. Park, D.M.; Jeong, S.S.; Seo, Y.S. Systematic Review on Chatbot Techniques and Applications. J. Inf. Process. Syst. 2022, 18, 26–47.
16. Tharammal, M.K.P.; Bashir, M.N.; Yusof, K.M.B.; Iqbal, S. ALICE Pattern Matching Based Chatbot for Natural Language
Communication: System Development and Testing. iKSP J. Comput. Sci. Eng. 2022, 2, 34–42.
17. Thomas, L.; Kumar, M.; Prashanth, B.S.; Sneha, H.R. Seq2seq and Legacy techniques enabled Chatbot with Voice assistance. In
Proceedings of the 2022 IEEE 2nd Mysore Sub Section International Conference (MysuruCon), Mysuru, India, 16–17 October
2022; pp. 1–4.
18. Kaczorowska-Spychalska, D. Chatbots in marketing. Management 2019, 23, 251–270. [CrossRef]
19. Hughes, T.M.; Leafstedt, J.M. Communicating with Generation Z. The COVID-19 Impact on Higher Education Stakeholders and
Institutional Services; Lexington Books: Lanham, MD, USA, 2022; Volume 93.
20. Barrett, M.; Branson, L.; Carter, S.; DeLeon, F.; Ellis, J.; Gundlach, C.; Lee, D. Using artificial intelligence to enhance educational
opportunities and student services in higher education. Inquiry J. Va. Community Coll. 2019, 22, 11.
21. Bačanin Džakula, N. Singibot-a student services chatbot. In Sinteza 2020-International Scientific Conference on Information Technology
and Data Related Research; Singidunum University: Belgrade, Serbia, 2020; pp. 318–323.
22. Hersi, A.H.; Hassan, M.M.; Hassan, A.A.; Mahdi, M.A.; Abdulle, A.W. An Intelligent Somali Language Chatbot Serving as an Online
Admission Help Desk; SORER: Mogadishu, Somalia, 2021.
23. Nazir, A.; Khan, M.Y.; Ahmed, T.; Jami, S.I.; Wasi, S. A novel approach for ontology-driven information retrieving chatbot for
fashion brands. Int. J. Adv. Comput. Sci. Appl. IJACSA 2019, 10. [CrossRef]
24. Vegesna, A.; Jain, P.; Porwal, D. Ontology based chatbot (for e-commerce website). Int. J. Comput. Appl. 2020, 179, 51–55.
[CrossRef]
25. Ranoliya, B.R.; Raghuwanshi, N.; Singh, S. Chatbot for university related faqs. In Proceedings of the 2017 International Conference
on Advances in Computing, Communications and Informatics (ICACCI), Udupi, India, 13–16 September 2017; pp. 1525–1530.
26. AgusSantoso, H.; Anisa, SriWinarsih, N.; Mulyanto, E.; Wilujeng Saraswati, G.; ; Enggar Sukmana, S.; Rustad, S.; SyaifurRohman,
M.; Nugraha, A.; Firdausillah, F. Dinus intelligent assistance (dina) chatbot for university admission services. In Proceedings of
the 2018 International Seminar on Application for Technology of Information and Communication, Semarang, Indonesia, 21–22
September 2018; pp. 417–423.
27. Colace, F.; DeSanto, M.; Lombardi, M.; Pascale, F.; Pietrosanto, A.; Lemma, S. Chatbot for e-learning: A case of study. Int. J. Mech.
Eng. Robot. Res. 2018, 7, 528–533. [CrossRef]
28. Dimitriadis, G. Evolution in education: Chatbots. Homo Virtualis 2020, 3, 47–54. [CrossRef]
29. McCarty, T.V.; Light, J.C. Supporting peer interactions for students with complex communication needs in inclusive settings:
Paraeducator roles. Perspect. Asha Spec. Interest Groups 2022, 7, 229–244. [CrossRef]
30. Sattar, A.; Surin, E.S.M.; Ahmad, M.N.; Ahmad, M.; Mahmood, A.K. Comparative analysis of methodologies for domain ontology
development: A systematic review. Int. J. Adv. Comput. Sci. Appl. 2020, 11. [CrossRef]
31. Abdelghany, A.; Darwish, N.R.; Hefni, H.A. An agile methodology for ontology development. Int. J. Intell. Eng. Syst. 2019, 12,
170–181. [CrossRef]
32. Kramer, M. Best practices in systems development lifecycle: An analyses based on the waterfall model. Rev. Bus. Financ. Stud.
2018, 9, 77–84.
33. Sidhu, D.M.; Deschamps, K.; Bourdage, J.S.; Pexman, P.M. Does the name say it all? Investigating phoneme-personality sound
symbolism in first names. J. Exp. Psychol. Gen. 2019, 148, 1595. [CrossRef] [PubMed]
34. Feine, J.; Gnewuch, U.; Morana, S.; Maedche, A. Gender bias in chatbot design. In Proceedings of the International Workshop on
Chatbot Research and Design, Amsterdam, The Netherlands, 19–20 November 2019; pp. 79–93.
35. McDonnell, M.; Baxter, D. Chatbots and gender stereotyping. Interact. Comput. 2019, 31, 116–121. [CrossRef]
Appl. Sci. 2023, 13, 2415 24 of 24

36. Singer, G.; Anuar, R.; Ben-Gal, I. A weighted information-gain measure for ordinal classification trees. Expert Syst. Appl. 2020,
152, 113375. [CrossRef]
37. Zebari, R.; Abdulazeez, A.; Zeebaree, D.; Zebari, D.; Saeed, J. A comprehensive review of dimensionality reduction techniques for
feature selection and feature extraction. J. Appl. Sci. Technol. Trends 2020, 1, 56–70. [CrossRef]
38. Wu, W.; Hou, J.; Zhang, Z.; Li, F.; Zhang, R.; Gao, L.; Ni, H.; Zhang, T.; Long, H.; Lei, M.; et al. Information entropy-based strategy
for the quantitative evaluation of extensive hyperspectral images to better unveil spatial heterogeneity in mass spectrometry
imaging. Anal. Chem. 2022, 94, 10355–10366. [CrossRef]
39. Toader, D.C.; Boca, G.; Toader, R.; Măcelaru, M.; Toader, C.; Ighian, D.; Rădulescu, A.T. The effect of social presence and chatbot
errors on trust. Sustainability 2020, 12, 256. [CrossRef]
40. Weninger, M.; Grünbacher, P.; G.; er E.; Schörgenhumer, A. Evaluating an interactive memory analysis tool: Findings from a
cognitive walkthrough and a user study. Proc. ACM Hum.-Comput. Interact. 2020, 4, 1–37. [CrossRef]
41. Díaz, E.; Arenas, J.J.; Moquillaza, A.; Paz, F. A systematic literature review about quantitative metrics to evaluate the usability of
e-commerce web sites. In Proceedings of the International Conference on Intelligent Human Systems Integration, Modena, Italy,
19–21 February 2019; pp. 332–338.
42. Kaushal, V.; Yadav, R. Exploring B2B Chatbots Adoption Experiences: Lessons for Successful Implementation in Businesses. Res.
Sq. 2022, preprint. [CrossRef]
43. Lala, S.K.; Kumar, A.; Subbulakshmi, T. Secure web development using owasp guidelines. In Proceedings of the 2021 5th
International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India, 6–8 May 2021; pp. 323–332.
44. Sarferaz, S. Lifecycle Management. In Compendium on Enterprise Resource Planning; Springer: Cham, Switzerland, 2022; pp. 559–
570.
45. Batia, L.; Rozovski-Roitblat, B. Incidental vocabulary acquisition: The effects of task type, word occurrence and their combination.
Lang. Teach. Res. 2011 15, 391–411.
46. Baxter, D.; McDonnell, M.; McLoughlin, R. Impact of chatbot gender on user’s stereotypical perception and satisfaction. In
Proceedings of the 32nd International BCS Human Computer Interaction Conference, Belfast, UK, 4–6 July 2018; pp. 1–5.
47. Rahul, K.; Banyal, R.K. Data lifecycle management in big data analytics. Procedia Comput. Sci. 2020, 173, 364–371. [CrossRef]
48. Matei, O.; Erdei, R.; Delinschi, D.; Andreica, L. Data based message validation as a security cornerstone in loose coupling software
architecture. In Computational Intelligence in Security for Information Systems Conference; Springer: Berlin/Heidelberg, Germany,
2021; pp. 214–223.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.