Behaviour & Information Technology

ISSN: 0144-929X (Print) 1362-3001 (Online) Journal homepage: https://www.tandfonline.com/loi/tbit20

An adaptive doctor-recommender system

Muhammad Waqar, Nadeem Majeed, Hassan Dawood, Ali Daud & Naif Radi
Aljohani

To cite this article: Muhammad Waqar, Nadeem Majeed, Hassan Dawood, Ali Daud & Naif Radi
Aljohani (2019): An adaptive doctor-recommender system, Behaviour & Information Technology,
DOI: 10.1080/0144929X.2019.1625441

To link to this article: https://doi.org/10.1080/0144929X.2019.1625441

Published online: 12 Jun 2019.

Submit your article to this journal

Article views: 79

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=tbit20
BEHAVIOUR & INFORMATION TECHNOLOGY
https://doi.org/10.1080/0144929X.2019.1625441

An adaptive doctor-recommender system

Muhammad Waqara, Nadeem Majeedb, Hassan Dawoodb, Ali Daudc and Naif Radi Aljohani d

a
Engineering Department, National University of Modern Languages (NUML), Islamabad, Pakistan; bSoftware Engineering Department, UET,
Taxila, Pakistan; cDepartment of Computer Science and Software Engineering, IIU, Islamabad, Pakistan; dFaculty of Computing and Information
Technology, KAU, Jeddah, Saudi Arabia

ABSTRACT ARTICLE HISTORY

Recommender systems use machine-learning techniques to make predictions about resources. The Received 1 March 2018
medical field is one where much research is currently being conducted on recommender system Accepted 25 May 2019
utility. In the last few years, the amount of information available online that relates to healthcare
KEYWORDS
has increased tremendously. Patients nowadays are more aware and look for answers to Recommender system;
healthcare problems online. This has resulted in a dire need of an effective reliable online content-based filtering;
system to recommend the physician that is best suited to a particular patient in a limited time. Collaborative filtering; doctor
In this article, a hybrid doctor-recommender system is proposed, by combining different recommendation; healthcare
recommendation approaches: content base, collaborative and demographic filtering to system
effectively tackle the issue of doctor recommendation. The proposed system addresses the issue
of personalization through analysing patient’s interest towards selecting a doctor. It uses a novel
adoptive algorithm to construct a doctor’s ranking function. Moreover, this ranking function is
used to translate patients’ criteria for selecting a doctor into a numerical base rating, which will
eventually be used in the recommendation of doctors. The system has been evaluated
thoroughly, and result show that recommendations are reasonable and can fulfil patient’s
demand for reliable doctor’s selection effectively.

1. Introduction
previously patients in general have two options when
Over the years, the amount of healthcare information on finding an appropriate doctor, but both are limited in
the Internet has been increased tremendously due to the scope and application. The first is to take up their
recent technological development. In this way, it has pro- family’s or relatives’ suggestions and the second is to
vided the user with ease to access information, but on the seek advice from friends or colleagues. Recommen-
other hand, such a large amount of information provided dations from such sources are considered to be reliable,
the user with a challenge to filter over relevant information. but the probability of having friends or family with the
This is particularly important in case of healthcare domain same medical history as one’s own is very low. Also,
where any delay or misinterpreted information can cause there is a possibility that suggestions from a patient’s
life-threatening situation. Recommender systems are prov- immediate social circle may prove insufficient to cover
ing to be a major interest of the user in recent time due to all the options in a given area. Likewise, it is often trou-
their utility of filtering out the required information blesome to find an appropriate doctor when moving to a
quickly. These systems apply machine-learning techniques new area. Another major issue is that the criteria based
to make predictions about unseen resources. on which the patient select a doctor, which is still a deba-
Recommender systems have become vitally important table problem. Although some recommender system and
for research since the mid-1990s. Many of the existing online health portal are developed to address the afore-
literature underline the application of recommender sys- mentioned problem, they are limited in scope. Most
tems in various domain like books, images, movies, etc. existing systems on doctor recommendation, which has
(Park et al. 2012). In recent times, the medical domain been thoroughly discussed in the literature review,
is one domain where the recommender system’s utility focus solely on recommendation techniques in which
has been recognised. Further research work is going on no valuable effort has been made for the generation of
to further enhance the capability of recommender sys- effective recommendations according to patient’s prefer-
tems in healthcare (Portugal, Alencar, and Cowan ence for a particular doctor. Furthermore, some existing
2018). In the background of doctor recommendation, health portals provide extensive information regarding

CONTACT Ali Daud [email protected], [email protected]

© 2019 Informa UK Limited, trading as Taylor & Francis Group
2 M. WAQAR ET AL.

healthcare, but finding relevant information in quick The overall outcome of the research is that rec-
time is a hectic task while dealing with these portals. ommendations generated by our proposed system have
There is a dire need to capture the preferences of the improved significantly in terms of accuracy and patient
patient regarding effective doctor recommendation. In satisfaction. It is imperative that any generalisation of
this regard, identification and prioritisation of features our research outcomes uses the additional research
of a doctor upon which patients make their choices data to incorporate additional features in our doctor
will help to make doctor recommendations more effec- ranking function. To obtain these data, various online
tive. The presented research proposes an algorithm healthcare web sources may be useful.
through which a doctor ranking function is constructed
using the analytical hierarchical process. The resultant
2. Literature review
ranking function includes many features of doctors and
their respective preferences as given by patients. The Advancements in technology have led to a need for inno-
proposed algorithm is adaptive, in a sense, because the vation in every field, and when it comes to the medical
final weight assigned to identify attributes can be field, it is vital to have a reliable system to recommend
adjusted, if required, by applying the proposed algor- a doctor quickly, according to a patient’s needs, as life
ithm. For this, only patient data are required, and after- is at stake (Haughom 2016). The availability of the Inter-
ward proposed algorithm will automatically calculate the net allows organisations and users to maintain and
weighting values for the attributes from these data. access health-related data online, and using a recommen-
The proposed research work presented an effective der system has enabled them to use the information
adoptive recommender system to address the problem of more accurately.
doctor recommendation. Propose algorithm focuses on
obtaining the preference of the patient regarding doctor
2.1. Recommender systems
recommendations for a particular case. Furthermore,
algorithm converts these subjective preferences of patients A recommender system is basically a tool to make sug-
into an objective nominal ranking function. This ranking gestions to end users about the products and items that
function is eventually used for ranking doctors and gener- will suit them. It promotes convenience for the user by
ation of recommendations. In a nutshell, recommendation giving them the item of their choice. It also provides
that is produced as a result of the application of the pro- ease in finding out the end user’s interests (Alluhaidan
posed algorithm is capturing patient’s consideration 2013). A recommender system’s main motive is to
regarding doctor selection effectively into an empirical make recommendations to users who possess certain
model. Thus, a system is developed in the context of core preferences, and these preferences act as the input.
user’s perspective of effectively using a system for his These systems make the recommendations on the basis
need, which is a core quality attribute of the recommender of a user’s history of liking or purchasing the product.
system’s reliable implementation. Item-based filtering is a technique (Badrul et al. 2001)
The main contributions of this research are as follows: in which a user’s previously selected items are used to
find similar items. These are then recommended to
. The features of a doctor, which influence patients, are other users. User-based collaborative filtering (Melville
identified through a survey. and Sindhwani 2013) is another type of technique in
. It proposes an algorithm to construct a ranking func- recommender systems. Similar users’ ratings are
tion from the identified attributes using analytical employed to recommend their most highly rated items
hierarchical processing. This helps patients to identify to other active users. Another variant of recommen-
and locate a doctor who meets their requirements, dation techniques is demographic filtering (Safoury
which is unprecedented, to the best of knowledge. and Salah 2013), which exploits the demographic attri-
. Propose algorithm constructs a hierarchical structure butes of users to recommend new items. In the hybrid
to assign weights to above-identified attributes. recommendation technique (Balabanović 1997), differ-
. An effective hybrid recommender system is con- ent recommendation techniques such as content filter-
structed by combining different machine-learning ing, collaborative filtering and so on are combined to
techniques, i.e., content, collaborative and demo- generate recommendations. Due to their utility, recom-
graphic filtering. This system will use our proposed mender systems are used extensively in various domains.
doctor ranking function to evaluate a doctor. The medical field is one in which they are proving their
. The performance of our proposed model is analysed by worth. With ever-increasing volumes of healthcare infor-
comprehensive experiments, with baseline and pro- mation available on the Internet, the demand for recom-
posed features. mender systems is growing daily. Although limited work
 BEHAVIOUR & INFORMATION TECHNOLOGY 3

has been carried out in the healthcare, significant gaps as a core component of healthcare recommender sys-
are recognised, which makes room for an effective sys- tems. The article proposed by Blue and OPTUM Inc
tem that can recommend doctors on the basis of patient (2018) entailed methods that could find similar patients
choice. related to patient X. Patient similarity metrics was calcu-
lated after which ranking of subsequent patients was
done. Furthermore, recommendations were generated
2.2. Recommender systems in healthcare
for top-ranked similar patients. Recommender systems
In the present era of information, people prefer to search also used to recommend treatment according to patient
in the Internet to obtain data regarding disease and its symptoms. An algorithm proposed by Ceyhan, Orhan,
related treatment. In the current scenario, many health and Domnori (2017) recommends medical tests on the
portals are providing the related information, but recom- basis of an analysis of patient’s symptoms and ana-
mender system are proving to be the best sources due to mneses. It uses sequential minimal optimisation, J48
their advantage of being able to filter out relevant infor- and Random Forest to recommend tests to patients
mation in quick time. Cheung et al. (2019) identified core according to their symptoms and diseases. Moreover,
challenges that are to be tackled to increase the efficiency the impact of patient’s emotions on recommender sys-
of the recommender system in the near future. Recent tems is discussed by Qian et al. (2019). The author
vast data availability on the Internet has caused the used the user’s reviews as emotional information and
need of optimal solution where there must be easy access used it along with the user’s rating and user’s social
to related information in quick time. Tabari and Memar- data. Three categories related to healthcare systems
iani (2019) presented a mathematical model that paves were identified at the end: health promotion, theoretical
the way for developing an effective decision management factors and behaviour change theories. To capture the
system for efficient resource sharing. Moreover, the emotion of a patient, an improvement in design related
article presented by Schafer, Hor-Fraile, and Pawan to healthcare systems was discussed by Chen et al.
(2017) also discussed existing literature published in (2017), in which video conference-based system was pro-
last decade in healthcare domain regarding recommen- posed to overcome the problem of time and space in the
der system utility. The author used a keyword search healthcare domain. 5G cognitive system having two
to filter out articles from top-ranked journals. The parts, resource cognitive engine and data cognitive
focus was on only journals published in English. The engine, was used. To further investigate the quality of
main aim was to identify gaps in the field of healthcare. system, speed emotion recognition prototype was
With the emerging increase of information on the Inter- deployed to evaluate the effectiveness of proposed
net today, digital world is composed of big data where work. While dealing with the patient’s data in healthcare,
there is a great interest in predicting customer’s interest privacy and confidentiality of patient’s data is one of the
and behaviour. Guha et al. (Blue and OPTUM Inc 2018) most obvious concerns, which has to be deal with utmost
further analysed the impact of data analytics on the care. Kaur, Kumar, and Batra (2018) used Paillier homo-
domain of healthcare and future prospects where the morphic encryption system to enhance accuracy, secur-
usage of technologies like cloud computing, Internet of ity and performance of healthcare. The presented
Things (IOT), and their associated challenges was high- article mainly focused on maintaining the confidentiality
lighted. Furthermore, recommender systems utility in of patients, which is very critical when dealing with the
medical is evaluated by Schafer, Hor-Fraile, and Pawan healthcare data. Furthermore, the recommender system
(2017) in which effort has been made to investigate the presented by Hoens, Blanton, and Chawla (2013)
progress made by recommender systems toward includes a privacy-friendly framework. Two architec-
finding appropriate healthcare and intervention for tures using a secure privacy framework and an anon-
medical assistance. The article analysed the perspective ymous contribution framework allow patients to rate
of a recommender system in the medical domain of doctor in a protected environment. However, the main
the future and corresponding challenges. focus is on encrypting the data to increase the system’s
reliability.
2.3. Patient-based recommender systems
2.4. Experts recommender systems in the
One of the important aspects in the healthcare system is
healthcare domain
finding similar patients related to a particular disease.
Patient-based decision-making in the medical domain In recent times, many expert systems in recommender sys-
can increase the utility of recommender systems to a tem domain have been deployed to tackle various issues.
great extent. In that regard, patient can be considered An algorithm has been developed to improve the
4 M. WAQAR ET AL.

performance of the Shanghai Medical League Appointment between a patient’s symptoms and his or her treatment to
Platform (Huang et al. 2012). This involves a time-sharing find similar patients and then recommended doctors who
mechanism to reserve doctors for patients, which solely are rated highly by them. The main drawback of the sys-
focuses on providing assistance in the effective manage- tem was lack of any proposed strategy to evaluate how a
ment of appointment and scheduling time between patients patient rates a particular doctor. The recommender sys-
and doctors while neglecting other important attributes. tem presented by Salunke, Kasar, and Smita (2015) used
Recommender system presented by Hussein et al. (2013) natural language processing and user ratings to construct
includes making predictions about the disease for a given a doctor profiler. The main focus was on generating rec-
patient using the multiple tree disease classification models, ommendations on the basis of user ratings, but exactly
but again the main focus was on disease and no effort has on which type of factors the patient rate for a particular
been made to propose any criteria or features by which doctor was still unclear. A doctor-recommender system
evaluation of a doctor for a given patient can be done. proposed by Guo et al. (2017) identified key opinion lea-
Wiesner and Pfeifer (2010) presented research that tried ders, which will influence people’s opinion regarding a
to exploit different concepts, problems and technical issues doctor. The authors used co-authorship and citation
relating to health recommender systems. They proposed a relations to identify doctors who are expert in a particular
recommender system whereby patients can enter their dis- disease and used these doctors’ expertise to recommend a
ease symptoms through an online system. Disease recog- doctor for a particular disease. The main weakness of the
nition is achieved through expert doctors registered on proposed work was that it worked only on online data. If
the system. However, there is no strategy to evaluate a there are no online data, the proposed algorithm fails to
doctor’s suitability for a given patient. Diet plays a critical provide any effective solution.
role in a person’s health and provides immunity against There has been very limited work carried out in the
disease. Mika (2011) provided a framework to recommend domain of doctor-recommender systems. The existing
an appropriate diet for patients, which helps with both the systems focus exclusively on obtaining recommen-
prevention and the treatment of heart-related disease. It is dations for doctors by using various machine-learning
based on a person’s family heritage, in addition to basic techniques. Also, the evaluations of the quality of such
information about the disease and the patient’s food prefer- systems are very limited in scope. As most related
ences. Furthermore, a mobile application–based recom- work on doctor recommendations has been based solely
mender system, Mobi Day (Lamber et al. 2009), on the recommendation techniques, some important
proposed a personalised context-aware mobile service research questions regarding doctor recommendation
that is embedded in hospital information systems. It arise, which are as follows: On which criteria do patients
helps patients in competing his day-to-day work in hospital choose a particular doctor? How can we understand the
by exploiting the related data, such as the patient’s current preferences of a patient while looking for a particular
location and their history of message-reading behaviour. It doctor? How may these criteria/preferences can be
consists of a server that sends a message then monitors the used to evaluate a particular doctor for a given patient?
user’s behaviour to predict trends. A nursing-recommender How can we evaluate the performance of a given doctor?
system presented by Duan, Street, and Lu (2008) presents Also, how can we analyse a medical recommender sys-
nursing plans using correlations among nursing diagnoses, tem to assess its quality? Proposed work investigates
outcomes and interventions to produce a rank list of aforementioned issues and proposes a novel adoptive
suggested care plans based on pre-selected items. The algorithm to address them.
work is based solely on association measures such as sup-
port and confidence.
3. Machine-learning recommendation
techniques
2.5. Existing doctor-recommender systems in
healthcare 3.1. Collaborative filtering
All previous works that have been discussed so far mainly Collaborative filtering is a machine-learning method to
relay on predicting disease on the basis of treatment. No make predictions and recommendations on the basis of
valuable effort has been made to specifically understand similar users in the system. The basic idea is to combine
the preference of patient while choosing a particular the opinions that are similar to those of the new user
doctor. Although some doctor-recommender systems and aggregate them in such a way that a reliable predic-
are developed, yet they are very limited in terms of their tion is generated for each new user. Collaborative filter-
scope and effectiveness. A recommender system proposed ing generally comprises two steps: The first is to find
by Narducci et al. (2015) utilised the semantic relationship users with a similar rating pattern to the new active
 BEHAVIOUR & INFORMATION TECHNOLOGY 5

user and the second is to use the rating of these similar vectors, and the similarity between them is represented
users to make a recommendation for that particular by the cosine angle. It is calculated by taking the dot pro-
new user. The filtering may one of the following two duct of vectors and dividing it by the product of magni-
types. tude of respective vectors. Its maximum value is 1,
indicating maximum similarity, and the minimum
3.1.1. User-based collaborative filtering value is 0, indicating minimum or no similarity.
User-based collaborative filtering is a simple type of col-
B
laborative filtering whereby we intend to find users simi- sin (A, B) = cos (u) = A. (1)
mag|A|∗mag|B|
lar to a new active user, find their ratings and, by
aggregation of those ratings, generate a recommendation where A and B are the two vectors for which cosine simi-
for an active user. larity is to be calculated.

3.1.2. Item-based collaborative filtering

Item-based collaborative filtering is a type used to calcu- 3.4. Hybrid approach
late the similarity between two items using user ratings of A hybrid approach generally combines content and col-
those particular items. It generally involves two steps. laborative filtering, aggregates their advantages and tries
The first is to use a similarity function to calculate the to minimise the disadvantages of each individual
similarity between item pairs. The second includes approach to make an efficient recommendation available
using the items most similar to a user’s already-rated for users. Collaborative filtering generally suffers from
items to generate a list of recommendations. sparsity, whereby users generally do not rate most of
the items, causing difficulty in finding users with same
3.1.3. Demographic collaborative filtering rating pattern as the user for whom the recommendation
Demographic collaborative filtering involves producing is to be made. Another problem is of ‘cold start’, where
recommendations based on certain user characteristics, no user has yet rated an item; hence it is not possible
such as age, gender, location and so on, and it is generally for it to be recommended to anyone. Users with unique
combined with collaborative filtering to make precise tastes often cause problems in recommendations. Con-
recommendations. tent filtering-based recommendations are often inaccur-
ate and inefficient. A hybrid recommender could use
3.2. Content-based filtering content-based filtering to match the new item to existing
items, based on the metadata, allowing it to be rec-
Content-based filtering is an approach that recommends ommended anyway. It also entails collaborative filtering,
items by comparing the content of an item with the con- so it can use the wisdom of crowds for efficient
tent of the user’s profile. The content of each item is gen- recommendations.
erally broken down by the process of tokenisation; each
token represents a keyword. Similarly, the user profile is
broken down into tokens, and the keywords are gener- 3.5. Weighted average
ated. After comparing the similarity of the keywords
on the item and the users’ profile, a recommendation is A weighted average is where each value to be averaged is
generated. multiplied by a particular weight (Investopedia 2016).
The value of these weights determines the relative impor-
tance of a particular attribute.
3.3. Similarity measure
Weighted average} = w1 ×1 + w2 ×2 + · · ·
A similarity measure in a machine-learning algorithm is
used to assess the similarity between various items or + w n xn (2)
users; by using this similarity, a prediction is made. where w1, w2, … wn are weights and x1, x2, … xn are the
There are many types of similarity measures in machine values that are to be averaged.
learning, including Euclidian distance, cosine similarity
measures and fuzzy distance.
4. Proposed methodology
3.3.1 Cosine similarity
4.1. Analytical hierarchy model
A cosine similarity is a similarity function used to
implement user-to-user collaborative filtering. It is a vec- The analytic hierarchy process (AHP) (Saaty 2013) is a
tor-based approach whereby users are represented by method that can be utilised to facilitate complex decision
6 M. WAQAR ET AL.

problems to choose the best alternative among other avail- underlying question of the criteria used to select a doctor
able alternatives. In the proposed system, a doctor ranking for a particular patient.
function has been constructed using the AHP model. AHP
assists in evaluating both subjective and objective parts of a
4.5. Entities selection criteria
decision-making process. The steps involved are as follows:
first, by understanding the patient’s criteria of choosing a All the medical entities identified as having direct invol-
particular doctor, some important attributes of doctor are vement in the assessment of a doctor’s attributes for
identified using a survey that was conducted through ques- selection by patients are included in the research.
tionnaires. Furthermore, a hierarchical structure of AHP
was built for these identified attributes by the application
4.6. Sample size determination
of a novel proposed algorithm. The AHP generates weights
for each identified attribute of a doctor. Higher weight attri- To determine the sample size, we used the confidence
butes will be given more importance. Finally, the AHP level and confidence interval as a matrix. We targeted a
combines the weight of all these attributes, and the corre- relatively small population living in the vicinity of Isla-
sponding quantitative doctor ranking function represent- mabad. Islamabad has been divided into many sectors,
ing a global score was generated. This ranking function and for proposed survey, G-9, G-8, G-6, and G-7 sectors
will be eventually used in the recommendations of doctors. are used. The data have been collected at random. By
keeping a confidence level of 95% and a confidence inter-
val of 7, by using the statistical model given by Best sur-
4.2. Survey vey software (2017), sample size obtained was of 150
Effort has been designed to design and conduct survey in samples.
accordance with the existing quality framework for con-
duction surveys (Stockemer 2019, Masood et al. 2018). 4.7. Sample questionnaires
First, we collected our data from three hospitals in Isla-
mabad, Pakistan. We designed questionnaires to gather Our main aim was to establish the core attributes that
information from patients. Three hospitals selected for influence patients when selecting a doctor for a particu-
our survey were (1) Pakistan Institute of Medical lar disease. We tried to investigate the aforementioned
Sciences (PIMS); (2) Polyclinic Hospital and (3) Capital issue to design a survey that asks patients first to mention
Development Authority Hospital. PIMS is a research- the doctors’ attributes that influenced them to select
oriented health sciences institute in Islamabad. It is one them for a particular disease and relative effectiveness
of the region’s leading tertiary-level hospitals and of doctor-recommender system if implemented effec-
includes 22 medical and surgical specialist centres. The tively. Next, we asked the patients to prioritise the
polyclinic is a 1100-bed hospital constructed to serve doctors’ attributes that influenced them to choose a par-
the residents of the city of Islamabad, while the Capital ticular doctor. The following is the survey sample list of
Development Hospital is of 250 bed that aims to serve questions used in the survey of an adaptive doctor-
the people of the capital territory. recommender system.

4.8. Questionnaire analysis

4.3. Instrument
In this experiment, a questionnaire was the primary
We designed a comprehensive and nominal question- source of data. We distributed the questionnaire
naire to collect data, using doctors and patients as the among patients in hospitals and gathered our data
unit of randomisation, to test the hypothesis that there from them. Our aim was to investigate the data and
are attributes by which a patient selects a particular use them to identify patients’ criteria when selecting a
doctor. Our basic aim was to come up with objective, particular doctor.
number-based criteria to evaluate and recommend a
doctor for a particular patient. The following describes
the survey sample that we used for our research. 4.9. Sample dataset overview
We gathered 150 patients’ data for our dataset, as this
was the required sample size. We collected the data
4.4. Sampling
from three different Islamabad hospitals. These are the
We built our model through questionnaires and analysed main hospitals in Islamabad, which is the capital of Paki-
the results to obtain a thorough understanding of stan, and most people like to be treated here due to good
 BEHAVIOUR & INFORMATION TECHNOLOGY 7

facilities provide here. For this reason, it was convenient Table 2. Demographics of patients participated in survey.
for us to find people belonging to various financial, social Characteristics Attribute type Data sample
and cultural backgrounds, helping us to make our Gender Male 80
Female 70
extracted patterns more realistic and diverse. We used Age 20 and below 10
the data gathered from these patients and applied to 21–25 30
25–30 40
our proposed algorithm. The final result of our proposed 31–35 40
algorithm is a doctor ranking function. This was used to 36 and above 30
generate a ranking for a doctor. In the end, to obtain Hospital name PIMS Hospital 65
Polyclinic Hospital 45
expert opinion on our proposed research and the system, Capital Development Authority Hospital 40
we used nine doctors as domain experts to evaluate our
work; eight patients had been used to evaluate our sys-
tem from a patient’s perspective. Table 3. Feature used in the system.
Name of
attribute Description Status Source
Location Location of a doctor New Survey during the
4.10. Feature engineering proposed research
work
An attribute set is formed by considering the require- Average Average fee charged by New //
charged the doctor
ments of our proposed algorithm as listed in Table 1, Education Education and courses New //
Question3. In general, patients specify different kind of taken by a doctor
Office General environment of New //
attributes regarding appropriate doctor’s selection, but environment doctor’s clinic
after critical analysis of questionnaires, seven doctor’s Experience Experience of the New //
attributes were identified, which are ought to be pre- doctor in years
Behaviour Behaviour of doctor New //
ferred most by patients while looking for doctors. Scheduling Ease of scheduling time New //
These attribute sets are built with regard to patient sub- time of a doctor
jective criteria for doctor selection (Table 2). An effort
has been made to categorise different attributes of a this preference into nominal doctor ranking function,
doctor under some generalise categories, which will which is a core utility of the proposed algorithm.
help in understanding patient criteria for doctor selec-
tion. Moreover, empirical framework will be built to
transform above subjective criteria into nominal model 4.11. Construction of hierarchical structure and
using proposed novel algorithm. Table 3 enlists these weight calculation
attributes. Furthermore, patients were asked to prioritise
For the weight calculation, first calculated value is r(i)indv ,
these seven attributes from priority number 1 (being
the individual rating of each patient for a given doctor’s
most important) to priority number 7 (being least
attribute. It is calculated using the following formula:
important). Table 4 shows the sample form, which is
used to gather priority number. This was actually done r(i)indv = Rmax /Pnumber(i) (3)
to analyse and to gather the relative preference of
where r(i)indv is the individual rating given by each patient
patient’s for above-mentioned doctor’s attributes. More-
to given attribute i; Rmax is the constant maximum rating,
over, proposed novel algorithm will eventually covert
which in our case is 5 and Pnumber is the priority number
given by the patient to given attribute i ranges from 1 to 7.
Table 1. Sample list of questions used in questionnaire-based In this way, r(i)indv was calculated from the priority
survey of an adaptive doctor recommendation system. numbers, as listed in Table 5, which is given by patients
Serial Acceptable
number Questions answer
to all seven attributes. After this calculation, calculation
1 What is your opinion about online doctor- Useful/Not useful
of r(i)agg was done, which is actually the aggregated rating
recommender system idea?
2 Have you ever used any online healthcare User answer/Free Table 4. Features with priority numbers.
plate form? If yes can you tell any answer
Attributes Priority number(1–7)
deficiencies in those systems?
3 What kind of doctor’s attributes you look User answer/Free Location
for while seeking an appropriate doctor? answer Average Fees charged
Enlist them Education + courses
4 Enlist sources from which you generally get User answer/Free Office environment
appropriate doctor’s recommendations answer Experience
5 In what ways doctor recommendation will User answer/Free Behaviour
be useful for you? answer Sheduling time
8 M. WAQAR ET AL.

calculated as follows: environment, Exp = experience, Beh = behaviour and

Shed = scheduling time.


N
r(i)indv
r(i)agg = (4) The overall weights assigned to doctors’ attributes are
i=0
N shown in Table 6. The above ranking functions show
that, of all the attributes, experience, fees and location
where r(i)agg is the aggregate rating of attribute ‘i’ by all are those preferred by patients. Equation 8 shows the
patients; r(i)indv is the individual rating given by each ranking function that can be used to evaluate doctors.
patient to the given attribute i and N is the total number It effectively translates patients’ criteria for selecting
of patients rated. doctors into a number-based doctor rating. The doctors
Now Rcom , which is the cumulative rating obtained by recommended on the basis of above rating are likely to
adding r(i)indv of all attributes given by a single patient, is prove highly efficient for patients using the system.
calculated as follows:


k
Rcom = r(i)indv (5) 4.12. System implementation
i=0
The proposal is a web-based system that is implemented
where k is the total number of attributes, which in our in php/mysql. It consists of a doctor module and a
case is 7. patient module. The overall system workflow is that
For final weight calculation, we used the following doctors register themselves on the system and update
equation: the information on their specialty, experience, location,
and average check-up fee. Patients register themselves


k
on the system and upload information such as their
w(i) = r(i)agg/Rcom (6)
i=1 location, age, disease, severity, and average expenditure
limit. The patients rate doctors on the basis of their
In this way, w(i) , which is the weight for a given attribute experience of a particular doctor. The proposed novel
‘i’, is obtained for all seven attributes. adoptive algorithm generates a doctor ranking function
Next, the final ranking function Frank was obtained by using AHP. An effort has been made to capture a
multiplying the corresponding calculated weight w(i) patient’s criteria for selecting a doctor by using the
with factor i, i.e. fi . above ranking function. Moreover, it is then used to gen-


k erate an overall rating for a doctor. Demographic, con-
Frank = wi ∗fi (7) tent-based and collaborative filtering are all used to
1=1 generate the recommendation of a doctor for a patient.
where Frank = final ranking function; k = number of attri- A weighted average is used to develop the overall rat-
butes, which in our case is 7. ing of a doctor. Each patient can rate a doctor only once;
After applying the above methodology and putting in if a patient tries to rate a particular doctor more than
the corresponding values, the final ranking function that once, the earlier rating will be removed and it will be
we obtained is as follows: updated by the new rating. In this way, an overall rating
of a particular doctor is maintained. Each patient’s
Frank = 0.15∗Loc + 0.20∗Fee + 0.10∗Edu + 0.08∗Env profile is maintained by asking them to provide infor-
+ 0.18∗Exp + 0.21∗Beh + 0.08∗Shed mation about their disease, severity, their location, age,
gender and average charge that they can afford. Each
(8)
new patient registering on the system will be matched
where Loc = location, Fee = fees, Edu = education, Env = to all other patients already registered using cosine simi-
larity. The top N matched patients will be found relative
Table 5. Priority number and corresponding r(i)indv . to a new active patient. The doctors rated most highly by
Sample priority number(1-7) given Calculated
Attributes by patient r (i)indv
Table 6. Attributes with weight values.
Location 1 5
Attribute Weight assigned
Average fee 2 2.5
charged Location 0.15
Education+courses 3 1.67 Average fee charged 0.20
Office 4 1.25 Education+courses 0.10
environment Office environment 0.08
Experience 5 1 Experience 0.18
Behaviour 6 0.83 Behaviour 0.21
Scheduling time 7 0.72 Scheduling time 0.08
 BEHAVIOUR & INFORMATION TECHNOLOGY 9

these patients will be recommended to the new patient. our novel proposed algorithm will find patient’s similar
The flow of proposed adoptive method is shown in to active patient using different machine-learning tech-
Figures 1 and 2. The proposed adaptive method ident- niques, i.e. collaborative, content and demographic filter-
ifies the weight values for a particular doctor’s attributes. ing. Furthermore, doctor recommendations are
Figures 1 and 2 show the overall workflow of the pro- generated for active patient. These doctor recommen-
posed system, whereas Figures 3–5 show implemented dations are generated after analysing the doctor prefer-
view of the proposed system into a web-based system ence of all similar patients, and afterward, highly
in php. ranked doctor by these similar patients are rec-
Figure 3 shows the main web page of the proposed ommended to active patient. The ranking of doctors is
system where patients and doctors can register them- constructed using proposed adoptive algorithm. Pro-
selves for the system. Moreover, profile is maintained posed system will also effectively locate recommended
for patients and doctors. Figure 4 shows the patient doctors by using its location tracking feature as depicted
profile view of system. Patient can add corresponding in Figure 5.
attributes, e.g., average expenditure, location. Afterward,

4.13. Proposed algorithm

The prime objective of the proposed hybrid technique is
to suggest a doctor for a patient according to the patient’s
choice through using the ratings of other patients who
have similar features/attributes to the active patient.
The proposed algorithm has the following steps:
Input: Patient data
Output: Recommended doctors
1. Count ‘n’ unique entities in dataset
2. Count number of responses for each unique entity
3. Create set for all unique entities of the dataset
4. While end of file
a. for iteration 1 to n, where n is total number of
patients surveyed
(i) Computer(i)indv using eq (3).
(ii) Compute r(i)avg using eq (4)
(iii) End For
b. for iteration 1 to k, where k is the total number of
attributes, which in our case is 7
(i) Compute Rcom using eq (5)
(ii) Calculate w(i) using eq (6)
(iii) Return a ranking function using eq (7)
(iv) End For
5. End While
6. Get the value-based final ranking functions
7. Recommend doctors on the basis of the above ranking
function, using various machine-learning techniques
8. END
User-based collaborative filtering is employed to find
patients similar to the active patient who logs onto the
system. Similar patients are then used to recommend a
doctor to the active patient. Demographic collaborative
filtering is used to recommend the nearest doctor to
the active patient using the haversine formula. Item-
based collaborative filtering is used to find related
doctors once the patient has marked any doctor as his
Figure 1. Proposed system workflow. or her favourite.
10 M. WAQAR ET AL.

Figure 2. Proposed adaptive algorithm.

Figure 3. Proposed system home page.

In this way, a hybrid approach is taken to solve the particular doctor was checked, and the final con-
problem of identifying the right doctor, which is often clusion of the survey was that customers should take
a headache. The above algorithm is an efficient approach, these ratings with a large pinch of salt, as ratings by
combining the advantages of various machine-learning a small group of either happy or unhappy patients
approaches. often have a undesirable effect on a physician’s overall
rating.
To address this problem, we used the trimmed mean
4.14. Improvements to the system’s reliability method to remove certain undesirable values from the
system.
The challenge with online public ratings is that they
often suffer from the problem of trustworthiness, as
the probability of inaccuracies is great. A recent survey 4.14.1. Trimmed mean
on the accuracy of physician ratings was conducted by It is often the case that the smaller the number of users
the Journal of Urology America (Glover 2014), where who have rated a particular item, the greater is the
by 500 US urologists were selected from the database chance of inaccuracy in the rating. If the number of
of 9,940 urologists. The accuracy of the ratings for a users who rated a particular item is great, the reliability
 BEHAVIOUR & INFORMATION TECHNOLOGY 11

Figure 4. Patient’s profile and doctor recommendations view.

Figure 5. Doctor’s location tracking page.

12 M. WAQAR ET AL.

is generally considered accurate for that item. Hence, the measured by

factor ‘number of users rated for a particular item’ is vital
in determining the accuracy of a rating. For this reason, F1 − measure = 2∗precision + recall/precsion
our system keeps track of the number of users who have + recall (11)
rated a particular doctor. In this way, with increasing
numbers of users in the system, reliability improves. The overall efficiency of the system is evaluated by
The trimmed mean (Andale 2014) has also been used analysing the recommendations generated for eight
as way to improve system reliability. It is an efficient way different patients and collecting their responses against
to improve the accuracy of ratings by removing a certain these recommendations. In this way, corresponding
percentage of the smallest and largest values before cal- values for all three metrices are obtained. Table 7
culating an overall rating. shows average evaluation of metrices by eight distin-
guished users, who were given five iterations to evaluate
a system. The main reason for giving five iterations is to
have thorough evaluation of given metrices to avoid any
5. Result and discussion outlier if it originates during evaluation process. Figure 6
To effectively asses the quality of the proposed system, an shows comparison of metrices results when evaluated by
effort has been made to evaluate a system from both
patient’s perspective and doctor’s perspective. Total of Table 7. Mean of evaluation metrices for 8 random patients.
nine doctors were used as domain experts to evaluate Patient number No of iterations Precision Recall F-measure
the system in context of its application domain. Selected Patient 1 I 85.71 84.67 86.61
doctors who were used for quality evaluation belong to II 66.22 63.23 65.12
III 71.47 61.62 65.73
various specialties such as gynaecology, general medicine IV 72.63 73.45 75.25
and pathology. User assessment of evaluating recom- V 84.02 83.35 85.07
Mean 72.89 70.67 71.66
mender system to quantify its objective is a very impor- Patient 2 I 43.74 54.59 54.12
tant factor in analysing recommender system in term of II 45.69 55.89 59.42
its quality. In this regard, eight patients were also used to III 78.72 83.75 84.49
IV 63.42 71.45 69.86
evaluate proposed system from patient’s perspective. V 45.81 53.95 53.12
Moreover, the proposed final ranking function F-rank Mean 55.13 63.15 62.20
Patient 3 I 71.65 70.24 72.11
was presented to doctors for evaluation. Metrices that II 82.12 82.85 85.95
were used for the performance evaluation of proposed III 63.24 66.50 62.87
IV 71.12 72.17 71.12
system are precision, recall and F-measure as given by V 84.12 83.85 83.97
Equations (9), (10) and (11), respectively. These metrices Mean 73.45 73.98 71.69
Patient 4 I 35.82 41.98 35.72
are used extensively when dealing with evaluation of II 52.95 62.95 55.49
recommender systems (Rawat and Dwivedi 2019), III 51.12 54.35 52.05
which lie under the umbrella of supervised machine IV 70.65 70.62 70.62
V 65.82 83.85 74.79
learning. Precision is a measure of most relevant items Mean 54.25 62.86 58.72
in recommendations. It tells about the accuracy of Patient 5 I 53.55 42.95 47.79
II 72.55 71.35 73.86
mode of predicted positive cases, which are actually cor- III 52.65 62.92 57.79
rect recommendations out of total recommendations. It IV 72.17 81.42 72.95
V 72.55 78.85 78.12
is given by Mean 62.07 66.08 56.21
Patient 6 I 61.75 54.12 69.07
T(P)
Precision = (9) II 45.42 50.61 59.97
T(P) + F(P) III 55.32 56.55 52.35
IV 82.72 63.22 71.85
V 91.82 71.92 77.65
where T(P) =true positive; F(P) =false positive. Mean 68.47 61.18 62.12
Recall is a metrics used to evaluate a model when Patient 7 I 43.72 45.76 45.80
there is high cost associated with false negative rec- II 75.72 74.75 72.72
III 73.75 82.40 75.78
ommendations. It is given by IV 85.82 86.90 85.83
V 85.87 86.90 85.88
T(P) Mean 73.77 75.56 73.56
Recall} = (10) Patient 8 I 51.12 74.71 61.07
T(P) + F(n) II 52.45 61.23 53.89
III 66.81 62.22 61.67
where T(P) =true positive; F(n) =false negative. IV 65.22 86.81 73.29
While F1 is function of recall, which is needed to V 73.97 86.81 84.86
Mean 61.25 73.26 65.45
obtain balance between precision and recall. It is
 BEHAVIOUR & INFORMATION TECHNOLOGY 13

different patients. These values prove the efficiency and system when evaluated by domain experts. Result shows
accuracy of system in all aspects. that the proposed system solved the problem of doctor
Owing to their critical nature, medical recommen- recommendations to good effect when evaluated by
dation systems are often thoroughly evaluated by domain experts.
domain experts in the context of their domain to mini-
mise error in the system. After careful analysis of the
6. Conclusions and future work
existing system, we came up with an evaluation metric,
mean absolute error (MAE), which has been used in There is a vast amount information that has been avail-
most of the existing systems, so we used this to evaluate able on the Internet due to recent development in tech-
our system from doctor’s perspective. nology worldwide. On the one hand, it is providing user
MAE is actually the difference between a predicted with facility to access required information easily, but on
value and the actual value. It calculates the error rate, the other hand, it often becomes hectic for user to filter
and a lower value means a highly accurate system. We out relative information from the huge bulk of infor-
used nine doctors’ reviews for the evaluation. The aver- mation residing on the Internet. The problem further
age rating obtained as the result of the doctors’ evalu- aggravates in case of healthcare information where any
ation of our system was 3.9. We used Equation 12 to delay can have life-threatening consequences. In recent
obtain the error rate for our proposed system: times, there has been an increased interest of users
regarding recommender systems utility as they filtered
1
i=9
out required information in an efficient way.
MAE = r ′ − r(i) (12)
Dtest i=1 (i) In this study, efforts were made to establish the impact
of proposed recommender system in domain of health-
where Dtest = number of ratings for which prediction is to care. An adoptive algorithm was proposed for effective
be made; r′ =predicted rating, which in our case is 5; doctor recommendations generation. Specific attributes
r=actual rating given by each doctor. related to doctors are identified by survey. Our proposed
r′ , r Dtest algorithm translates the above features and their relative
importance into a nominal empirical model effectively
Dtest in our case is 9, as we used nine doctors for our using analytical hierarchical process (APH). This
evaluation. By using the actual rating value given by model is further used to make a recommendation of
each of these nine doctors and putting values in the doctors to patients. Recommendations are generated by
above formula, the MAE value obtained is 0.22. The various machine-learning techniques. The recommen-
value of MAE demonstrates the accuracy of the proposed dations produced mainly underline how patients have

Figure 6. Precision, Recall and F-measure values for the proposed system.
14 M. WAQAR ET AL.

interest in group of patients having similar disease or Chen, Min, Jun Yang, Yixue Hao, Shiwen Mao, and Kai
symptoms, their preferences for a particular doctor, Hwang. 2017. “A 5G Cognitive System for Healthcare.”
demographic data and so on. The proposed system is Big Data and Cognitive Computing 1 (1): 2.
Cheung, K. L., D. Durusu, X. Sui, and H. de Vries. 2019. “How
easy to use and devoid of any complex information Recommender Systems Could Support and Enhance
retrieval process. Naive user can use it with utmost Computer-Tailored Digital Health Programs: A Scoping
ease. The integration of map with the system will help Review.” Digital Health 5. 2055207618824727.
patient to locate doctor in quick time with accuracy Duan, Lian, W. Nick Street, and Der-Fa Lu. 2008. “A Nursing
The proposed system was evaluated by domain experts Care Plan Recommended System Using a Data Mining
Approach.” In Proceedings of the 3rd INFORMS Workshop
to ascertain its effectiveness. The results show that our
on Data Mining and Health Informatics, edited by J. Li, D.
proposed system solves the problem of locating a reliable Aleman, & R. Sikora, 1–6. Washington, DC.
doctor efficiently. Glover, Lacie. 2014. “Are Online Physician Ratings Any
In future, to obtain more information on doctors, this Good?” U.S.News, December 19. http://health.usnews.
system could include patients’ reviews and ratings of com/health-news/patient-advice/articles/2014/12/19/are-
doctors from social media; it would help to improve online-physician-ratings-any-good.
Guo, L., Bo Jin, Y. Cuili, Y. Haoyu, D. Huang, and F. Wang.
the quality of the system. The system could be further 2017. “Which Doctorto Trust: A Recommender System
improved by adding patients’ treatments and the symp- for Identifying the Right Doctors.” Journal of Medical
toms of a particular disease. The proposed system can be Internet Research. doi:10.2196/jmir.6015.
integrated into any existing hospital management sys- Haughom, John. 2016. “Knowledge Management in
tem, and this could assist patient to find right doctor Healthcare: It’s More Important Than You Realize.”
Health Catalyst, July. Accessed August 12, 2016. www.
in a health critical situation.
healthcatalyst.com/enable-knowledge-management-in-
healthcare.
Hoens, T. Ryan, M. Blanton, and Nitesh V. Chawla. 2013.
Disclosure statement “Reliable Medical Recommendation Systems with Patient
Privacy.” Journal of Medical Internet Research 4: 67.
No potential conflict of interest was reported by the authors. Huang, Y., P. Liu, Q. Pan, and J. S. Lin. 2012. “A Doctor
Recommendation Algorithm Based on Doctor
Performances and Patient Preferences.” In Proceedings of
the Wavelet Active Media Technology and Information
ORCID Processing. doi:10.1109/ICWAMTIP.2012.6413447.
Hussein, Asmaa S., Wail M. Omar, Xue Li, and Modafar Ati.
Naif Radi Aljohani http://orcid.org/0000-0001-9153-1293
2013. “Efficient Chronic Disease Diagnosis Prediction and
Recommendation System.” Biomedical Engineering and
Sciences (IECBES). doi:10.1109/IECBES.2012.6498117.
References Investopedia. 2016. Accessed 12 August, 2016. www.
investopedia.com/terms/w/weightedaverage.asp.
Alluhaidan, A. 2013. “Recommender System Using Kaur, Harmanjeet, Neeraj Kumar, and Shalini Batra. 2018. “An
Collaborative Filtering Algorithm.” School of Computing Efficient Multi-Party Scheme for Privacy Preserving
and Information Systems 10 (4): 155. Collaborative Filtering for Healthcare Recommender
Andale. 2014. “Statistics How To.” Statistics How To, System.” Future Generation Computer Systems 86: 297–307.
December 10. Accessed 25 September, 2017. http://www. Lamber, P., A. Girardello, F. Ricci, and M. Mitterer. 2009.
statisticshowto.com/trimmed-mean/. “MobiDay: Apersonalized Context-Aware Mobile Service
Badrul, S., G. Karypis, J. Konstan, and J. Riedl. 2001. Item- for Day Hospital Workflow Support.” In Proceeding of the
Based Collaborative Filtering Recommendation Algorithms. AIME09 International Workshop on Personalization for e-
Minneapolis, MN and Hong Kong: Department of Health, 15–19. Italy: Proceedings of the AIME09
Computer Science and Engineering. 5545In WWW10 International Workshop.
Balabanović, Marko. 1997. “An Adaptive Web Page Masood, Isma, Yongli Wang, Ali Daud, Naif Radi Aljohani,
Recommendation Service.” AGENTS ‘97 Proceedings of 1st and Hassan Dawood. 2018. “Privacy Management of
International conference on Autonomous agents, 5–8 Feb, Patient Physiological Parameters.” Telematics and
378–385. doi:10.1145/267658.267744. Informatics 35 (4): 677–701.
Best survey software. 2017. Accessed September 2, 2017. Melville, P., and V. Sindhwani. 2013. Recommender Systems.
https://www.surveysystem.com/sscalc.htm. Yorktown Heights, NY: IBM T.J. Watson Research
Blue, J. and OPTUM Inc. 2018. Healthcare Similarity Engine. Center. [email protected].
U.S. Patent Application 10/127,359. Mika, Stefanie. 2011. Challenges for Nutrition Recommender
Ceyhan, Migena, Z. Orhan, and E. Domnori. 2017. “e-Medical Systems. Erlangen: Department of Computer Science, 8
Test Recommendation System Based on the Analysis of Friedrich-Alexander-University Erlangen-Nuremberg
Patients’ Symptoms and Anamneses.” Part of the IFMBE Haberstraße 2, D-91058.
Proceedings book series (IFMBE, volume 62). March 15, Narducci, F., C. Musto, M. Polignano, M. de Gemmis, P. Lops,
2017. and G. Semeraro. 2015. “A Recommender System for
 BEHAVIOUR & INFORMATION TECHNOLOGY 15

Connecting Patients to the Right Doctors in the Health Net System.” Lecture Notes on Software Engineering 1 (3): 303–
Social Network.” In Proceedings of the 24th International 307.
Conference on World Wide Web, 81–82. doi:10.1145/ Salunke, Archana, B. and Kasar, and L. Smita. 2015.
2740908.2742748. “Personalized Recommendation System for Medical
Park, D. H., H. K. Kim, I. Y. Choi, and J. K. Kim. 2012. “A Literature Assistance Using Hybrid Filtering.” International Journal
Review and Classification of Recommender Systems Research.” of Computer Applications 128 (9): 6–10.
Expert Systems with Applications 39 (11): 10059–10072. Schafer, H., S. Hor-Fraile, and R. Pawan. 2017. “Towards
Portugal, Ivens, Paulo Alencar, and Donald Cowan. 2018. “The Health (Aware) Recommender Systems.” In Proceedings of
Use of Machine Learning Algorithms in Recommender dh’17, 157–161. London: Proceedings of the 2017 inter-
Systems: A Systematic Review.” Expert Systems with national conference on digital health ACM.
Applications 97: 205–227. Stockemer, Daniel. 2019. “Conducting a Survey.” In
Qian, Y., Y. Zhang, X. Ma, H. Yu, and L. Peng. 2019. “EARS: Quantitative Methods for the Social Sciences, 57–71.
Emotion-Aware Recommender System Based on Hybrid Cham: Springer.
Information Fusion.” Information Fusion 46: 141–146. Tabari, Mahdiyeh Yousefi, and Azizollah Memariani. 2019.
Rawat, Bhupesh, and Sanjay K. Dwivedi. 2019. “Selecting “Developing a Decision Support System for Big Data
Appropriate Metrics for Evaluation of Recommender Analysis and Cost Allocation in National Healthcare.” In
Systems.” InternationalJournal of Information Technology Healthcare Data Analytics and Management, 89–109.
and Computer Science 1: 14–23. Academic Press.
Saaty, Thomas L. 2013. “Analytic Hierarchy Process.” In Wiesner, Martin, and Daniel Pfeifer. 2010. “Adapting
Encyclopedia of Operations Research and Management Recommender Systems to the Requirements of Personal
Science, 52–64. Boston, MA: Springer. Health Record Systems.” In Proceedings of the 1st ACM
Safoury, L., and A. Salah. 2013. “Exploiting User Demographic International Health Informatics Symposium, 410–414.
Attributes for Solving Cold-Start Problem in Recommender doi:10.1145/1882992.1883053.