European Heart Journal - Digital Health (2022) 3, 415–425 REVIEW

https://doi.org/10.1093/ehjdh/ztac035

Current state of artificial intelligence-based

algorithms for hospital admission prediction in
patients with heart failure: a scoping review
P. M. Croon 1,*, J. L. Selder1, C. P. Allaart 1, H. Bleijendaal1,2,

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

S. A. J. Chamuleau1, L. Hofstra1, I. Išgum3,4, K. A. Ziesemer5, and M. M. Winter1
1
Heart Center, Department of Clinical and Experimental Cardiology, Amsterdam Cardiovascular Sciences, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands;
2
Department of Clinical Epidemiology, Biostatistics & Bioinformatics, Amsterdam UMC, University of Amsterdam, Meibergdreef 9, Amsterdam, The Netherlands; 3Department of
Biomedical Engineering and Physics, Amsterdam University Medical Centers-location AMC, University of Amsterdam, Amsterdam, The Netherlands; 4Department of Radiology and
Nuclear Medicine, Amsterdam University Medical Centers - Location AMC, University of Amsterdam, Amsterdam, The Netherlands; and 5Medical Library, Vrije Universiteit,
Amsterdam, The Netherlands

Received 5 January 2022; revised 20 May 2022; accepted 31 May 2022; online publish-ahead-of-print 24 June 2022

Aims Patients with congestive heart failure (HF) are prone to clinical deterioration leading to hospital admissions,
burdening both patients and the healthcare system. Predicting hospital admission in this patient group could enable
timely intervention, with subsequent reduction of these admissions. To date, hospital admission prediction remains
challenging. Increasing amounts of acquired data and development of artificial intelligence (AI) technology allow for
the creation of reliable hospital prediction algorithms for HF patients. This scoping review describes the current
literature on strategies and performance of AI-based algorithms for prediction of hospital admission in patients
with HF.
.........................................................................................................................................................................................
Methods and PubMed, EMBASE, and the Web of Science were used to search for articles using machine learning (ML) and deep learn-
results ing methods to predict hospitalization in patients with HF. After eligibility screening, 23 articles were included. Sixteen
articles predicted 30-day hospital (re-)admission resulting in an area under the curve (AUC) ranging from 0.61 to 0.79. Six
studies predicted hospital admission over longer time periods ranging from 6 months to 3 years, with AUC’s ranging
from 0.65 to 0.78. One study prospectively evaluated performance of a disposable sensory patch at home after hospi-
talization which resulted in an AUC of 0.89 for unplanned hospital admission prediction.
.........................................................................................................................................................................................
Conclusion AI has the potential to enable prediction of hospital admission in HF patients. Improvement of data management, adding
new data sources such as telemonitoring data and ML models and prospective and external validation of current models
must be performed before clinical applicability is possible.
------------------------------------------------------------------------------------------------------------------------------------------------------------

* Corresponding author. Tel: +31646123217, Email: [email protected]

© The Author(s) 2022. Published by Oxford University Press on behalf of the European Society of Cardiology.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (https://creativecommons.org/licenses/by-nc/4.0/), which
permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact
[email protected]
416 P.M. Croon et al.

Graphical Abstract

EHR Data
Results are impossible to
accurately compare due to
Machine learning Sixteen articles reported heterogeneous methods in
30 day (re)admission terms of data and
(AUC 0.61−0.79) algorithms.

Administrative data
AI methods using
administrative or EHR data

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

Six articles reported achieved moderate
long term (re)admission performance; telemonitoring
(AUC 0.61−0.79) achieved good performance.

Deep learning

Telemonitoring Data
EHR data generally
One prospective
outperformed administrative
telemonitoring study using
data in terms of AUC.
a multi sensory patch
(AUC 0.89)

.........................................................................................................................................................................................
Keywords Hospital admission prediction • Artificial intelligence • Machine learning • Heart failure • eHealth • Preventive
health

telemonitoring data generally do not perform well enough for clinical

Introduction use mainly because they result in many false positive alerts.11–15
Heart failure (HF) is a complex chronic clinical syndrome affect- Invasive devices perform better but are expensive and prone to
ing at least 36 million people worldwide.1 Owing to the aging complications.16
population and improved treatment strategies its prevalence is AI-driven methods have shown great potential in many fields in
increasing.2 Patients suffering from HF can unexpectedly deteri- healthcare including precision medicine and diagnosis, as they enable
orate leading to hospital admissions that burden both patients the interpretation of datasets too large or complex for human inter-
and the healthcare system.2,3 Predicting hospital admission could pretation and have shown to outperform conventional ap-
lead to timely intervention, with a subsequent reduction of clin- proaches.17–19 Artificial intelligence-based analysis has the potential
ical deterioration and hospital admissions.4 These predictions re- to create preventive and predictive care for patients suffering from
main challenging. Artificial intelligence’s (AI’s) capability to learn HF allowing intervention before symptomatic deterioration and
and recognize patterns from large and complex data sets could thus possibly preventing costly hospital admission.20 However, be-
enable creating trustworthy hospital admission prediction algo- fore broad and successful implementation of such analysis methods
rithms.5–7 becomes possible many hurdles including technical, ethical and legal
Before the AI era, several methods have been described to predict difficulties must be administered.
hospital admission in patients with HF. Non-AI algorithms were devel- In this scoping review, we describe the current literature on strat-
oped using readily available variables derived in the hospital or at egies and performance of AI-based algorithms for prediction of hos-
home using telemonitoring.8–10 Performance of algorithms using hos- pital admission in patients with HF. Finally, we give, our view
pital derived data remains weak, with area under ROC curve (AUC) regarding recommendations for future research on the use of AI pre-
between 0.6 and 0.7.10 Telemonitoring enables the ability to acquire diction algorithms in HF.
many data points from a single patient at home, adding valuable infor-
mation to data acquired during the sparse visits to the (outpatient)
clinic.11 Even though telemonitoring has made important early steps
Methods
enabling home monitoring at low cost for large patient groups, previ- This systematic review was guided by the PRISMA extension for scoping
ously proposed algorithms based on non-invasively derived reviews.21
AI-based algorithms 417

Table 1 Full search query in PubMed

Search PubMed Query August 3, 2021 Results

..................................................................................................................................................................................
#3 #1 AND #2 870
#2 “Artificial Intelligence”[Mesh] OR “Artificial Intelligence*”[tiab] OR “AI”[tiab] OR “computational Intelligen*”[tiab] OR “machine
intelligen*”[tiab] OR “deep learning”[tiab]
#1 “Heart Failure”[Mesh] OR “heart failure*”[tiab] OR “cardiac failure*”[tiab] OR “heart decompensat*”[tiab] OR “myocardial 242 172
failure*”[tiab] OR “cardiac decompensat*”[tiab] OR “myocardial decompensat*”[tiab] OR “HFREF”[tiab] OR “diastolic
dysfunction*”[tiab] OR “systolic dysfunction*”[tiab]

Search and selection method algorithm was extracted together with LR for comparison. All results

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

Three bibliographic databases (PubMed, Embase.com and Clarivate were described in a table and the best performing and largest cohort
Analytics/Web of Science Core Collection) were searched for relevant lit- were described in text. Results were grouped in the following subgroups:
erature until August 3, 2021 (Table 1). Searches were devised in collabor- 30-day hospital admission prediction, longer term hospital admission
ation with a medical information specialist (K.A.Z.). Search terms including prediction and hospital admission prediction using telemonitoring.
synonyms, closely related words, and keywords were used as free-text Meta-analysis of the selected studies was not possible due to the hetero-
words: ‘artificial intelligence’ and ‘heart failure’. The searches contained geneity of the included studies and therefore qualitative assessment of
no date or language restrictions that would limit results to specific studies. the different algorithms predictive performance was conducted.
Articles were considered eligible if they reported retrospective or
prospective data on hospital admission prediction in patients with HF
as defined by the European Society of Cardiology (ESC) using AI. Results
Traditional statistical models such as logistic regression (LR) is not
The literature search generated a total of 2293 references: 871 in
included except if used as comparison to ML or data acquisition includes
ML such as natural language processing(NLP). Unsupervised learning is PubMed, 892 in Embase.com and 530 in Clarivate Analytics/Web
mainly used for the identification of patterns in unlabelled data. of Science Core Collection. After removing duplicates of references
Clustering of these patterns might correlate with increased or decreased that were selected from more than one database, 1562 references
risk of adverse events, which is not in the scope of this review. remained. After screening titles and abstracts using ASReview 34 ar-
Screening for eligibility was conducted by two individual reviewers (PC ticles remained for full-text screening. Five articles were excluded for
and HB). For title and abstract screening both reviewers used ASReview the use of unsupervised learning, five for not including hospital admis-
(University Utrecht 2021). In short, ASReview uses active learning, a type sion as outcome and one for an experimental setup testing many dif-
of machine learning (ML), to help researchers efficiently screen title and ferent sub groups, models and outcomes making it impossible to
abstracts accelerating the process in a time with a huge information over- extract the right information, which resulted in 23 articles included
load were searches might exceed multiple thousands of results and has
for analysis after full-text screening (Table 2).23–45
proven to be an reliable asset in one to one comparisons.22 An extensive
overview of the selection process using ASReview is provided in the sup-
plementary methods.
Of all included articles, year of publication, study design, number of pa-
Thirty-day hospital readmission
tients, outcome, data source, feature types, used algorithm and AUC was prediction
extracted. In addition, if reported, the confidence interval (CI), standard
deviation (SD), specificity, sensitivity, true positive value (TPV), true nega- Sixteen articles reported performance of AI algorithms for 30-day all-
tive value (TNV), accuracy, precision, recall, and F-score were collected cause and/or HF hospital readmission prediction with AUC ranging
and reported. In case multiple algorithms were used, the best performing from 0.61 up to 0.79. Results from data extraction are summarized in

Table 2 Overview of included articles

Year Authors Title Journal Data and code

availability
..................................................................................................................................................................................
2000 Atienza, F., Risk stratification in heart failure using artificial neural networks Proc AMIA Symp Code and data not
Martinez-Alzamora, N., available
et al.43
2016 Mortazavi, B. J., Downing, Analysis of Machine Learning Techniques for Heart Failure Circ Cardiovasc Code and data not
N. S., et al.32 Readmissions Qual Outcomes available
2016 Turgeman, L., May, J. H.41 A mixed-ensemble model for hospital readmission Artif Intell Med Code and data not
available

Continued
418 P.M. Croon et al.

Table 2 Continued

Year Authors Title Journal Data and code

availability
..................................................................................................................................................................................
2017 Frizzell, J. D., Liang, L., Prediction of 30-Day All-Cause Readmissions in Patients JAMA Cardiol Code and data not
et al.29 Hospitalized for Heart Failure: Comparison of Machine Learning available
and Other Statistical Approaches
2017 Shameer, K., Johnson, Predictive modelling of hospital readmission rates using electronic Pac Symp Code and data not
K. W., et al.31 medical record-wide machine learning: a case-study using Biocomput available
mount sinai heart failure cohort
2018 Golas, S. B., Shibahara, T., A machine learning model to predict the risk of 30-day BMC Med Inform Data available upon
et al.27 readmissions in patients with heart failure: a retrospective Decis Mak request, code not

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

analysis of electronic medical records data available
2018 Xiao, C., Ma, T., et al.30 Readmission prediction via deep contextual embedding of clinical PLoS One Code available, data not
concepts available
2018 Mahajan, S. M., Mahajan, Predicting Risk of 30-Day Readmissions Using Two Emerging Stud Health Code and data not
A. S., et al.34 Machine Learning Methods Technol Inform available
2018 Larburu, N., Artetxe, A., Artificial Intelligence to Prevent Mobile Heart Failure Patients Mobile Information Code and data not
et al.45 Decompensation in Real Time: Monitoring-Based Predictive Systems available
Model
2019 Allam, A., Nagy, M., et al.23 Neural networks vs. Logistic regression for 30 days all-cause Sci Rep Code and data publicly
readmission prediction available
2019 Ashfaq, A., Sant’Anna, A., Readmission prediction using deep learning on electronic health J Biomed Inform Saple of the data and
et al.25 records code available
2019 Awan, S. E., Bennamoun, M., Machine learning-based prediction of heart failure readmission or ESC Heart Fail Data available after
et al.28 death: implications of choosing the right model and the right approval, code
metrics available
2019 Awan, S. E., Bennamoun, M., Feature selection and transformation by machine learning reduce PLoS One Code available, data
et al.26 variable numbers and improve prediction for heart failure available upon
readmission or death request
2019 McKinley, D., Impact of a Pharmacist-Led Intervention on 30-Day Readmission Am J Mens Health Code and data not
Moye-Dickerson, et al.35 and Assessment of Factors Predictive of Readmission in African available
American Men With Heart Failure
2019 Mahajan, S. M., Ghani, R.38 Using Ensemble Machine Learning Methods for Predicting Risk of Stud Health Code and data not
Readmission for Heart Failure Technol Inform available
2019 Mahajan, S. M., Ghani, R.33 Combining Structured and Unstructured Data for Predicting Risk Stud Health Code and data not
of Readmission for Heart Failure Patients Technol Inform available
2020 Angraal, S., Mortazavi, B. J., Machine Learning Prediction of Mortality and Hospitalization in JACC Heart Fail Code and data not
et al.24 Heart Failure With Preserved Ejection Fraction available
2020 Chu, J., Dong, W., et al.36 Endpoint prediction of heart failure using electronic health records J Biomed Inform Code available, data not
available
2020 Chen, P., Dong, W., et al.37 Interpretable clinical prediction via attention-based neural BMC Med Inform Code not available, data
network Decis Mak available upon
request
2020 Desai, R. J., Wang, S. V., Comparison of Machine Learning Methods With Traditional JAMA Netw Open Code and data not
et al.39 Models for Use of Administrative Claims With Electronic available
Medical Records to Predict Heart Failure Outcomes
2020 Stehlik, J., Schmalfuss, C., Continuous Wearable Monitoring Analytics Predict Heart Failure Circ Heart Fail Code and data not
et al.44 Hospitalization: The LINK-HF Multicentre Study available
2021 Lewis, M., Elad, G., Beladev, Comparison of deep learning with traditional models to predict Sci Rep Code available, data on
M., et al.40 preventable acute care use and spending among heart failure request
patients
2021 Ben-Assuli, O., Heart, T., Profiling Readmissions Using Hidden Markov Model-the Case of Information Code and data not
et al. 42 Congestive Heart Failure Systems available
Management
Table 3 Results for 30-day prediction models

Year Authors Design No. Outcome Data source Data used Model used AUC Performance metrics
patients (30-day reported
readmission)
..................................................................................................................................................................................................................................................
AI-based algorithms

2016 Mortazavi, B. J., Retrospective 977 HF EHR + 472 features including clinical data, Random forest 0.68 (CI: PPV 0.22, Sens 61%, Spec
Downing, N. S., Telemonitoring physical examination, laboratory, 0.670– 61%, F-score 0.32
et al.32 demographics, socio-economics 0.678)
2016 Turgeman, L., May, Retrospective 4840 All-cause EHR Demographics, vitals, laboratory, C5 SVM ensemble 0.7 TH 0.7 Sens 26%, spec
J. H.41 comorbidities 91%, PPV 0.260, NPV
0.911, F1 0.261
2017 Frizzell, J. D., Liang, L., Retrospective 56477 HF Insurance Demographics, socio-economic status, Tree augmented 0.62
et al.29 medical history, medication, vital signs, naive bayesion
laboratory, interventions network
2017 Shameer, K., Johnson, Retrospective 1068 All-cause EHR ICD-codes, medication, laboratory, vital Naive Bayes 0.78 Accuracy 83,19%
K. W., et al.31 signs, procedures
2018 Xiao, C., Ma, T., et al.30 Retrospective 5393 All-cause EHR Comorbidity, laboratory, medication, Gated recurrent 0.61 (SD PR-AUC 0.39, accuracy
unit RNN 0.0153) 0.6934
2018 Golas, S. B., Shibahara, Retrospective 11510 All-cause EHR Demographic and clinical data Deep Unified 0.71 Accuracy 0.646, Precision
T., et al.27 (comorbidity, loboratory, medication, Network 0.360, recall 0.652, f1
procedures), notes 0.464
2018 Mahajan, S. M., Retrospective 1778 All-cause EHR Laboratory, vitals, demographics Boosted trees 0.719
Mahajan, A. S.,
et al.34
2019 Allam, A., Nagy, M., Retrospective 272.778 All-cause National database socio-demographics (age/gender), RNN with 0.642 (CI TPV 0.57, FPV 0.37
et al.23 Pay-source, hospital adission events, conditional 0.640–
diagnosis, procedures random fields 0.645)
2019 Awan, S. E., Retrospective 10757 HF Nation-wide demographics, admission characteristics, Multi-layer 0.66 Sens 56%, spec 66%
Bennamoun, M., system(EHR, GP medical history, socio-economics, perception
et al. etc) medication history
2019 Mahajan, S. M., Ghani, retrospective 1629 All-cause EHR Laboratory, vitals, demographics, clinical NLP for 0.65(CI
R. notes unstructured 0.59-0.71)
data, LR for
prediction
2019 Awan, S. E., Retrospective 10757 HF Nation-wide system demographics, admission characteristics, Multi-layer 0.63 Sens 48%, spec 70%
Bennamoun, M., (EHR, GP, and so medical history, socio-economics, perception
et al. on) medication history
2019 Mahajan, S. M., Ghani, Retrospective 36245 All-cause EHR Demographics, vitals, laboratory, ExtraTrees 0.70 (CI F1 score 0.58
R. comorbidities 0.69–
0.71)

Continued
419

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

420 P.M. Croon et al.

Table 3. Ben-Assuli et al.42 reported the highest AUC with 0.79, using a

Accuracy 0.77, FPV 0.214,

..................................................................................................................................................................................................................................................
Performance metrics
Hidden Markov Model on data on 4661 adult HF patients with at least
five in- or outpatient clinic visits. Thirty-eight time-varying covariates
reported

were retrospectively derived from 6 hospital information systems in-

Accuracy 0.89
F1 score 0.51 cluding a comprehensive electronic health record (EHR), laboratory

FNR 0.250
information management system and picture archiving and communica-
tion system (PACS) for imaging results. The algorithm’s performance
enhanced when data from more clinical visits were present, resulting
in an AUC of 0.79 after four visits for all patients. Patients were divided
AUC

0.77 (SD:

in three states depending on readmission risk. The model performed

0.005)

K nearest neighbour 0.768

worst in the most vulnerable group and best in the medium risk group
0.79 with an AUC after four visits of 0.65 and 0.77, respectively.42 Shameer
et al. reported an algorithm with similar performance, using retrospect-

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

Model used

RNN with LSTM

ive EHR data of 1068 HF patients, in whom more than 4000 features
Hidden Markov

were extracted from the EHR including diagnosis codes, medication, la-
Model

boratory measurements and vital signs. Correlation-based feature selec-

tion of EHR data was performed and a naïve bayes classifier for
readmission prediction which resulted in an AUC of 0.78.31
Medications yielded the most predictive value. The study with the lar-
Demographics, comorbidity, admission

Demographic, social, and clinical data

laboratory results, ICD-codes and

gest patient group was by Allam et al. who included 273 000 patients
type, medication, all procedures,

Demographics, laboratory, medical

in their analysis retrieved of an administrative insurance claims dataset

including socio-demographics, hospital admission event, comorbidity,
Data used

and diagnosis. The authors best performing deep learning (DL) algo-
imaging, comorbidity,

rithm utilized a recurrent neural network (RNN) combined with condi-

tional random fields, which resulted in an AUC for 30-day readmission
prediction of 0.64.23 Penalized logistic regression (LASSO) had a similar
AUC of 0.64 (Table 4). Prediction algorithms based on EHR data gener-
more

ally outperform algorithms based on data from national databases or in-

surance data and only EHR-based algorithms achieved moderate
performance (Table 3).
Data source

Long-term hospital admission prediction

In total, six studies predicted hospital (re)admission for a time period
EHR

EHR

EHR

longer than 30 days (Table 5). Five studies used retrospective data
and one used prospective data. Outcomes ranged from 6-month re-
readmission)
Outcome

admission to 3-year readmission. All but one reported AUC ranging

(30-day

from 0.65 to 0.78. The best performing two algorithms both re-
All-cause

All-cause

ported an AUC of 0.78. Lewis et al.40 used insurance claims retro-

spective data of 92 000 patients, divided in knowledge-driven and
HF

data-driven features using NLP, for predicting 6-month all-cause ad-

patients

mission. The authors reported that DL networks, a convolutional

No.

132
7655

4661

EHR, electronic health record; PPV, positive predictive value.

neural network with LSTM in this case, outperforms traditional ML

models (feed forward neural network and LR). Moreover, sequential
2019 Ashfaq, A., Sant’Anna, Retrospective

Retrospective

2021 Ben-Assuli, O., Heart, Retrospective

DL models outperformed non-sequential DL models. The second

Design

algorithm-derived data from both EHR and insurance claims predict-

ing 1-year HF hospitalization.39 A total of 54 variables were collected
from the claims including demographics, no. of hospitalizations, medi-
Moye-Dickerson, P.,

cation and comorbidity, and eight variables from the EMR laboratory
and left ventricular EF. The authors utilized multiple different models
Table 3 Continued

Authors

of which a gradient boosted model (GBM) performed best with an

2019 McKinley, D.,

AUC of 0.78 (compared with 0.75 when only claims data was
A., et al.

T., et al.
et al.

used). The longest prediction period (3 years) utilized a random for-

est classifier resulting in an AUC of 0.76 for admission prediction.24
This result was achieved using data from a trial including patients with
Year

HF with preserved ejection fraction including of baseline demograph-

ic and clinical data, ECG, laboratory data, and questionnaires.
AI-based algorithms 421

Table 4 Performance of logistic regression versus artificial intelligence

Year Authors Model used AUC AI AUC LR

..................................................................................................................................................................................
2016 Mortazavi, B. J., Downing, N. S., et al. 32 Random forest 0.68 (CI: 0.670–0.687) 0.54 (CI: 0.563–0.550)
2016 Turgeman, L., May, J. H. 41 C5 ensemble 0.70 0.70
2017 Frizzell, J. D., Liang, L., et al. 29 Tree augmented naive bayesion network 0.62 0.62
2018 Xiao, C., Ma, T., et al.30 Gated recurrent unit RNN 0.61 (SD: 0.013) 0.59 (SD: 0.012)
2018 Mahajan, S. M., Mahajan, A. S., et al.34 Boosted trees 0.72 0.62
2018 Golas, S. B., Shibahara, T., et al.27 Deep Unified Network 0.70 0.66
2019 Awan, S. E., Bennamoun, M., et al.26 Multi-layer perception 0.63 0.55
2019 Allam, A., Nagy, M., et al.23 RNN with conditional random fields 0.64 (SD: 0.0027) 0.64 (SD: 0.0028)
2019 Mahajan, S. M., Ghani, R.38 ExtraTrees 0.70 0.70

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

2020 Chen, P., Dong, W., et al.37 Attention-based model 0.69 (SD: 0.047) 0.68 (SD: 0.039)
2020 Angraal, S., Mortazavi, B. J., et al.24 Random forest 0.76 (CI: 0.71–0.77) 0.73 (CI: 0.67–0.79)
2020 Desai, R. J., Wang, S. V., et al.39 GBM 0.78 (CI: 0.753–0.802) 0.74 (CI: 0.711–0.766)
2021 Lewis, M., Elad, G., et al.40 Sequential deep learning 0.78 (CI: 0.784–0.790) 0.75 (CI: 0.744–0.751)

Telemonitoring for hospitalization Although some AI-models seem promising, only the AUC was re-
ported in all studies and additional performance metrics were only
prediction
sometimes reported. A more detailed evaluation is needed to pro-
Three studies used AI algorithms on telemonitoring data for hospital
vide comprehensive insight in the performance of a model.
admission prediction (Table 6). Stehlik et al.44 monitored one hun-
Machine learning algorithms are prone to overfitting, meaning the al-
dred subjects prospectively at home for 3 months using a disposable
gorithm is trained extensively within a population. As a consequence,
multisensory patch monitoring vital parameters including ECG (ar-
this can result in a model that performs well within this population,
rhythmia burden, heart rate variability), skin impedance, tempera-
but not outside of this population, making generalizability of the re-
ture, and so on. The data acquired by the device was continuously
sults questionable. Moreover, many articles proposed a novel meth-
uploaded through a smartphone. Patients were instructed to wear
od and compared this with existing methods. We chose to only
the device 24 h a day. After initial placement of the device, baseline
report the best performing algorithm which always is the proposed
patterns in the data are identified. When this baseline is finalized,
method. However, it is questionable if the same amount of feature
the device switched to surveillance mode that compared temporal
engineering and/or hyperparameter optimization was performed
data to the baseline patterns using similarity-based modelling. This
on the compared method as to the proposed method, making the
approach resulted in an AUC of 0.89 for HF readmission and 0.84
best performing method more prone to overfitting. Overfitting
for unplanned non-trauma hospital admission. Sensitivity for HF hos-
can be minimized by using appropriate validation methods such as
pitalization was 86% at a specificity of 87.5%. The remaining two
nested cross validation and tested with external or prospective val-
studies used retrospective data both derived from clinical trials in
idation.46 The risk of overfitting is a major limitation of all included
which eHealth was used,32,45 which resulted in and AUC of 0.68
articles.
and 0.67 utilizing a random forest and Naïve Bayes, respectively.
Hospital (re-)admission of HF patients is a major problem re-
sulting in high patient disease burden and health care expenses.2,3
Discussion One proven effective strategy for reducing hospital admission in
this patient group is through signalling pre-clinical deterioration
This scoping review evaluated the current performance of AI-based using invasive pulmonary artery pressure home monitoring, which
hospital (re-)admission prediction algorithms based on EHR, admin- yielded a 37% reduction in hospitalization.4,16 This indicates that
istrative and telemonitoring data, and resulted in the following main detection of early, pre-clinical, deterioration can indeed result in
findings. (i) Multiple different classifiers were proposed all using dif- a reduction of hospital admissions. Randomized controlled trials
ferent sets of data to predict hospital admission for patients with on the effect of well performing, prospectively validated,
HF making it challenging to accurately compare results. (ii) AI meth- AI-based hospital admission prediction reduction are needed to
ods using EHR or administrative data achieved moderate perform- further evaluate if and to what extend early deterioration predic-
ance, one AI method using telemonitoring data achieved good tion leads to prevention of hospital (re-)admission and subse-
performance; (iii) EHR data generally performed better in compari- quent disease burden and cost.
son to administrative data or research data which is consistent Similar moderate performance of AI-hospitalization-prediction al-
with clinical expectation since EHR data yields much more compre- gorithms were achieved in non-HF patients. For example, Rajkomar
hensive clinical information; and (iv) almost all studies were per- et al.47 utilized a deep neural network (DNN) in a cohort of 216,221
formed on retrospective data without external or prospective adult patients hospitalized for more than 24 h for any cause and re-
validation. ported an AUC of 0.75–0.76 for 30-day hospital readmission using
Table 5 Results for long-term hospital admission prediction
422

Year Authors n. patients Design Outcome Data source Data used Model used AUC
..................................................................................................................................................................................................................................................
2016 Mortazavi, B. J., 977 HF patients Retrospective 180-day HF readmission EHR + 472 features including clinical data, Random forest in to SVM 0.657 (CI: 0.652– PPV 0.51, Sens 0.95, spec
Downing, N. S., et al. Telemonitoring physical examination, 0.661) 0.15, F-score 0.66
laboratory, demographics,
socio-economics
2020 Chen, P., Dong, W., 736 HF patients Retrospective 1-year all-cause readmission EHR 105 features including Attention-based model 0.691 (SD: 0.047) Acc 0.667, Precision
et al. demographics, vital signs, 0.710, Recall 0.795, F1
laboratory, echocardiography, 0.749
comorbidities, medication
2020 Angraal, S., Mortazavi, 1767 HFpEF patients Retrospective 3-year HF admission EHR Demographics, clinical data, ECG, Random forest 0.76 (CI: 0.71–0.85)
B. J., et al. laboratory, questionnaires
2020 Desai, R. J., Wang, S. V., 9502 HF patients Prospective 1-year HF admission EHR + Insurance 54 variables including ICD score, GBM 0.78 (CI: 0.753–
et al. ICD/CRT-D, HF related 0.802)
medication, frailty index,
laboratory, BMI, ejection
fraction
2021 Lewis, M., Elad, G., 92260 HF patients Retrospective 6-month all-cause admission, Insurance demographics, episode counts and Sequential deep learning 0.78 (CI: 0.784–0.79)
Beladev, M., et al. 6-month ED admission trends, hospital length of stay,
readmission rates, costs,
comorbidity, procedures,
medication, machine derived
features using NLP.
2000 Atienza, F., Martinez- 132 HF patients Retrospective 1-year admission EHR Demographic, clinical, non-invasive Neural Network Accuracy 93.2% sens 81%, Spec 94%
Alzamora, N., et al. laboratory (AUC not
reported)
2020 Chu, J., Dong, W., et al. 25402HF patiënts retrospective HF readmission EHR demographics, medication, RNN for data processing, LR 0.644 (SD: 0.005)
procedures, diagnosis, for endpoint prediction
laboratory, vital signs, physical
examination, etc.,
P.M. Croon et al.

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

AI-based algorithms 423

EHR data including notes. Hilton et al.48 reported an AUC of 0.76 for

..................................................................................................................................................................................................................................................
prediction of 30-day readmission using a DNN trained, validated and

HF hospitalization sens 76, spec 85,

Additional performance

tested on 708 089 any cause hospitalized patients using EHR data.
PPV 0.22, Sens 0.61, Spec 0.61,

All-cause sens 69, spec 85

Interestingly, Morawski et al.49 predicted 6-month all-cause hospital
metrics

admission for all patients aged older than 18 years in the EHR includ-

Sens 0.76, 28.64 FA/py

ing outpatient clinic patients of a large care facility in Massachusetts
F-score 0.32

with an AUC of 0.84. Translating this to our HF population, it might

be possible to predict hospital admission in outpatient clinic patients
which increases the population leading to the prevention of more
hospital admissions than when predicting readmission alone.
One of the questions of using AI for clinical decision support is
All-cause
AUC

whether it is able to outperform traditional statistical methods

0.678 (CI:
0.670–
0.678)

HF 0.86,

0.80

such as LR. A systematic review published in 2019 including

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

0.67

71 studies that compared ML with LR concluded that ML generally

did not result in better performance than LR.50 Risk of bias was
found to be higher in the studies that did report higher AUC
Wearable sensory patch with ECG Similarity-based
472 features including clinical data, Random forest
Model

modelling

Telemonitoring (vital signs, weight, Naive Bayes

for ML in comparison to LR. However, this review used articles

published before 2017. Our results show that ML achieves better
performance in terms of AUC in most studies but confidence in-
tervals overlap in most cases. However, the performance of ML
temperature, activity, posture
arrythmia), skin impendance

(demographics, comorbitity,

increases over time. If this increase of performance will continue

laboratory, demographics,

questionnaires), baseline

in the near future with further advances made in model develop-

(respiratory rate), skin
Data used

(heart rate variability,

physical examination,

ment ML might be able to structurally outperform LR for this pur-

socio-economics

pose (Table 5).

laboratory)

Future perspective
In the future, connecting multiple conventional, readily available (e.g.
Telemonitoring

Telemonitoring
Data source

Wearable sensor

clinical variables, imaging data etc.) and newer data sources (health
trackers, social media behaviour, genomic sequencing) could add to
patch

a bright future by exposing new preventive, diagnostic and thera-

EHR +

HF decompensation (readmission EHR +

peutic advantages. For example, prediction models may not only pre-
Table 6 Results for hospitalization prediction using telemonitoring

dict deterioration but also to predict and thus suggest effect of

30-day HF readmission, 180 day

different treatment strategies further preventing patient deterior-

ation In theory, this would prevent hospitalization, enable persona-
+ home intervention)
Outcome

lized medication advice, and provide insight into who needs an

HF readmission

HF hospitalization

outpatient clinic visits, making it possible to scale down given care un-
til suggested differently. In summary, prediction models could en-
hance scalability and cost-effectiveness of our healthcare system.
In order to get to the point of AI-driven clinical support systems
advising doctors and patients daily and thus transforming current re-
active ‘disease care’ towards proactive personalized and predictive
Retrospective

Retrospective
design

Prospective
study

medicine, substantial development of such models is still required.

To enable effective use of multiple data sources (such as EHR, medi-
cation prescription systems, telemonitoring platforms and more),
‘FACT’ and ‘FAIR’ principles might be important to take in to consid-
patients
No. of

patients

patients

patients

eration. FACT and FAIR are acronyms for ‘Fairness, Accuracy,

977 HF

100 HF

241 HF

Confidentiality, and Transparency’ and ‘Findable, Accessible,

Interoperable, and Re-usable’ respectively.51 Moreover, researchers
and developers must find a way to work with data given that privacy
Stehlik, J., Schmalfuss,

Larburu, N., Artetxe,

Downing, N. S.,
Authors

is guaranteed in accordance with all the regulations. Before broad im-

Mortazavi, B. J.,

C., et al.44

A., et al.45

plementation can be pursued aspects as patient benefit and cost-

et al.32

effectiveness can be evaluated. In addition, a cultural change must

be accomplished meaning both patient and health care provider
will become more familiar with such systems to efficiently change
Year

2018
2016

2020

the current workflow. If this can all be accomplished, AI-based clinical

support systems have the potential to positively change HFcare.
424 P.M. Croon et al.

Limitations 7. Bajwa J, Munir U, Nori A, Williams B. Artificial intelligence in healthcare: transforming

the practice of medicine. Future Healthc J 2021;8:e188–e194.
This review has several limitations. Included studies are extensively 8. Hammill BG, Curtis LH, Fonarow GC, Heidenreich PA, Yancy CW, Peterson ED,
heterogenetic in terms of population, data types, and their represen- Hernandez AF. Incremental value of clinical data beyond claims data in predicting
30-day outcomes after heart failure hospitalization. Circulation Cardiovasc Qual
tation and used model making comparison challenging. Moreover,
Outcomes 2011;4:60–67.
the aim of studies differed between studies. Some aimed to develop 9. Thavendiranathan P, Yingchoncharoen T, Grant A, Seicean S, Landers SH, Gorodeski
an prediction algorithm with state of the art results whereas others EZ, Marwick TH. Prediction of 30-day heart failure-specific readmission risk by echo-
cardiographic parameters. Am J Cardiol 2014;113:335–341.
aimed to compare different technical strategies to evaluate if they can
10. Kansagara D, Englander H, Salanitro A, Kagen D, Theobald C, Freeman M, Kripalani
enhance performance in a small data set. We did not conduct a S. Risk prediction models for hospital readmission: a systematic review. JAMA 2011;
search in Google Scholar which may have led to missed relevant ar- 306:1688–1698.
11. Latifi R. The Modern Hospital - Patiënts centered, disease based, research oriented, tech-
ticles. For eligibility assessment, we used ASReview which is quite
nology driven. Springer; 2019.
new and not yet broadly implemented. However, ASReview is based 12. Drazner MH, Rame JE, Stevenson LW, Dries DL. Prognostic importance of elevated
on active learning which is extensively reviewed and adoption of jugular venous pressure and a third heart sound in patients with heart failure. N Eng J
Med 2001;345:574–581.

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

ML-based selection methods are used more and more. Moreover,
13. Maier SKG, Paule S, Jung W, Koller M, Ventura R, Quesada A, Quesada A, Bordachar
one by one screening by humans is far from perfect leading to 10% P, García-Fernández FJ, Schumacher B, Lobitz N, Takizawa K, Ando K, Adachi K,
missed articles.52 To reduce the chance of relevant missed articles, Shoda M. Evaluation of thoracic impedance trends for implant-based remote mon-
two separate reviewers conducted the full selection protocol which itoring in heart failure patients - Results from the (J-)HomeCARE-II Study. J
Electrocardiol 2019;53:100–108.
did not lead to largely different results. 14. Koole MAC, Kauw D, Winter MM, Dohmen DAJ, Tulevski II, de Haan R, Somsen GA,
Schijven MP, Robbers-Visser D, Mulder BJM, Bouma BJ, Schuuring MJ. First real-
world experience with mobile health telemonitoring in adult patients with congenital
Conclusion heart disease. Neth Heart J 2019;27:30–37.
15. Chaudhry SI, Mattera JA, Curtis JP, Spertus JA, Herrin J, Lin Z, Phillips CO, Hodshon
In conclusion, AI has the potential to play an important role in early BV, Cooper LS, Krumholz HM. Telemonitoring in Patients with Heart Failure. N Engl
J Med 2010;363:2301–2309.
prediction of clinical deterioration in HFpatients. Current AI algo- 16. Abraham WT, Stevenson LW, Bourge RC, Lindenfeld JA, Bauman JG, Adamson PB.
rithms use EHR or telemonitoring data, and resulting in moderate Sustained efficacy of pulmonary artery pressure to guide adjustment of chronic heart
to good prediction models. However, lack of external and prospect- failure therapy: complete follow-up results from the CHAMPION randomised trial.
Lancet 2016;387:453–461.
ive validation, resulting in a high risk of overfitting, is a major limitation
17. Attia ZI, Noseworthy PA, Lopez-Jimenez F, Asirvatham SJ, Deshmukh AJ, Gersh BJ,
to all studies. Prospective studies, with external validation are war- Carter RE, Yao X, Rabinstein AA, Erickson BJ, Kapa S, Friedman PA. An artificial
ranted to improve current models and validate their performance. intelligence-enabled ECG algorithm for the identification of patients with atrial fibril-
lation during sinus rhythm: a retrospective analysis of outcome prediction. Lancet
2019;394:861–867.
Supplementary material 18. Ardila D, Kiraly AP, Bharadwaj S, Choi B, Reicher JJ, Peng L, Tse D, Etemadi M, Ye W,
Corrado G, Naidich DP, Shetty S. End-to-end lung cancer screening with three-
dimensional deep learning on low-dose chest computed tomography. Nat Med
Supplementary material is available at European Heart Journal – Digital 2019;25:954–961.
Health. 19. Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future
Healthc J 2019;6:94–98.
20. Barrett M, Boyne J, Brandts J, Brunner-La Rocca HP, De Maesschalck L, De Wit K,
Funding Dixon Lana, Eurlings C, Fitzsimons D, Golubnitschaja O, Hageman A, Heemskerk
University of Amsterdam Research Priority Agenda Program AI for F, Hintzen A, Helms TM, Hill L, Hoedemakers T, Marx N, McDonald K, Mertens
M, Müller-Wieland D, Palant A, Piesk J, Pomazanskyi A, Ramaekers J, Ruff P,
Heath Decision-Making. Schütt K, Shekhawat Y, Ski CF, Thompson DR, Tsirkin A, van der Mierden K,
Watson C, Zippel-Schultz B. Artificial intelligence supported patient self-care in
Conflict of interest: None declared. chronic heart failure: a paradigm shift from reactive to predictive, preventive and
personalised care. EPMA J 2019;10:445–464.
21. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, Moher D, Peters
Data availability MDJ, Horsley T, Weeks L, Hempel S, Akl EA, Chang C, McGowan J, Stewart L,
No new data were generated or analyzed in this review. Hartling L, Aldcroft A, Wilson MG, Garritty C, Lewin S, Godfrey CM, Macdonald
MT, Langlois EV, Soares-Weiser K, Moriarty J, Clifford T, Tunçalp Ö, Straus SE.
PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation.
References Ann Intern Med 2018;169:467–73.
1. Ponikowski P, Anker SD, AlHabib KF, Cowie MR, Force TL, Hu S, Jaarsma T, Krum H, 22. van de Schoot R, de Bruin J, Schram R, Zahedi P, de Boer J, Weijdema F, Kramer B,
Rastogi V, Rohde LE, Samal UC, Shimokawa H, Budi Siswanto B, Sliwa K, Filippatos G. Huijts M, Hoogerwerf M, Ferdinands G, Harkema A, Willemsen J, Ma Y, Fang Q,
Heart failure: preventing disease and death worldwide. ESC Heart Fail 2014;1:4–25. Hindriks S, Tummers L, Oberski DL. An open source machine learning framework
2. Braunschweig F, Cowie MR, Auricchio A. What are the costs of heart failure? for efficient and transparent systematic reviews. Nat Mach Intell 2021;3:125–133.
Europace 2011;13:ii13–ii17. 23. Allam A, Nagy M, Thoma G, Krauthammer M. Neural networks versus Logistic re-
3. Lesyuk W, Kriza C, Kolominsky-Rabas P. Cost-of-illness studies in heart failure: a sys- gression for 30 days all-cause readmission prediction. Sci Rep 2019;9:9277.
tematic review 2004-2016. BMC Cardiovascu Disord 2018;18:74. 24. Angraal S, Mortazavi BJ, Gupta A, Khera R, Ahmad T, Desai NR, Jacoby DL, Masoudi
4. Abraham WT, Adamson PB, Bourge RC, Aaron MF, Costanzo MR, Stevenson LW, FA, Spertus JA, Krumholz HM. Machine learning prediction of mortality and hospi-
Strickland W, Neelagaru S, Raval N, Krueger S, Weiner S, Shavelle D, Jeffries B, Yadav talization in heart failure with preserved ejection fraction. JACC Heart Fail 2020;8:
JS, CHAMPION Trial Study Group. Wireless pulmonary artery haemodynamic mon- 12–21.
itoring in chronic heart failure: a randomised controlled trial. Lancet 2011:377: 25. Ashfaq A, Sant’Anna A, Lingman M, Nowaczyk S. Readmission prediction using deep
658–666. learning on electronic health records. J Biomed Inform 2019;97:103256.
5. Weber GM. Using artificial intelligence in an intelligent way to improve efficiency of a 26. Awan SE, Bennamoun M, Sohel F, Sanfilippo FM, Chow BJ, Dwivedi G. Feature selec-
heart failure care team. J Cardiac Fail 2018;24:363–364. tion and transformation by machine learning reduce variable numbers and improve
6. Silva-Cardoso J, Juanatey JRG, Comin-Colet J, Sousa JM, Cavalheiro A, Moreira E. The prediction for heart failure readmission or death. PLoS One 2019;14:e0218760.
future of telemedicine in the management of heart failure patients. Cardiac Fail Rev 27. Golas SB, Shibahara T, Agboola S, Otaki H, Sato J, Nakae T, Hisamitsu T, Kojima G,
2021;7:e11. Felsted J, Kakarmath S, Kvedar J, Jethwani K. A machine learning model to predict the
AI-based algorithms 425

risk of 30-day readmissions in patients with heart failure: a retrospective analysis of with electronic medical records to predict heart failure outcomes. JAMA Network
electronic medical records data. BMC Med Inform Decis Mak 2018;18:44. Open 2020;3:e1918962.
28. Awan SE, Bennamoun M, Sohel F, Sanfilippo FM, Dwivedi G. Machine learning-based 40. Lewis M, Elad G, Beladev M, Maor G, Radinsky K, Hermann D, Litani Y, Geller T,
prediction of heart failure readmission or death: implications of choosing the right Pines JM, Shapiro Nl, Figueroa JF. Comparison of deep learning with traditional mod-
model and the right metrics. ESC Heart Fail 2019;6:428–435. els to predict preventable acute care use and spending among heart failure patients.
29. Frizzell JD, Liang L, Schulte PJ, Yancy CW, Heidenreich PA, Hernandez AF, Bhatt DL, Sci Rep 2021;11:1164.
Fonarow GC, Laskey WK. Prediction of 30-day all-cause readmissions in patients 41. Turgeman L, May JH. A mixed-ensemble model for hospital readmission. Artif Intell
hospitalized for heart failure: comparison of machine learning and other statistical Med 2016;72:72–82.
approaches. JAMA Cardiol 2017;2:204–209. 42. Ben-Assuli O, Heart T, Vest JR, Ramon-Gonen R, Shlomo N, Klempfner R. Profiling
30. Xiao C, Ma T, Dieng AB, Blei DM, Wang F. Readmission prediction via deep context- Readmissions Using Hidden Markov Model-the Case of Congestive Heart Failure.
ual embedding of clinical concepts. PLoS One 2018;13:e0195024-e. Inform Sys Manag 2021;38:237–249.
31. Shameer K, Johnson KW, Yahi A, Miotto R, Li LI, Ricks D, Jebakaran J, Kovatch P, 43. Atienza F, Martinez-Alzamora N, De Velasco JA, Dreiseitl S, Ohno-Machado L.
Sengupta PP, Gelijns S, Moskovitz A, Darrow B, David DL, Kasarskis A, Tatonetti Risk stratification in heart failure using artificial neural networks. Proc AMIA Symp
NP, Pinney S, Dudley JT. Predictive modeling of hospital readmission rates using elec- 2000;32:6.
tronic medical record-wide machine learning: a case-study using mount sinai heart 44. Stehlik J, Schmalfuss C, Bozkurt B, Nativi-Nicolau J, Wohlfahrt P, Wegerich S, Rose K
failure cohort. Pac Symp Biocomput 2017;22:276–287. Ray R, Schofield R, Deswal A, Sekaric J, Anand S, Richards D, Hanson H, Pipke M,
32. Mortazavi BJ, Downing NS, Bucholz EM, Dharmarajan K, Manhapra A, Li SX, Pham M. Continuous wearable monitoring analytics predict heart failure hospitaliza-

Downloaded from https://academic.oup.com/ehjdh/article/3/3/415/6617146 by guest on 13 September 2024

Negahban SN, Krumholz HM. Analysis of machine learning techniques for heart fail- tion: The LINK-HF multicenter study. Circ Heart Fail 2020;13:e006513.
ure readmissions. Circulation Cardiovasc Qual Outcomes 2016;9:629–640. 45. Larburu N, Artetxe A, Escolar V, Lozano A, Kerexeta J. Artificial intelligence to pre-
33. Mahajan SM, Ghani R. Combining structured and unstructured data for predicting vent mobile heart failure patients decompensation in real time: monitoring-based
risk of readmission for heart failure patients. Stud Health Technol Inform 2019;264: predictive model. Mobile Inform Syst 2018;2018:1546210.
238–242. 46. Parvandeh S, Yeh H-W, Paulus MP, McKinney BA. Consensus features nested cross-
34. Mahajan SM, Mahajan AS, King R, Negahban S. Predicting risk of 30-day readmissions validation. Bioinformatics 2020;36:3093–3098.
using two emerging machine learning methods. Stud Health Technol Inform 2018;250: 47. Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, et al. Scalable and accurate
250–255. deep learning with electronic health records. NPJ Digit Med 2018;1:18.
35. McKinley D, Moye-Dickerson P, Davis S, Akil A. Impact of a pharmacist-led interven- 48. Hilton CB, Milinovich A, Felix C, Vakharia N, Crone T, Donovan C, et al. Personalized
tion on 30-day readmission and assessment of factors predictive of readmission in predictions of patient outcomes during and after hospitalization using artificial intel-
African American men with heart failure. Am J Men’s Health 2019;13: ligence. NPJ Digit Med 2020;3:51.
1557988318814295. 49. Morawski K, Dvorkis Y, Monsen CB. Predicting hospitalizations from electronic
36. Chu J, Dong W, Huang Z. Endpoint prediction of heart failure using electronic health health record data. Am J Manag Care 2020;26:e7–e13.
records. J Biomed Inform 2020;109:103518. 50. Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A sys-
37. Chen P, Dong W, Wang J, Lu X, Kaymak U, Huang Z. Interpretable clinical prediction tematic review shows no performance benefit of machine learning over logistic re-
via attention-based neural network. BMC Med Inform Decis Mak 2020;20:131. gression for clinical prediction models. J Clini Epidemiol 2019;110:12–22.
38. Mahajan SM, Ghani R. Using ensemble machine learning methods for predicting risk 51. van der Aalst WMP, Bichler M, Heinzl A. Responsible data science. Business Inform
of readmission for heart failure. Stud Health Technol Inform 2019;264:243–247. Syst Eng 2017;59:311–313.
39. Desai RJ, Wang SV, Vaduganathan M, Evers T, Schneeweiss S. Comparison of ma- 52. Wag Z, Nayfeh T, Tetzlaff J, O’Blenis P, Murad MH. Error rates of human reviewers
chine learning methods with traditional models for use of administrative claims during abstract screening in systematic reviews. PLoS One 2020;15:e0227742.