Clinical Prediction Models A Practical Approach to

Development, Validation, and Updating - 2nd Edition

Visit the link below to download the full version of this book:

https://medipdf.com/product/clinical-prediction-models-a-practical-approach-to-d
evelopment-validation-and-updating-2nd-edition/

Click Download Now

Ewout W. Steyerberg

Clinical Prediction Models

A Practical Approach to Development,
Validation, and Updating

Second Edition

123
Ewout W. Steyerberg
Department of Biomedical Data Sciences
Leiden University Medical Center
Leiden, The Netherlands

ISSN 1431-8776 ISSN 2197-5671 (electronic)

Statistics for Biology and Health
ISBN 978-3-030-16398-3 ISBN 978-3-030-16399-0 (eBook)
https://doi.org/10.1007/978-3-030-16399-0
1st edition: © Springer Science+Business Media, LLC 2009
2nd edition: © Springer Nature Switzerland AG 2019
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt from
the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
herein or for any errors or omissions that may have been made. The publisher remains neutral with regard
to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
For Aleida, Matthijs, Laurens and Suzanne
For my father Wim
Preface

The first edition of this book was made during the years 2005–2007. Since then
quite some new developments have taken place, both in the general scientific
direction that prediction research is taking and specific technical innovations. These
developments have been addressed as far as possible in the second edition. Many
new references have been added. Some detailed material has been moved from print
to the web. Many figures have been redrawn in color for better clarity and attrac-
tiveness. In all, many changes have been made to nearly every chapter.
Prediction models are important in widely diverse fields, including medicine,
physics, engineering, meteorology, and finance. Prediction models are becoming
more relevant in the medical field with the increase in biological knowledge on
potential predictors of outcome, e.g., from “omics” (including genomics, tran-
scriptomics, proteomics, glycomics, metabolomics). Also, the Big Data era implies
we will have increasing access to large volumes of routinely collected data. The
number of applications for prediction models will increase, e.g., with targeted early
detection of disease, and individualized approaches to diagnostic testing and
treatment.
We are moving to an era of personalized evidence-based medicine that asks for
an individualized approach to shared medical decision-making. Evidence-based
medicine has a central place for meta-analysis to summarize results from ran-
domized controlled trials; prediction models summarize the effects of predictors to
provide individualized predictions of the absolute risk of a diagnostic or prognostic
outcome. Prediction models and related algorithms will increasingly form the basis
for personalized evidence-based medicine and individualized decision-making.

Why Read This Book?

My motivation for working on the first and second editions of this book stems
primarily from the fact that the development and applications of prediction models
are often suboptimal in medical publications. With this book, I hope to contribute to

vii
viii Preface

better understanding of relevant issues and give practical advice on better modeling
strategies than are nowadays used.
Issues include the following:
(a) Better predictive modeling is sometimes readily possible, e.g., a large data set
with high-quality data is available, but all continuous predictors are dichot-
omized, which is known to have several disadvantages.
(b) Small samples are used:
– Studies are underpowered, implying unreliable answers to difficult questions
such as “Which are the most important predictors in this prediction
problem?”
– The problem of small sample size is aggravated by doing a complete case
analysis which discards information from nearly complete records.
Statistical imputation methods are nowadays available to exploit all avail-
able information, especially “multiple imputations.”
– Predictors are omitted that should reasonably have been included based on
subject matter knowledge. Analysts rely too much on the limited data that
they have available in their data set, instead of wisely combining information
from several sources, such as medical literature and experts in the field.
– Stepwise selection methods are abundant when researchers apply regression
modeling, while these methods are suboptimal, especially in small data sets.
– Modeling approaches are used that require higher numbers. Data-hungry
techniques, such as neural network modeling, machine learning or artificial
intelligence techniques, should not be used in small data sets.
– No attempts are made towards validation, or validation is done inefficiently.
For example, a split-sample approach is followed, leading to a smaller
sample for model development and a smaller sample for model validation.
Better methods are nowadays available and should be used far more often,
specifically bootstrap resampling.
(c) Claims are exaggerated:
– Often, we see statements such as “the independent predictors were identi-
fied”; in many instances, such findings are purely exploratory and may not
be reproducible; they may largely represent noise.
– Models are not internally valid, with overoptimistic expectations of model
performance in new patients.
– One modern machine learning method with a fancy name is claimed as
being superior to a more traditional regression approach, while no con-
vincing evidence is presented, and a suboptimal modeling strategy was
followed for the regression model. Fair comparisons between well-used
statistical methods and machine learning methods are required.
– Researchers are insufficiently aware of overfitting, implying that their
apparent findings are merely coincidental.
Preface ix

(d) Poor generalizability:

– If models are not internally valid, we cannot expect them to generalize to
new patients.
– Models are developed for each local situation, discarding earlier findings on
effects of predictors and earlier models; a framework for continuous
improvement and updating of prediction models is required.
In this book, I suggest many small improvements in modeling strategies.
Combined, these improvements should lead to better development, validation, and
updating of prediction models.

Intended Audience

Readers should have a basic knowledge of biostatistics, especially regression

analysis, but no strong background in mathematics is required. The number of
formulas is deliberately kept small. The focus is on concepts in prediction research,
which are also relevant to computer scientists and data scientists working on pre-
diction in the field of Predictive Analytics.
Usually, a bottom-up approach is followed in teaching regression analysis
techniques, starting with the required type of data, model assumptions, estimation
methods, and basic interpretation. This book is more top-down: given that we want
to predict an outcome, how can we best utilize regression and related techniques?
Three levels of readers are envisioned:
(a) The core intended audience is formed by epidemiologists and applied bio-
statisticians who want to develop, validate, or update a prediction model. Both
students and professionals should find practical guidance in this book, espe-
cially by the proposed seven steps to develop a valid model (Part II).
(b) The second group is formed by clinicians, policy-makers, and healthcare profes-
sionals who want to judge a study that presents or validates a prediction model.
This book should aid them in a critical appraisal, providing explanations of terms
and concepts that are common in publications on prediction models. They should
try to read chapters of particular interest, or read the main text of the chapters. They
can skip the examples and more technical sections (indicated with*).
(c) The third group includes more theoretical researchers, such as (bio)statisticians
and computer scientists, who want to improve the methods that we use in
prediction models. They may find inspiration for further theoretical work and
simulation studies in this book. Many of the methods in prediction modeling
are not fully developed yet, and common sense or intuition underlies some
of the proposed approaches in this book. Improvements are welcome!
x Preface

Other Sources

Many excellent textbooks exist on regression analysis techniques, but these usually
do not have a focus on modeling strategies for prediction. The main exception is
Frank Harrell’s book “Regression Modeling Strategies”. He brings advanced bio-
statistical concepts to practical application, supported by the rms package for R.
Harrell’s book may, however, be too advanced for clinical and epidemiological
researchers. This also holds for the Hastie, Tibshirani, and Friedman quite thorough
textbook “The Elements of Statistical Learning”. These books are very useful for a
more in-depth discussion of statistical techniques and strategies. Harrell’s book
provided the main inspiration for the presented work here. Another good com-
panion book is the Vittinghoff et al. book on “Regression Methods in Biostatistics”.
Various sources at the Internet can be used that explain terms used in this book.
Frank Harrell maintains a useful glossary: [http://hbiostat.org/doc/glossary.pdf].

Structure

It has been found that people learn by example, by checklists, and by own dis-
covery. Therefore, many examples are provided throughout the text, including the
essential computer code and output. I also suggest a checklist for prediction
modeling (Part II). Own discovery is possible with exercises per chapter, with data
sets and scripts provided at the book’s website: www.clinicalpredictionmodels.org.
Many statistical techniques and approaches are readily possible with any modern
software package. Personally, I have worked with SPSS for simple, straightforward
analyses. This package is insufficient for more advanced analyses which are essential
in prediction modeling. The SAS computer package is more advanced, but may not
be so practical for some. A package such as Stata is very suitable. It is similar in
capabilities to R software for the key elements of prediction modeling. The R soft-
ware has several advantages: the software is for free, and innovations in biostatistical
methods become readily available. Therefore, R is the natural choice as the software
accompanying this book. R software is available at www.cran.r-project.org, with help
files and a tutorial.
An important disadvantage of R is a relatively slow learning curve; it takes time
and efforts to learn R. Some R commands are provided in this book; full programs
Preface xi

can be downloaded from the book’s website (www.clinicalpredictionmodels.org).

This website also provides a number of data sets that can be downloaded for
application of the described techniques. I provide data files in SPSS format that can
readily be imported in R and other packages. Many useful R tips and tricks are
available on the web.

Leiden, The Netherlands Ewout W. Steyerberg

Acknowledgements

Many have made small to large contributions to this book and the revision. I’m very
grateful to all. Frank Harrell has been a source of inspiration for my research in the
area of clinical prediction models, together with Hans van Houwelingen, who has
developed many of the theoretical innovations that are presented in this book. I’m
grateful to be his successor as a chair of the Department of Biomedical Data
Sciences at the Leiden University Medical Center. At the Department of Public
Health, Erasmus MC, Rotterdam, Dik Habbema, and René Eijkemans have
sharpened my thinking on prediction modeling. Hester Lingsma was very sup-
portive in the last phase of finishing the first edition of this book and has been a
wonderful colleague over many years. Lex Burdorf, Daan Nieboer, and Jan
Verbeek (Erasmus MC) provided specific comments for the second edition.
My insights in meta-analysis have benefitted from a project with Carl Moons and
Thomas Debray (Utrecht). I have enjoyed the vigorous discussions about the
evaluation of model performance with Ben van Calster, Michael Pencina, Stuart
Baker, and Andrew Vickers, which is reflected in further textual changes in the
second edition. Several Ph.D. students, colleagues, and external reviewers provided
input and made specific comments on various chapters.
I specifically would like to thank investigators who allowed their data sets to be
made available for didactic purposes, including Kerry Lee (Duke University) for the
GUSTO-I data, Andrew Maas (Antwerp University) for the IMPACT data, Yolanda
van der Graaf (Utrecht University) for the SMART data, and all other investigators
and patients who were involved in the studies used in this book. Finally, I thank my
family for their love and support over the years, and for allowing me to devote
private time to this book.

Leiden, The Netherlands Ewout W. Steyerberg

February 2019

xiii
Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 1
1.1 Diagnosis, Prognosis, and Therapy Choice in Medicine . . . . .. 1
1.1.1 Predictions for Personalized Evidence-Based
Medicine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 1
1.2 Statistical Modeling for Prediction . . . . . . . . . . . . . . . . . . . .. 5
1.2.1 Model Assumptions . . . . . . . . . . . . . . . . . . . . . . .. 5
1.2.2 Reliability of Predictions: Aleatory and Epistemic
Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.3 Sample Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Structure of the Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.1 Part I: Prediction Models in Medicine . . . . . . . . . . . 9
1.3.2 Part II: Developing Internally Valid Prediction
Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 9
1.3.3 Part III: Generalizability of Prediction Models . . . .. 10
1.3.4 Part IV: Applications . . . . . . . . . . . . . . . . . . . . . .. 11

Part I Prediction Models in Medicine

2 Applications of Prediction Models . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1 Applications: Medical Practice and Research . . . . . . . . . . . . . 15
2.2 Prediction Models for Public Health . . . . . . . . . . . . . . . . . . . . 16
2.2.1 Targeting of Preventive Interventions . . . . . . . . . . . . 16
2.2.2 *Example: Prediction for Breast Cancer . . . . . . . . . . 17
2.3 Prediction Models for Clinical Practice . . . . . . . . . . . . . . . . . . 18
2.3.1 Decision Support on Test Ordering . . . . . . . . . . . . . 18
2.3.2 *Example: Predicting Renal Artery Stenosis . . . . . . . 19
2.3.3 Starting Treatment: The Treatment Threshold . . . . . . 20
2.3.4 *Example: Probability of Deep Venous
Thrombosis . . . . . . . . . . . . . . . . . . . . . . . . . ..... 21
2.3.5 Intensity of Treatment . . . . . . . . . . . . . . . . . . ..... 22

xv
xvi Contents

2.3.6 *Example: Defining a Poor Prognosis Subgroup

in Cancer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.7 Cost-Effectiveness of Treatment . . . . . . . . . . . . . . . . 23
2.3.8 Delaying Treatment . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.9 *Example: Spontaneous Pregnancy Chances . . . . . . . 24
2.3.10 Surgical Decision-Making . . . . . . . . . . . . . . . . . . . . 25
2.3.11 *Example: Replacement of Risky Heart Valves . . . . 26
2.4 Prediction Models for Medical Research . . . . . . . . . . . . . . . . . 27
2.4.1 Inclusion and Stratification in a RCT . . . . . . . . . . . . 28
2.4.2 *Example: Selection for TBI Trials . . . . . . . . . . . . . 28
2.4.3 Covariate Adjustment in a RCT . . . . . . . . . . . . . . . . 29
2.4.4 Gain in Power by Covariate Adjustment . . . . . . . . . 31
2.4.5 *Example: Analysis of the GUSTO-III Trial . . . . . . . 31
2.4.6 Prediction Models and Observational Studies . . . . . . 32
2.4.7 Propensity Scores . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.4.8 *Example: Statin Treatment Effects . . . . . . . . . . . . . 33
2.4.9 Provider Comparisons . . . . . . . . . . . . . . . . . . . . . . . 34
2.4.10 *Example: Ranking Cardiac Outcome . . . . . . . . . . . 34
2.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3 Study Design for Prediction Modeling . . . . . . . . . . . . . . . . . . .... 37
3.1 Studies for Prognosis . . . . . . . . . . . . . . . . . . . . . . . . . . . .... 37
3.1.1 Retrospective Designs . . . . . . . . . . . . . . . . . . . .... 38
3.1.2 *Example: Predicting Early Mortality in
Esophageal Cancer . . . . . . . . . . . . . . . . . . . . . .... 39
3.1.3 Prospective Designs . . . . . . . . . . . . . . . . . . . . .... 39
3.1.4 *Example: Predicting Long-term Mortality
in Esophageal Cancer . . . . . . . . . . . . . . . . . . . .... 40
3.1.5 Registry Data . . . . . . . . . . . . . . . . . . . . . . . . . .... 40
3.1.6 *Example: Surgical Mortality in Esophageal
Cancer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .... 40
3.1.7 Nested Case–Control Studies . . . . . . . . . . . . . . .... 41
3.1.8 *Example: Perioperative Mortality in Major
Vascular Surgery . . . . . . . . . . . . . . . . . . . . . . .... 41
3.2 Studies for Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . .... 41
3.2.1 Cross-sectional Study Design and Multivariable
Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2.2 *Example: Diagnosing Renal Artery Stenosis . . . . . . 42
3.2.3 Case–Control Studies . . . . . . . . . . . . . . . . . . . . . . . 42
3.2.4 *Example: Diagnosing Acute Appendicitis . . . . . . . . 43
3.3 Predictors and Outcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.3.1 Strength of Predictors . . . . . . . . . . . . . . . . . . . . . . . 43
3.3.2 Categories of Predictors . . . . . . . . . . . . . . . . . . . . . 43
Contents xvii

3.3.3 Costs of Predictors . . . . . . . . . . . . . . . . . . . . . . . . . 44

3.3.4 Determinants of Prognosis . . . . . . . . . . . . . . . . . . . . 44
3.3.5 Prognosis in Oncology . . . . . . . . . . . . . . . . . . . . . . 45
3.4 Reliability of Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.4.1 Observer Variability . . . . . . . . . . . . . . . . . . . . . . . . 45
3.4.2 *Example: Histology in Barrett’s Esophagus . . . . . . 46
3.4.3 Biological Variability . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.4 Regression Dilution Bias . . . . . . . . . . . . . . . . . . . . . 46
3.4.5 *Example: Simulation Study on Reliability
of a Binary Predictor . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.6 Choice of Predictors . . . . . . . . . . . . . . . . . . . . . . . . 47
3.5 Outcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.5.1 Types of Outcome . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.5.2 Survival End Points . . . . . . . . . . . . . . . . . . . . . . . . 49
3.5.3 *Examples: 5-Year Relative Survival in Cancer
Registries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.5.4 Composite End Points . . . . . . . . . . . . . . . . . . . . . . . 49
3.5.5 *Example: Composite End Points in Cardiology . . . . 50
3.5.6 Choice of Prognostic Outcome . . . . . . . . . . . . . . . . 50
3.5.7 Diagnostic End Points . . . . . . . . . . . . . . . . . . . . . . . 50
3.5.8 *Example: PET Scans in Esophageal Cancer . . . . . . 51
3.6 Phases of Biomarker Development . . . . . . . . . . . . . . . . . . . . . 51
3.7 Statistical Power and Reliable Estimation . . . . . . . . . . . . . . . . 52
3.7.1 Sample Size to Identify Predictor Effects . . . . . . . . . 52
3.7.2 Sample Size for Reliable Modeling . . . . . . . . . . . . . 54
3.7.3 Sample Size for Reliable Validation . . . . . . . . . . . . . 56
3.8 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4 Statistical Models for Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.1 Continuous Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.1.1 *Examples of Linear Regression . . . . . . . . . . . . . . . 60
4.1.2 Economic Outcomes . . . . . . . . . . . . . . . . . . . . . . . . 60
4.1.3 *Example: Prediction of Costs . . . . . . . . . . . . . . . . . 60
4.1.4 Transforming the Outcome . . . . . . . . . . . . . . . . . . . 61
4.1.5 Performance: Explained Variation . . . . . . . . . . . . . . 61
4.1.6 More Flexible Approaches . . . . . . . . . . . . . . . . . . . 63
4.2 Binary Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.2.1 R2 in Logistic Regression Analysis . . . . . . . . . . . . . 65
4.2.2 Calculation of R2 on the Log-Likelihood Scale . . . . . 65
4.2.3 Models Related to Logistic Regression . . . . . . . . . . . 68
4.2.4 Bayes Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.2.5 Prediction with Naïve Bayes . . . . . . . . . . . . . . . . . . 69
4.2.6 Calibration and Naïve Bayes . . . . . . . . . . . . . . . . . . 70
xviii Contents

4.2.7 *Logistic Regression and Bayes . . . . . . . . . . . . . . . 70

4.2.8 Machine Learning: More Flexible Approaches . . . . . 70
4.2.9 Classification and Regression Trees . . . . . . . . . . . . . 71
4.2.10 *Example: Mortality in Acute MI Patients . . . . . . . . 72
4.2.11 Advantages and Disadvantages of Tree Models . . . . 72
4.2.12 Trees Versus Logistic Regression Modeling . . . . . . . 73
4.2.13 *Other Methods for Binary Outcomes . . . . . . . . . . . 74
4.2.14 Summary of Binary Outcomes . . . . . . . . . . . . . . . . . 75
4.3 Categorical Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.3.1 Polytomous Logistic Regression . . . . . . . . . . . . . . . 76
4.3.2 Example: Histology of Residual Masses . . . . . . . . . . 76
4.3.3 *Alternative Models . . . . . . . . . . . . . . . . . . . . . . . . 77
4.3.4 *Comparison of Modeling Approaches . . . . . . . . . . 78
4.4 Ordinal Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.4.1 Proportional Odds Logistic Regression . . . . . . . . . . . 80
4.4.2 *Relevance of the Proportional Odds Assumption
in RCTs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.5 Survival Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.5.1 Cox Proportional Hazards Regression . . . . . . . . . . . 82
4.5.2 Prediction with Cox Models . . . . . . . . . . . . . . . . . . 83
4.5.3 Proportionality Assumption . . . . . . . . . . . . . . . . . . . 84
4.5.4 Kaplan–Meier Analysis . . . . . . . . . . . . . . . . . . . . . . 84
4.5.5 *Example: Impairment After Treatment
of Leprosy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.5.6 Parametric Survival . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.5.7 *Example: Replacement of Risky Heart Valves . . . . 86
4.5.8 Summary of Survival Outcomes . . . . . . . . . . . . . . . 86
4.6 Competing Risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.6.1 Actuarial and Actual Risks . . . . . . . . . . . . . . . . . . . 87
4.6.2 Absolute Risk and the Fine & Gray Model . . . . . . . 88
4.6.3 Example: Prediction of Coronary Heart Disease
Incidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.6.4 Multistate Modeling . . . . . . . . . . . . . . . . . . . . . . . . 90
4.7 Dynamic Predictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.7.1 Multistate Models and Landmarking . . . . . . . . . . . . 91
4.7.2 Joint Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.8 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5 Overfitting and Optimism in Prediction Models . . . . . . . . . . . . . . . 95
5.1 Overfitting and Optimism . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.1.1 Example: Surgical Mortality in Esophagectomy . . . . 96
5.1.2 Variability Within One Center . . . . . . . . . . . . . . . . . 97
Contents xix

5.1.3 Variability Between Centers: Noise Versus True

Heterogeneity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.1.4 Predicting Mortality by Center: Shrinkage . . . . . . . . 98
5.2 Overfitting in Regression Models . . . . . . . . . . . . . . . . . . . . . . 99
5.2.1 Model Uncertainty and Testimation Bias . . . . . . . . . 99
5.2.2 Other Modeling Biases . . . . . . . . . . . . . . . . . . . . . . 101
5.2.3 Overfitting by Parameter Uncertainty . . . . . . . . . . . . 101
5.2.4 Optimism in Model Performance . . . . . . . . . . . . . . . 102
5.2.5 Optimism-Corrected Performance . . . . . . . . . . . . . . 104
5.3 Bootstrap Resampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.3.1 Applications of the Bootstrap . . . . . . . . . . . . . . . . . 105
5.3.2 Bootstrapping for Regression Coefficients . . . . . . . . 105
5.3.3 Bootstrapping for Prediction: Optimism
Correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.3.4 Calculation of Optimism-Corrected Performance . . . . 107
5.3.5 *Example: Stepwise Selection in 429 Patients . . . . . 108
5.4 Cost of Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.4.1 *Degrees of Freedom of a Model . . . . . . . . . . . . . . 110
5.4.2 Practical Implications . . . . . . . . . . . . . . . . . . . . . . . 110
5.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6 Choosing Between Alternative Models . . . . . . . . . . . . . . . . . . . . . . 113
6.1 Prediction with Statistical Models . . . . . . . . . . . . . . . . . . . . . 113
6.1.1 Testing of Model Assumptions and Prediction . . . . . 114
6.1.2 Choosing a Type of Model . . . . . . . . . . . . . . . . . . . 114
6.2 Modeling Age–Outcome Relations . . . . . . . . . . . . . . . . . . . . . 115
6.2.1 *Age and Mortality After Acute MI . . . . . . . . . . . . . 115
6.2.2 *Age and Operative Mortality . . . . . . . . . . . . . . . . . 116
6.2.3 *Age–Outcome Relations in Other Diseases . . . . . . . 119
6.3 Head-to-Head Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.3.1 StatLog Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6.3.2 *Cardiovascular Disease Prediction Comparisons . . . 121
6.3.3 *Traumatic Brain Injury Modeling Results . . . . . . . . 122
6.4 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Part II Developing Valid Prediction Models

7 Missing Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7.1 Missing Values and Prediction Research . . . . . . . . . . . . . . . . . 127
7.1.1 Inefficiency of Complete Case Analysis . . . . . . . . . . 128
7.1.2 Interpretation of CC Analyses . . . . . . . . . . . . . . . . . 129
7.1.3 Missing Data Mechanisms . . . . . . . . . . . . . . . . . . . . 129
7.1.4 Missing Outcome Data . . . . . . . . . . . . . . . . . . . . . . 130
7.1.5 Summary Points . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
xx Contents

7.2 Prediction Under MCAR, MAR and MNAR Mechanisms . . . . 132

7.2.1 Missingness Patterns . . . . . . . . . . . . . . . . . . . . . . . . 132
7.2.2 Missingness and Estimated Regression
Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7.2.3 Missingness and Estimated Performance . . . . . . . . . 137
7.3 Dealing with Missing Values in Regression Analysis . . . . . . . 137
7.3.1 Imputation Principle . . . . . . . . . . . . . . . . . . . . . . . . 138
7.3.2 Simple and More Advanced Single Imputation
Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.3.3 Multiple Imputation . . . . . . . . . . . . . . . . . . . . . . . . 139
7.4 Defining the Imputation Model . . . . . . . . . . . . . . . . . . . . . . . 141
7.4.1 Types of Variables in the Imputation Model . . . . . . . 141
7.4.2 *Transformations of Variables . . . . . . . . . . . . . . . . . 142
7.4.3 Imputation Models for SI and MI: X and y . . . . . . . 142
7.4.4 Summary Points . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7.5 Success of Imputation Under MCAR, MAR and MNAR . . . . . 143
7.5.1 Imputation in a Simple Model . . . . . . . . . . . . . . . . . 143
7.5.2 Other Simulation Results . . . . . . . . . . . . . . . . . . . . . 143
7.5.3 *Multiple Predictors . . . . . . . . . . . . . . . . . . . . . . . . 144
7.6 Guidance to Dealing with Missing Values in Prediction
Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
7.6.1 Patterns of Missingness . . . . . . . . . . . . . . . . . . . . . . 146
7.6.2 Simple Approaches . . . . . . . . . . . . . . . . . . . . . . . . . 146
7.6.3 More Advanced Approaches . . . . . . . . . . . . . . . . . . 147
7.6.4 Maximum Fraction of Missing Values Before
Omitting a Predictor . . . . . . . . . . . . . . . . . . . . . . . . 147
7.6.5 Single or Multiple Imputation for Predictor
Effects? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.6.6 Single or Multiple Imputation for Deriving
Predictions? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
7.6.7 Missings and Predictions for New Patients . . . . . . . . 150
7.6.8 *Performance Across Multiple Imputed
Data Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.6.9 Reporting of Missing Values in Prediction
Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
7.7 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.7.1 Summary Statements . . . . . . . . . . . . . . . . . . . . . . . . 152
7.7.2 *Available Software and Challenges . . . . . . . . . . . . 153
8 Case Study on Dealing with Missing Values . . . . . . . . . . . . . . . . . . 157
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
8.1.1 Aim of the IMPACT Study . . . . . . . . . . . . . . . . . . . 157
8.1.2 Patient Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . 158