Vincent and Moreno Critical Care 2010, 14:207

http://ccforum.com/14/2/207

REVIEW

Clinical review: Scoring systems in the critically ill

Jean-Louis Vincent*1 and Rui Moreno2

The objective of this review is to give the intensivist

Abstract
without any particular knowledge or expertise in this
General illness severity scores are widely used in the area an overview of the current status of these
ICU to predict outcome, characterize disease severity instruments and their possible applications. For a more
and degree of organ dysfunction, and assess resource detailed explanation of the development, application and
use. In this article we review the most commonly used limitations of these models, the reader is referred to a
scoring systems in each of these three groups. We recent review [1].
examine the history of the development of the initial
major systems in each group, discuss the construction Outcome prediction scores
of subsequent versions, and, when available, provide The original outcome prediction scores were developed
recent comparative data regarding their performance. more than 25 years ago to provide an indication of the
Importantly, the different types of scores should be risk of death of groups of ICU patients; they were not
seen as complementary, rather than competitive and designed for individual prognostication. Patient demo-
mutually exclusive. It is possible that their combined graphics, disease prevalence, and intensive care practice
use could provide a more accurate indication of have changed considerably since [2], and statistical and
disease severity and prognosis. All these scoring computational techniques have also progressed. As a
systems will need to be updated with time as ICU result, all three of the major scores in this category have
populations change and new diagnostic, therapeutic been recently updated to ensure their continued accuracy
and prognostic techniques become available. in today’s ICU (Table 1).

Acute Physiology and Chronic Health Evaluation

Introduction The original APACHE score was developed in 1981 to
Scoring systems used in critically ill patients can be classify groups of patients according to severity of illness
broadly divided into those that are specific for an organ and was divided into two sections: a physiology score to
or disease (for example, the Glasgow Coma Scale (GCS)) assess the degree of acute illness; and a preadmission
and those that are generic for all ICU patients. In this evaluation to determine the chronic health status of the
article, we focus on the generic scores, which can broadly patient [3]. In 1985, the original model was revised and
be divided into scores that assess disease severity on simplified to create APACHE II [4], now the world’s most
admission and use it to predict outcome (for example, widely used severity of illness score. In APACHE II, there
Acute Physiology and Chronic Health Evaluation are just 12 physiological variables, compared to 34 in the
(APACHE), Simplified Acute Physiology Score (SAPS), original score. The effects of age and chronic health status
Mortality Probability Model (MPM)), scores that assess are incorporated directly into the model, weighted
the presence and severity of organ dysfunction (for according to their relative impact, to give a single score
example, Multiple Organ Dysfunction Score (MODS), with a maximum of 71. The worst value recorded during
Sequential Organ Failure Assessment (SOFA)), and the first 24 hours of a patient’s admission to the ICU is
scores that assess nursing workload use (for example, used for each physiological variable. The principal
Therapeutic Intervention Scoring System (TISS), Nine diagnosis leading to ICU admission is added as a category
Equivalents of Nursing Manpower Use Score (NEMS)). weight so that the predicted mortality is computed based
on the patient’s APACHE II score and their principal
diagnosis at admission. The reason for ICU admission is,
*Correspondence: [email protected]
1
Department of Intensive Care, Erasme University Hospital, Route de Lennik 808,
therefore, an important variable in predicting mortality,
1070 Brussels, Belgium even when previous health status and the degree of acute
Full list of author information is available at the end of the article physiological dysfunction are similar.
APACHE III was developed in 1991 [5] and was
© 2010 BioMed Central Ltd © 2010 BioMed Central Ltd validated and further updated in 1998 [6]. Equations for
Vincent and Moreno Critical Care 2010, 14:207 Page 2 of 9
http://ccforum.com/14/2/207

Table 1. Comparison of general outcome prediction models

APACHE SAPS APACHE II MPMa APACHE III SAPS II MPM IIb SAPS 3 APACHE IV MPM III
Characteristics [3] [10] [4] [14] [5] [11] [15] [12] [8] [17]
Year 1981 1984 1985 1985 1991 1993 1993 2005 2006 2007
Countries 1 1 1 1 1 12 12 35 1 1
ICUs 2 8 13 1 40 137 140 303 104 135
Patients 705 679 5,815 2,783 17,440 12,997 19,124 16,784 110,558 124,855
Selection of Panel Panel Panel Multiple Multiple Multiple Multiple Multiple Multiple Multiple
variables and of of of logistic logistic logistic logistic logistic logistic logistic
their weights experts experts experts regression regression regression regression regression regression regression
Variables
Age No Yes Yes Yes Yes Yes Yes Yes Yes Yes
Origin No No No No Yes No No Yes Yes No
Surgical status No No Yes Yes Yes Yes Yes Yes Yes Yes
Chronic Yes No Yes Yes Yes Yes Yes Yes Yes Yes
health status
Physiology Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Acute diagnosis No No Yes No Yes No Yes Yes Yes Yes
Number of variables 34 14 17 11 26 17 15c 20 142 16d
Score Yes Yes Yes No Yes Yes No Yes Yes No
Mortality prediction No No Yes Yes Yes Yes Yes Yes Yes Yes
a
These models are based on previous versions, developed by the same authors. bThe numbers presented are those for the admission component of the model
(MPM0 II). cMPM24 II uses only 13 variables. dPlus 7 interaction terms. APACHE, Acute Physiology and Chronic Health Evaluation; SAPS, Simplified Acute Physiology
Score; MPM, Mortality Probability Model. Adapted from [64] with permission.

predicting risk-adjusted ICU length of stay were also patients from 303 ICUs in 35 countries [12]. The SAPS 3
developed using the APACHE III model [7]. Most score includes 20 variables divided into three subscores
recently, APACHE IV was developed using a database of related to patient characteristics prior to admission, the
over 100,000 patients admitted to 104 ICUs in 45 circumstance of the admission, and the degree of
hospitals in the USA in 2002/2003, and remodeling physiological derangement within 1 hour (in contrast to
APACHE III with the same physiological variables and the 24-hour time window in the SAPS II model) before or
weights but different predictor variables and refined after ICU admission. The total score can range from 0 to
statistical methods [8]. APACHE IV again provides ICU 217. Unlike the other scores, SAPS 3 includes customized
length of stay prediction equations, which can provide equations for prediction of hospital mortality in seven
benchmarks for the assessment and comparison of ICU geographical regions: Australasia; Central, South America;
efficiency and resource use [9]. Central, Western Europe; Eastern Europe; North Europe;
Southern Europe, Mediterranean; and North America. It
Simplified Acute Physiology Score should be noted that the sample size for development of
SAPS, developed and validated in France in 1984, used 13 some of these equations was relatively small, which may
weighted physiological variables and age to predict risk of compromise their prognostic accuracy. The SAPS 3 score
death in ICU patients [10]. Like the APACHE scores, has been shown to exhibit good discrimination,
SAPS was calculated from the worst values obtained calibration, and goodness of fit [12]. SAPS 3 has also been
during the first 24 hours of ICU admission. In 1993, Le used to examine variability in resource use between ICUs
Gall and colleagues [11] used logistic regression analysis using the standardized resource use parameter based on
to develop SAPS II, which includes 17 variables: 12 the length of stay in the ICU adjusted for severity of acute
physiological variables, age, type of admission, and 3 illness [13].
variables related to underlying disease. The SAPS II score
was validated using data from consecutive admissions to Mortality Probability Model
137 ICUs in 12 countries [11]. The first MPM, developed from data from patients in one
In 2005, a completely new SAPS model, the SAPS 3, ICU, consisted of an admission model using seven
was created. Complex statistical techniques were used to admission variables, and a 24-hour model using seven
select and weight variables using a database of 16,784 24-hour variables [14]. A revised MPM, MPM II, was
Vincent and Moreno Critical Care 2010, 14:207 Page 3 of 9
http://ccforum.com/14/2/207

developed in 1993 using logistic regression techniques on providing separate customized equations of different
a large database of 12,610 ICU patients from 12 countries geographical regions. Nevertheless, local customization
[15]. MPM II also consists of two scores: MPM0, the may still help improve the calibration of these scores in
admission model, which contains 15 variables; and MPM24 individual countries or regions as demonstrated for the
the 24-hour model, which contains 5 of the admission APACHE III in Cleveland, Ohio [21], or more recently for
variables and 8 additional variables and is designed for the SAPS 3 score in Austria [22]. In a retrospective
patients who stay in the ICU for more than 24  hours. analysis of prospectively collected data from a surgical
Unlike the APACHE and SAPS systems where variables ICU, Sakr and colleagues [23] reported that the discrimi-
are weighted, in MPM II each variable (except age, which native ability of SAPS 3 was similar to that of APACHE II
is entered as the actual age in years), is designated as and SAPS II (area under the receiver operating charac-
present or absent and given a score of 1 or 0 accordingly. teristic curve 0.80 for APACHE II, 0.83 for SAPS II, and
A logistic regression equation is then used to provide a 0.84 for SAPS 3). All three scores had poor calibration,
probability of hospital mortality. The authors also which improved after customization to the local popu-
developed a Weighted Hospital Days scale (WHD-94) by lation. In the UK, investigators have developed a new
subjectively assigning weights to days in the ICU and to scoring system specifically for use in UK ICU patients
hospital days after ICU discharge from the first ICU stay, [24]. This score uses elements of the APACHE, SAPS,
and an equation to predict an ICU’s mean WHD-94, thus and MPM systems and was developed using the large
providing an index of resource utilization [16]. Intensive Care National Audit and Research Centre
MPM0 has recently been updated using a database of (ICNARC) database and calibrated for adult critically ill
124,885 patients from 135 ICUs in 98 hospitals (all in patients admitted to ICUs in the UK. It performed better
North America except for one in Brazil) collected in 2001 than SAPS II, APACHE II and III, and MPM II [24], but
to 2004 [17]. MPM0-III uses 16 variables, including 3 has not been compared to the latest versions of these
physiological parameters, obtained within 1 hour of ICU scores.
admission to estimate mortality probability at hospital When using these instruments, in addition to the issues
discharge; the MPM0 characterization is, therefore, based related to local customization and regular updates
on patient condition largely before ICU care begins. The discussed above, a few important limitations should be
WHD-94 predictive equation has also been updated [18]. kept in mind. First, all general outcome prediction
models can only at their best predict the behavior of a
Discussion group of patients that exactly matches the patients in the
Several studies have compared the different outcome development population. For example, the APACHE and
prediction scoring systems. For example, in a study of MPM scores were largely based on North American
10,393 patients from Scottish ICUs, Livingston and populations and the SAPS score on European patients,
colleagues [19] compared the APACHE II and III, an while SAPS 3 developers used a database that included a
APACHE II using United Kingdom-derived coefficients geographically more heterogeneous group of patients
(UK APACHE II), SAPS II, and MPM0 and MPM24. These [12]. In addition, in most of the scores, specific
authors reported that all models showed good discrimi- populations were excluded from the original databases
nation, although observed mortality was significantly (for example, patients with burns, patients aged less than
different from that predicted by all models. SAPS II had 16 or 18 years, patients with a very short length of ICU
the best performance overall, but APACHE II had better stay, and so on).
calibration. In a retrospective study of 11,300 patients Second, the accuracy of any scoring system is highly
from 35 hospitals in California, Kuzniewicz and dependent on the quality of the input. To be used
colleagues [20] recently used logistic regression to correctly, the definitions, time of data collection, rules for
re-estimate the coefficients for the APACHE IV, MPM0- missing data, and so on must exactly match those applied
III and SAPS II scores and applied the new equations to when building the model. The reported reliability of the
assess risk-adjusted mortality rates. These authors noted systems (intra- and inter-observer) must also be taken
that discrimination and calibration were adequate for all into account.
models, with discrimination of APACHE IV slightly Third, there is an inherent bias in many of the derived
better than that of the other two scores (area under the equations used to predict mortality in that they are
receiver operating characteristic curve 0.892 for created from a limited population of patients from ICUs
APACHE IV, 0.873 for SAPS II, and 0.809 for MPM0 III, that are specifically interested in measuring (and
P < 0.001). improving) ICU performance.
In addition to using a more geographically hetero- Fourth, the outcome used in all these instruments is
geneous database for development, the SAPS 3 model the vital status at hospital discharge; consequently, the
attempted to address any geographic variation by use of other outcome measures (such as the vital status at
Vincent and Moreno Critical Care 2010, 14:207 Page 4 of 9
http://ccforum.com/14/2/207

Table 2. Comparison of three organ dysfunction scores

Characteristics LODS [29] MODS [30] SOFA [31]
Year of publication 1996 1995 1996
Selection of variables and their weights Multiple logistic regression Literature review and logistic Panel of experts
regression
Variables used to assess organ dysfunction
Neurologic Glasgow Coma Scale Glasgow Coma Scale Glasgow Coma Scale
Cardiovascular Heart rate, systolic blood Pressure-adjusted heart rate Mean arterial blood pressure,
pressure vasopressor use
Renal Serum urea or urea nitrogen, Serum creatinine Serum creatinine, urine output
creatinine, urine output
Respiratory PaO2/FiO2 ratio, mechanical PaO2/FiO2 ratio PaO2/FiO2 ratio, mechanical
ventilation ventilation
Hematologic White blood cell count, Platelet count Platelet count
platelet count
Hepatic Serum bilirubin, prothrombin time Serum bilirubin Serum bilirubin
LODS, Logistic Organ Dysfunction Score; MODS, Multiple Organ Dysfunction Score; SOFA, Sequential Organ Dysfunction Score.

ICU discharge) will compromise the accuracy of the survival. The severity of organ dysfunction varies widely
predictive equations. Nevertheless, some models have among individuals and within an individual over time
additional equations to assess use of resources, usually and organ failure scores must be able to take both time
measured as risk-adjusted, weighted, ICU- or hospital and severity into account. Many organ dysfunction scores
days [9,13,18]. have been developed over the past few decades, but we
Fifth, the statistical methodology used to assess will limit our discussion to three of the scores most
calibration of a predictive model, most commonly the commonly used in general ICU patients: the Logistic
Hosmer-Lemeshow statistic, may be influenced by Organ Dysfunction System (LODS) [29], MODS [30],
various factors, including the number of covariates being and SOFA [31] (Table 2).
assessed, the manner in which observations with equal
probabilities of outcome are sorted, and the sample size Logistic Organ Dysfunction Score
(both small and large) [25]. Interpretation of the accuracy The LODS was developed using a database of 13,152
of predictive models should, therefore, include some admissions to 137 ICUs in 12 countries [29]. Using
knowledge of the statistical tests used. Different statistical multiple logistic regression, 12 variables were selected to
techniques may be required for the larger models represent the function of six organ systems (neurologic,
increasingly used to develop predictive models, such as cardiovascular, renal, pulmonary, hematologic, hepatic).
the use of calibration graphs and, more recently, the Cox The worst value for each variable in the first 24 hours of
test of calibration and related statistics [26]. admission is recorded, and for each system, a score of 0
Sixth, despite the fact that predictive models have been (no dysfunction) to 5 (maximum dysfunction) is awarded.
developed in large populations, in almost all cases when Unlike the MODS and SOFA scores, LODS is a weighted
they are applied to new populations calibration deterior- system: for the respiratory and coagulation systems, the
ates, although discrimination hardly changes. Two recent maximum score allowed is 3, and for the liver the
examples of this effect were given in validation studies of maximum score is 1. LODS values, therefore, can range
SAPS 3 in Austria and in Italy [22,27]. from 0 to 22.
Seventh, the use of automatic patient data management The LODS lies somewhere between a mortality predic-
systems can, by changing the sampling rate for the tion score and an organ failure score as it combines a
physiological variables, change the accuracy of the model. global score summarizing the total degree of organ
Bosman and colleagues [28] reported that predicted dysfunction across the organ systems, and a logistic
mortality was greater with data management charting regression equation that can be used to convert the score
than with manual charting for APACHE II, SAPS II, and into a probability of mortality. Within organ systems,
MPM II. greater severity of organ dysfunction was consistently
associated with higher mortality [32], and a LODS of 22
Organ dysfunction scores was associated with a mortality of 99.7% [29]. The LODS
Organ failure scores are primarily designed to describe was not initially validated for repeated use during the ICU
the degree of organ dysfunction rather than to predict stay, but in a study of 1,685 patients in French ICUs, the
Vincent and Moreno Critical Care 2010, 14:207 Page 5 of 9
http://ccforum.com/14/2/207

LODS was accurate in characterizing the progression of Table 3. ‘Ideal’ descriptors of organ dysfunction in ICU
organ dysfunction during the first week of ICU stay [33]. patients
Simple and inexpensive
Multiple Organ Dysfunction Score Routinely available in all ICUs
The development of the MODS was based on a literature
Reliable (intra and inter-observer)
review of 30 publications that had characterized organ
Objective (that is, observer independent)
dysfunction [30,34]. Seven organ systems were then
selected for further consideration (respiratory, cardio- Specific to the function of the organ in question
vascular, renal, hepatic, hematological, central nervous Therapy independent
system, gastrointestinal), and variables for each organ Sequential (available at ICU admission or shortly thereafter and then at fixed
system were chosen according to a set of ‘ideal descriptor’ periods of time)
criteria (Table  3). No accurate descriptor of gastro- Not affected by transient, reversible abnormalities associated with
intestinal function could be identified, so this system was therapeutic or practical interventions
not included in the final model. For the cardiovascular Reflect acute dysfunction of the organ in question but not chronic
system, Marshall and colleagues [30] created a composite dysfunction
variable, the pressure-adjusted heart rate (heart rate × Reproducible in large, heterogeneous groups of ICU patients
central venous pressure/mean arterial pressure); in Reproducible in several types of ICUs from different regions of the globe
patients without a central line, this variable is assumed to Abnormal in one direction only
be normal. For each of the six organs, the first parameters
Using continuous rather than dichotomous variables
of the day are used to calculate the score and a score of 0
Modified from [34].
(normal) to 4 (most dysfunction) is awarded, giving a
total maximum score of 24. The score was developed in
336 patients admitted to one surgical ICU and validated time are also useful in predicting outcome. In a
in 356 patients admitted to the same ICU [30]. Although prospective study of 352 ICU patients, an increase in
not designed to predict ICU mortality, increasing MODS SOFA score during the first 48 hours in the ICU,
values do correlate with ICU outcome [30]. ICU mortality independent of the initial score, predicted a mortality
also increases with increasing numbers of failing organ rate of at least 50%, while a decrease was associated with
systems [30,35]. The delta MODS, defined as the an ICU mortality rate of just 27% [41]. In a prospective
difference between the MODS at admission and the observational study of 1,340 patients with multiple organ
maximum score, may be more predictive of outcome dysfunction syndrome, Cabrè and colleagues [42]
than individual scores [30]. reported 100% mortality for patients with age over
60 years, a total maximum SOFA greater than 13 on any
Sequential Organ Failure Assessment of the first 5  days of ICU admission, minimum SOFA
The SOFA was developed in 1994 during a consensus greater than 10 at all times, and a positive or unchanged
conference [31]. Six organ systems (respiratory, cardio- SOFA trend over the first 5 days of ICU admission.
vascular, renal, hepatic, central nervous, coagulation)
were selected based on a review of the literature, and the Discussion
function of each is scored from 0 (normal function) to 4 Several studies have directly compared the various organ
(most abnormal), giving a possible score of 0 to 24. Unlike dysfunction scoring systems. Pettilä and colleagues [43]
the MODS score in which the first value of each day is reported comparable discriminative power of APACHE
used, for the SOFA score, the worst value on each day is III, LODS, SOFA, and MODS to predict hospital
recorded. Another key difference is in the cardiovascular mortality in a single center study. Peres Bota and
component; instead of the composite variable, the SOFA colleagues [44] reported no significant differences between
score uses a treatment-related variable (dose of vaso- MODS and SOFA for mortality prediction in 949 general
pressor agents). This is not ideal, as treatment protocols ICU patients. However, when using the cardiovascular
vary among institutions, among patients and over time, component, outcome prediction was better for the SOFA
but it is difficult to avoid, especially for the cardiovascular score at all time intervals compared to the MODS, a
system. finding confirmed by other studies [45]. In a multicenter
The SOFA was initially validated in a mixed, medical- study, Timsit and colleagues [33] reported good accuracy
surgical ICU population [31,36] and has since been and internal consistency for both the SOFA and LODS.
validated and applied in various patient groups [37-39]. However, in a Canadian study of 1,436 ICU patients [45],
In a prospective analysis of 1,449 patients, a maximum SOFA and MODS had only a modest ability to discriminate
total SOFA score greater than 15 correlated with a between survivors and non-survivors. More recently, SOFA
mortality rate of 90% [40]. Changes in SOFA score over was reported to have superior discriminative ability for
Vincent and Moreno Critical Care 2010, 14:207 Page 6 of 9
http://ccforum.com/14/2/207

hospital mortality and unfavorable neurologic outcome determined from a 1-week observational cross-sectional
compared to MODS in patients with brain injury [46]. study and the results compared with those of the TISS-28
items in a cohort of 99 ICUs in 15 countries. At the end
Severity assessment based on nursing workload use of this process, a total of five new items and 14 sub-items
The Therapeutic Intervention Scoring System (TISS) describing nursing activities in the ICU (for example,
TISS was originally developed in 1974 to assess severity monitoring, care of relatives, administrative tasks) were
of illness and compare patient care based on the added to the TISS-28 list. The new activities accounted
measurement of nursing workload [47]. The original for 60% of the average nursing time; and in the
score included 57 therapeutic activities with points development study, NAS activities accounted for 81% of
assigned for each activity conducted during a 24-hour the nursing time (versus 43% in TISS-28) [54].
period; higher values were given for more specialized or
time-consuming activities. In 1983, the score was Discussion
updated and expanded to include 76 items [48]. However, These scores have been used mainly to assess nurse
TISS-76 was criticized for being too time-consuming and staffing in the ICU, although higher scores are associated
cumbersome, and in 1996, a simplified version was with worse outcomes [55,56]. All the scores are limited
devised using advanced statistical analysis [49]. TISS-28 by the items included, and can be prone to subjective
includes just 28 items, divided into 7 groups: basic interpretation and influenced by patient case-mix, local
activities, ventilatory support, cardiovascular support, admission and discharge policies, and local management
renal support, neurological support, metabolic support, protocols. Use of these scores to compare units may,
and specific interventions. The scoring is weighted to therefore, be difficult; however, within a unit they can
give a total score of 78. TISS-28 was validated in 22 provide a valuable indication of changing workload
Dutch ICUs [49] and in 19 ICUs in Portugal [50]. needs. These scores may also be used to estimate overall
According to this system, each nurse can provide care for costs for groups of ICU patients, although they are less
46.35 TISS-28 points per shift, with each TISS-28 point reliable on an individual patient basis [57]. Instruments,
requiring 10.6 minutes of each nurse’s shift. This such as the Work Utilization Ratio, which evaluates the
information can be useful for planning the allocation of total number of points actually scored divided by the
nursing manpower, to evaluate the efficacy in the use of total possible points, have been proposed to evaluate the
nursing workload use and to objectively classify ICUs effectiveness of the use of nursing workload resources
based on the amount (and not the complexity) of care [51]. A recent position statement by the European
provided [51]. Federation of Critical Care Nursing Associations recom-
mends that all units use such a system on a regular basis
Nine Equivalents of Nursing Manpower Use Score to monitor the efficiency of the use of nursing manpower
NEMS was derived from the TISS-28 with the aim of [58].
creating a simpler system that would be more widely
used [52]. Nursing activities are separated into nine Other uses of scoring systems
categories: basic monitoring, intravenous medication, In addition to their use in outcome prediction, organ
mechanical ventilatory support, supplementary ventila- function assessment, and nursing workload evaluation
tory care, single vasoactive medication, multiple vaso- discussed above, scoring systems have several other
active medication, dialysis techniques, specific inter- potential uses, including use in clinical trials for case-mix
ventions in the ICU, specific interventions outside the comparisons and use in the assessment and comparison
ICU. Each of these is awarded weighted points, giving a of ICU quality and performance.
maximum score of 56. NEMS has been validated in large
cohorts of ICU patients and is easy to use with almost no Clinical trials
interrater variability [53]. Again, this system can be used to Scoring systems are increasingly being incorporated into
evaluate the efficacy of nursing workload use at the ICU clinical trial design. Outcome prediction scores, such as
level so as to objectively classify ICUs based on the amount APACHE and SAPS, have been used for some time to
(and not only on the complexity) of care provided [51]. compare patient populations in clinical trials and even
for the identification of eligible patients for inclusion. The
Nursing Activities Score analysis of results from one recent randomized controlled
Based on the TISS-28, the Nursing Activities Score study [59], which showed improved outcomes in patients
(NAS) includes several additional nursing activities not with higher APACHE II scores, led to the drug under
necessarily related to the severity of illness of the patients investigation, drotrecogin alfa (activated), being licensed
[54]. The list of items was developed by consensus. The in the United States for use only in patients with severe
average time consumption of the activities was sepsis who are at a high risk of death, that is, those with
Vincent and Moreno Critical Care 2010, 14:207 Page 7 of 9
http://ccforum.com/14/2/207

an APACHE II score above 25. However, this is a be used to assess the severity of individual organ
controversial approach and these scores were not dysfunctions or to monitor patient progress over time.
designed for this purpose [60]. Although organ dysfunction scores correlate with
The realization that mortality alone is inadequate as an outcomes, this is not what they were developed for and
outcome measure for interventional studies in ICU outcome prediction should be left to scores such as the
patients has led many trials, especially in sepsis, to APACHE and SAPS systems. The workload scores
include an organ dysfunction score as part of ongoing complete the picture by offering information on how the
patient assessment so that effects on morbidity can also patient’s disease will impact on staffing requirement and
be evaluated. Increased economic pressure has also led to resource use. We envisage that, increasingly, all patients
greater concerns about cost-effectiveness of new and will be initially evaluated using a general outcome predic-
established interventions and nursing workload scores tion model computed on admission or within the first 24
are also being incorporated into clinical trial design, hours, then by repeated organ failure (for example,
particularly for interventions likely to impact on nursing SOFA) and nursing workload (for example, TISS-28)
workload. scores during their ICU stay. When used together, these
three approaches could provide a more accurate
Assessment of ICU performance indication of disease severity and prognosis, which could
Costs of care for an ICU patient have been estimated as be of help both to the clinician in charge of the patient
being three times the costs of care for a general ward and to the manager involved in resource allocation and
patient [61]. Monitoring ICU performance is, therefore, performance assessment.
increasingly important in the fight to control hospital
Abbreviations
expenses. While crude mortality data may offer some APACHE = Acute Physiology and Chronic Health Evaluation; LODS = Logistic
global guidance as to ICU performance, adjusting Organ Dysfunction Score; MODS = Multiple Organ Dysfunction Score;
mortality rates according to disease severity, by using MPM = Mortality Probability Model; NAS = Nursing Activities Score; NEMS =
Nine Equivalents of Nursing Manpower Use Score; SAPS = Simplified Acute
outcome prediction scores to calculate the standardized Physiology Score; SOFA = Sequential Organ Failure Assessment; TISS =
mortality ratio, can help improve quality assessment. Therapeutic Intervention Scoring System; WHD-94 = Weighted Hospital Days
Such severity-adjusted indicators can be used to assess scale.

performance of a single ICU over time or to compare Author details

several or more units. However, this approach has several 1
Department of Intensive Care, Erasme University Hospital, Route de Lennik
808, 1070 Brussels, Belgium. 2Department of Intensive Care, Hospital de St
limitations, including potential effects of pre-ICU
Antonio dos Capuchos, Centro Hospitalar de Lisboa Central, EPE, 1169-050
admission factors, implications of different ICU discharge Lisbon, Portugal.
policies [62], and effects of different patient case-mix and
Competing interests
hence disease severity between units or in the same unit The authors declare that they have no competing interests.
at different times [63]. Nevertheless, there are large
variations in risk-adjusted mortality rates among hospi- Published: 26 March 2010
tals [20] and repeated quality assessment may help References
determine the reasons underlying these differences and 1. Moreno RP: Outcome prediction in intensive care: why we need to
reinvent the wheel. Curr Opin Crit Care 2008, 14:483-484.
enable programs to be developed to improve 2. Moreno R, Jordan B, Metnitz P: The changing prognostic determinants in
performance. the critically ill patient. In 2007 Yearbook of Intensive care and Emergency
Medicine. Edited by Vincent JL. Heidelberg: Springer; 2007:899-907.
3. Knaus WA, Zimmerman JE, Wagner DP, Draper EA, Lawrence DE: APACHE-
Conclusions acute physiology and chronic health evaluation: a physiologically based
General illness severity scores are widely used in the ICU classification system. Crit Care Med 1981, 9:591-597.
to assess resource use, predict outcome, and characterize 4. Knaus WA, Draper EA, Wagner DP, Zimmerman JE: APACHE II: A severity of
disease classification system. Crit Care Med 1985, 13:818-829.
disease severity and degree of organ dysfunction. All the
5. Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, Sirio
scores were developed to be used in mixed groups of ICU CA, Murphy DJ, Lotring T, Damiano A, Harrell FE: The APACHE III prognostic
patients and their accuracy in subgroups of patients can system: Risk prediction of hospital mortality for critically ill hospitalized
be questioned; disease-specific scoring systems are adults. Chest 1991, 100:1619-1636.
6. Zimmerman JE, Wagner DP, Draper EA, Wright L, Alzola C, Knaus WA:
increasingly being developed. As ICU populations Evaluation of acute physiology and chronic health evaluation III
change and new diagnostic, therapeutic and prognostic predictions of hospital mortality in an independent database. Crit Care
techniques become available, all the scoring systems will Med 1998, 26:1317-1326.
7. Knaus WA, Wagner DP, Zimmerman JE, Draper EA: Variations in mortality
need to be updated. Importantly, the different scoring and length of stay in intensive care units. Ann Intern Med 1993, 118:753-761.
systems have different purposes and measure different 8. Zimmerman JE, Kramer AA, McNair DS, Malila FM: Acute Physiology and
parameters; we believe they should be seen as comple- Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for
today’s critically ill patients. Crit Care Med 2006, 34:1297-1310.
menting each other, rather than competing with one 9. Zimmerman JE, Kramer AA, McNair DS, Malila FM, Shaffer VL: Intensive care
another. For example, outcome prediction models cannot unit length of stay: Benchmarking based on Acute Physiology and
Vincent and Moreno Critical Care 2010, 14:207 Page 8 of 9
http://ccforum.com/14/2/207

Chronic Health Evaluation (APACHE) IV. Crit Care Med 2006, 34:2517-2529. 31. Vincent JL, Moreno R, Takala J, Willatts S, de Mendonça A, Bruining H, Reinhart
10. Le Gall J-R, Loirat P, Alperovitch A, Glaser P, Granthil C, Mathieu D, Mercier P, CK, Suter PM, Thijs LG: The SOFA (Sepsis-related Organ Failure Assessment)
Thomas R: A simplified acute physiology score for ICU patients. Crit Care score to describe organ dysfunction/failure. Intensive Care Med 1996,
Med 1984, 12:975-977. 22:707-710.
11. Le Gall J-R, Lemeshow S, Saulnier F: A new simplified acute physiology 32. Metnitz PG, Lang T, Valentin A, Steltzer H, Krenn CG, Le Gall JR: Evaluation of
score (SAPS II) based on a European/North American multicenter study. the logistic organ dysfunction system for the assessment of organ
JAMA 1993, 270:2957-2963. dysfunction and mortality in critically ill patients. Intensive Care Med 2001,
12. Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, Iapichino G, 27:992-998.
Edbrooke D, Capuzzo M, Le Gall JR: SAPS 3 - from evaluation of the patient 33. Timsit JF, Fosse JP, Troche G, De Lassence A, Alberti C, Garrouste-Org,
to evaluation of the intensive care unit. Part 2: Development of a Bornstain C, Adrie C, Cheval C, Chevret S: Calibration and discrimination by
prognostic model for hospital mortality at ICU admission. Intensive Care daily Logistic Organ Dysfunction scoring comparatively with daily
Med 2005, 31:1345-1355. Sequential Organ Failure Assessment scoring for predicting hospital
13. Rothen HU, Stricker K, Einfalt J, Bauer P, Metnitz PG, Moreno RP, Takala J: mortality in critically ill patients. Crit Care Med 2002, 30:2003-2013.
Variability in outcome and resource use in intensive care units. Intensive 34. Marshall JC: Multiple organ dysfunction syndrome. In Clincial Trials for the
Care Med 2007, 33:1329-1336. Treatment of Sepsis. Edited by Sibbald WJ, Vincent JL. Heidelberg: Springer-
14. Lemeshow S, Teres D, Pastides H, Avrunin JS, Steingrub JS: A method for Verlag; 1995:122-138.
predicting survival and mortality of ICU patients using objectively derived 35. Cook R, Cook D, Tilley J, Lee K, Marshall J: Multiple organ dysfunction:
weights. Crit Care Med 1985, 13:519-525. baseline and serial component scores. Crit Care Med 2001, 29:2046-2050.
15. Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, Rapoport J: Mortality 36. Moreno R, Vincent JL, Matos A, de Mendonça A, Cantraine F, Thijs J, Takala J,
Probability Models (MPM II) based on an international cohort of intensive Sprung C, Antonelli M, Bruining H, Willatts S: The use of maximum SOFA
care unit patients. JAMA 1993, 270:2478-2486. score to quantify organ dysfunction/failure in intensive care. Results of a
16. Rapoport J, Teres D, Lemeshow S, Gehlbach S: A method for assessing the prospective, multicentre study. Intensive Care Med 1999, 25:686-696.
clinical performance and cost-effectiveness of intensive care units: a 37. Ceriani R, Mazzoni M, Bortone F, Gandini S, Solinas C, Susini G, Parodi O:
multicenter inception cohort study. Crit Care Med 1994, 22:1385-1391. Application of the sequential organ failure assessment score to cardiac
17. Higgins TL, Teres D, Copes WS, Nathanson BH, Stark M, Kramer AA: Assessing surgical patients. Chest 2003, 123:1229-1239.
contemporary intensive care unit outcome: an updated Mortality 38. Lorente JA, Vallejo A, Galeiras R, Tomicic V, Zamora J, Cerda E, De La Cal MA,
Probability Admission Model (MPM0-III). Crit Care Med 2007, 35:827-835. Esteban A: Organ dysfunction as estmated by the SOFA score is related to
18. Nathanson BH, Higgins TL, Teres D, Copes WS, Kramer A, Stark M: A revised outcome in critically ill burned patients. Shock 2009, 31:125-131.
method to assess intensive care unit clinical performance and resource 39. Vosylius S, Sipylaite J, Ivaskevicius J: Sequential organ failure assessment
utilization. Crit Care Med 2007, 35:1853-1862. score as the determinant of outcome for patients with severe sepsis. Croat
19. Livingston BM, MacKirdy FN, Howie JC, Jones R, Norrie JD: Assessment of the Med J 2004, 45:715-720.
performance of five intensive care scoring models within a large Scottish 40. Vincent JL, de Mendonça A, Cantraine F, Moreno R, Takala J, Suter P, Sprung C,
database. Crit Care Med 2000, 28:1820-1827. Colardyn FC, Blecher S: Use of the SOFA score to assess the incidence of
20. Kuzniewicz MW, Vasilevskis EE, Lane R, Dean ML, Trivedi NG, Rennie DJ, Clay T, organ dysfunction/failure in intensive care units: Results of a multicentric,
Kotler PL, Dudley RA: Variation in ICU risk-adjusted mortality: impact of prospective study. Crit Care Med 1998, 26:1793-1800.
methods of assessment and potential confounders. Chest 2008, 41. Lopes Ferreira F, Peres Bota D, Bross A, Melot C, Vincent JL: Serial evaluation
133:1319-1327. of the SOFA score to predict outcome. JAMA 2001, 286:1754-1758.
21. Sirio CA, Shepardson LB, Rotondi AJ, Cooper GS, Angus DC, Harper DL, 42. Cabrè L, Mancebo J, Solsona JF, Saura P, Gich I, Blanch L, Carrasco G, Martin
Rosenthal GE: Community-wide assessment of intensive care outcomes MC: Multicenter study of the multiple organ dysfunction syndrome in
using a physiologically based prognostic measure: implications for critical intensive care units: the usefulness of Sequential Organ Failure
care delivery from Cleveland Health Quality Choice. Chest 1999, Assessment scores in decision making. Intensive Care Med 2005, 31:927-933.
115:793-801. 43. Pettilä V, Pettila M, Sarna S, Voutilainen P, Takkunen O: Comparison of
22. Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG: Austrian multiple organ dysfunction scores in the prediction of hospital mortality
validation and customization of the SAPS 3 Admission Score. Intensive Care in the critically ill. Crit Care Med 2002, 30:1705-1711.
Med 2009, 35:616-622. 44. Peres BD, Melot C, Lopes FF, Nguyen B, V, Vincent JL: The Multiple Organ
23. Sakr Y, Krauss C, Amaral AC, Rea-Neto A, Specht M, Reinhart K, Marx G: Dysfunction Score (MODS) versus the Sequential Organ Failure
Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their Assessment (SOFA) score in outcome prediction. Intensive Care Med 2002,
customized prognostic models in a surgical intensive care unit. Br J 28:1619-1624.
Anaesth 2008, 101:798-803. 45. Zygun DA, Laupland KB, Fick GH, Sandham JD, Doig CJ: Limited ability of
24. Harrison DA, Parry GJ, Carpenter JR, Short A, Rowan K: A new risk prediction SOFA and MOD scores to discriminate outcome: a prospective evaluation
model for critical care: the Intensive Care National Audit & Research in 1,436 patients. Can J Anaesth 2005, 52:302-308.
Centre (ICNARC) model. Crit Care Med 2007, 35:1091-1098. 46. Zygun D, Berthiaume L, Laupland K, Kortbeek J, Doig C: SOFA is superior to
25. Kramer AA, Zimmerman JE: Assessing the calibration of mortality MOD score for the determination of non-neurologic organ dysfunction in
benchmarks in critical care: The Hosmer-Lemeshow test revisited. Crit Care patients with severe traumatic brain injury: a cohort study. Crit Care 2006,
Med 2007, 35:2052-2056. 10:R115.
26. Miller ME, Hui SL, Tierney WM: Validation techniques for logistic regression 47. Cullen DJ, Civetta JM, Briggs BA, Ferrara LC: Therapeutic intervention
models. Stat Med 1991, 10:1213-1226. scoring system: a method for quantitative comparison of patient care. Crit
27. Poole D, Rossi C, Anghileri A, Giardino M, Latronico N, Radrizzani D, Langer M, Care Med 1974, 2:57-60.
Bertolini G: External validation of the Simplified Acute Physiology Score 48. Keene AR, Cullen DJ: Therapeutic intervention scoring system: Update
(SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care 1983. Crit Care Med 1983, 11:1-3.
units. Intensive Care Med 2009, 35:1916-1924. 49. Miranda DR, de Rijk A, Schaufeli W: Simplified Therapeutic Intervention
28. Bosman RJ, Oudemane van Straaten HM, Zandstra DF: The use of intensive Scoring System: the TISS-28 items - results from a multicenter study. Crit
care information systems alters outcome prediction. Intensive Care Med Care Med 1996, 24:64-73.
1998, 24:953-958. 50. Moreno R, Morais P: Validation of the simplified therapeutic intervention
29. Le Gall JR, Klar J, Lemeshow S, Saulnier F, Alberti C, Artigas A, Teres D, ICU scoring system on an independent database. Intensive Care Med 1997,
Scoring Group.: The logistic organ dysfunction system: A new way to 23:640-644.
assess organ dysfunction in the intensive care unit. JAMA 1996, 51. Moreno R, Reis MD: Nursing staff in intensive care in Europe: the mismatch
276:802-810. between planning and practice. Chest 1998, 113:752-758.
30. Marshall JC, Cook DJ, Christou NV, Bernard GR, Sprung CL, Sibbald WJ: 52. Reis MD, Moreno R, Iapichino G: Nine equivalents of nursing manpower use
Multiple organ dysfunction score: A reliable descriptor of a complex score (NEMS). Intensive Care Med 1997, 23:760-765.
clinical outcome. Crit Care Med 1995, 23:1638-1652. 53. Rothen HU, Kung V, Ryser DH, Zurcher R, Regli B: Validation of “nine
Vincent and Moreno Critical Care 2010, 14:207 Page 9 of 9
http://ccforum.com/14/2/207

equivalents of nursing manpower use score” on an independent data 60. Vincent JL, Opal SM, Marshall JC: Ten reasons why we should NOT use
sample. Intensive Care Med 1999, 25:606-611. severity scores as entry criteria for clinical trials or in our treatment
54. Miranda DR, Nap R, de Rijk A, Schaufeli W, Iapichino G: Nursing activities decisions. Crit Care Med 2010, 38:283-287.
score. Crit Care Med 2003, 31:374-382. 61. Cooper LM, Linde-Zwirble WT: Medicare intensive care unit use: analysis of
55. Padilha KG, Sousa RM, Kimura M, Miyadahira AM, da Cruz DA, Vattimo MF, incidence, cost, and payment. Crit Care Med 2004, 32:2247-2253.
Fusco SR, de Campos ME, Mendes EM, Mayor ER: Nursing workload in 62. Kahn JM, Kramer AA, Rubenfeld GD: Transferring critically ill patients out of
intensive care units: a study using the Therapeutic Intervention Scoring hospital improves the standardized mortality ratio: a simulation study.
System-28 (TISS-28). Intensive Crit Care Nurs 2007, 23:162-169. Chest 2007, 131:68-75.
56. Padilha KG, de Sousa RM, Queijo AF, Mendes AM, Reis MD: Nursing Activities 63. Glance LG, Osler T, Shinozaki T: Effect of varying the case mix on the
Score in the intensive care unit: analysis of the related factors. Intensive Crit standardized mortality ratio and W statistic: A simulation study. Chest
Care Nurs 2008, 24:197-204. 2000, 117:1112-1117.
57. Dickie H, Vedio A, Dundas R, Treacher DF, Leach RM: Relationship between 64. Moreno R, Metnitz P: Tools for the evaluation of patients and intensive care
TISS and ICU cost. Intensive Care Med 1998, 24:1009-1017. units. In Principles of Diagnosis and Management in the Adult. Edited by Parrillo
58. European Federation of Critical Care Nursing Associations: Position JE, Dellinger RP. Philadelphia: Mosby, Elsevier; 2008:1547-1565.
Statement on Workforce Requirements Within European Critical Care
Nursing. 2007 [http://www.efccna.org/downloads/Position%20
Statement%20Workforce%20EfCCNa%202007.pdf ]
59. Bernard GR, Vincent JL, Laterre PF, LaRosa SP, Dhainaut JF, Lopez-Rodriguez A,
doi:10.1186/cc8204
Steingrub JS, Garber GE, Helterbrand JD, Ely EW, Fisher CJ Jr: Efficacy and
Cite this article as: Vincent J-L, Moreno R: Scoring systems in the critically ill.
safety of recombinant human activated protein C for severe sepsis. N Engl Critical Care 2010, 14:207.
J Med 2001, 344:699-709.