Patient Safety Indicator PDF
Patient Safety Indicator PDF
The Agency for Healthcare Research and Quality (AHRQ) Quality Indicators (QIs) are one
Agency response to this need for multidimensional, accessible quality indicators. They include a family of
measures that providers, policy makers, and researchers can use with inpatient data to identify apparent
variations in the quality of inpatient or outpatient care. AHRQs Evidence-Based Practice Center (EPC) at
the University of California and Stanford University adapted, expanded, and refined these indicators
based on the original Healthcare Cost and Utilization Project (HCUP) Quality Indicators developed in the
early 1990s.
The new AHRQ QIs are organized into three modules, which are being published as a series:
Prevention Quality Indicators, Inpatient Quality Indicators, and Patient Safety Indicators. All three
modules are available and can be downloaded from AHRQs Web site at
http://www.qualityindicators.ahrq.gov/. The QIs were developed as an accessible and low-cost screening
tool to help organizations identify potential problems in quality of care and target promising areas for in-
depth review.
This third module focuses on potentially preventable complications and iatrogenic events for
patients treated in hospitals. The Patient Safety Indicators (PSIs) are measures that screen for adverse
events that patients experience as a result of exposure to the health care system; these events are likely
amenable to prevention by changes at the system or provider level. The PSIs include 20 hospital-level
and 6 area level indicators.
Full technical information on the first two modules can be found in Evidence Report for
Refinement of the HCUP Quality Indicators, prepared by the UCSF-Stanford EPC. It can be accessed at
AHRQs Web site. The technical report for the third module, entitled Evidence Report for Measures of
Patient Safety Based on Hospital Administrative DataThe Patient Safety Indicators, is also available on
AHRQs Web site.
Improving the quality of inpatient hospital services is a critical part of efforts to provide high quality
health care in the United States. This guide is intended to facilitate such efforts. As always, we would
appreciate hearing from those who use our measures and tools so that we can identify how they are
used, how they can be refined, and how we can measure and improve the quality of the tools themselves.
The programs for the Patient Safety Indicators (PSIs) can be downloaded from
http://www.qualityindicators.ahrq.gov/. Instructions on how to use the programs to calculate the PSI
rates are contained in the companion text, Patient Safety Indicators: Software Documentation.
We welcome your feedback. Support staff are available to answer your questions and respond to
comments. They can be reached at [email protected].
The following staff from the Evidence-based Practice Center (EPC) at UCSF-Stanford
performed the evidence review, completed the empirical evaluation, and created the programming code
and technical documentation for the new Quality Indicators:
The following staff from Social & Scientific Systems, Inc., developed this software product,
documentation, and guide:
We wish to also acknowledge the following individuals and organizations for their aid in this
report: Doug Staiger, Dept. of Economics, Dartmouth College; Ros McNally, National Primary Care
Research and Development Centre, University of Manchester; Rita Scichilone and the American Health
Information Management Association; the various professional organizations that provided nominatations
for our clinical review panels; the clinical panelists; the peer reviewers of the evidence report; and the
beta-testers of the software products, all of whose input was invaluable.
One approach to detecting, characterizing, and reporting potentially preventable adverse events
is to develop screening measures based on routinely collected administrative data. These data can be
used to identify indicators of potential problems that result from exposure to the health care system and
are likely to be prevented as a result of system-level changes.
Hospital administrative data offer a window into the medical care delivered in our nations
hospitals. These data, which are collected as a routine step in the delivery of hospital services, provide
information on patients diagnoses, procedures, age, gender, admission source, and discharge status.
From these data elements, it is possible to construct a picture of the qualityand safetyof medical
care. Although assessments based on administrative data cannot be definitive, they can be used to flag
potential safety problems and success stories, which can then be further investigated and studied.
Hospital associations, individual hospitals, purchasers, regulators, and policymakers at the local, State,
and Federal levels can use readily available hospital administrative data to begin the assessment of
patient safety.
The Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) are a
tool that takes advantage of hospital administrative data. The PSIs represent the current state-of-the-art
in measuring the safety of hospital care through analysis of inpatient discharge data.
The PSIs are a set of measures that can be used with hospital inpatient discharge data to provide
a perspective on patient safety. Specifically, PSIs screen for problems that patients experience as a
result of exposure to the healthcare system and that are likely amenable to prevention by changes at the
system or provider level. These are referred to as complications or adverse events. PSIs are defined on
two levels: the hospital level and the area level.
Hospital-level indicators provide a measure of the potentially preventable complication for patients
who received their initial care and the complication of care within the same hospitalization. Hospital-
level indicators include only those cases where a secondary diagnosis code flags a potentially
preventable complication.
Area-level indicators capture all cases of the potentially preventable complication that occur in a given
area (e.g., metropolitan service area or county) either during hospitalization or result in subsequent
hospitalization. Area-level indicators are specified to include principal diagnosis, as well as
secondary diagnoses, for the complications of care. This specification adds cases where a patients
risk of the complication occurred in a separate hospitalization.
1
Institute of Medicine. To Err is Human: Building a Safer Health System. Kohn LT, Corrigan JM, Donaldson MS
(eds.) Washington DC: National Academy Press, 2000.
In addition, the following PSIs were modified into area-level indicators to assess the total incidence of the
adverse event within geographic areas.
Widespread consensus exists that health care organizations can reduce patient injuries by
improving the environment for safetyfrom implementing technical changes, such as electronic medical
record systems, to improving staff awareness of patient safety risks. Clinical process interventions also
have strong evidence for reducing the risk of adverse events related to a patients exposure to hospital
care.2 PSIs, which are based on computerized hospital discharge abstracts from the AHRQs Healthcare
Cost and Utilization Project (HCUP), can be used to better prioritize and evaluate local and national
initiatives. Analyses of these and similar inexpensive, readily available administrative data sets may
provide a screen for potential medical errors and a method for monitoring trends over time. The scenario
on the following page illustrates one potential application of the PSIs.
This guide provides information that hospitals, State data organizations, hospital associations,
and others can use to decide how to use the PSIs. First, it describes the origin of the entire family of
AHRQ Quality Indicators. Second, it provides an overview of the methods used to identify, select, and
evaluate the AHRQ PSIs. Third, the guide summarizes the PSIs specifically, describes strengths and
limitations of the indicators, documents the evidence that links the PSIs to the quality of health care
services, and then provides in-depth two-page descriptions of each PSI. Finally, two appendices present
additional technical background information. Appendix A outlines the specific definitions of each PSI, with
complete ICD-9-CM coding specifications. Appendix B provides the details of the empirical methods used
to explore the PSIs.
A hospital association recognizes its member hospitals need for information that can
help them evaluate the quality of care they provide. There is significant interest in assessing,
monitoring and improving the safety of inpatient care. After learning about the AHRQ PSIs,
the association decides to apply the indicators to the discharge abstract data submitted by
individual hospitals. For each hospital, the association develops a report with graphic
presentation of the risk-adjusted data to show how the hospital performs on each indicator
compared to its peer group, the State as a whole, and other comparable States. National
and regional averages from the AHRQ Healthcare Cost and Utilization Project (HCUP)
database are also provided as additional external benchmarks. Three years of trend data
are included to allow the hospital to examine any changing patterns in its performance.
One member hospital, upon receiving the report, convenes an internal work group
comprised of clinicians and quality improvement professionals to review the information and
identify potential areas for improvement. The hospital leadership is committed to
performance excellence and providing a culture supportive of systems evaluation and
redesign. To begin their evaluation, they apply the AHRQ software to their internal
administrative data to distinguish those patients who experienced the complication or
adverse event from those who did not. This step establishes the focus for chart review.
After the initial analysis of the administrative and clinical data, the work group meets
with clinical departments involved in care of these patients. They begin an in-depth analysis
of the system and processes of care. Through application of process improvement
concepts, they begin to identify opportunities for improvement. After selection of their priority
area (for example, reduction of postoperative complications), they begin work, including:
Review and synthesize the evidence base and best practices from scientific literature.
Work with the multiple disciplines and departments involved in care of surgical patients to
redesign care based on best practices with an emphasis on coordination and
collaboration.
AHRQ developed these measures, called the HCUP Quality Indicators, to take advantage of a
readily available data sourceadministrative data based on hospital claimsand quality measures that
had been reported elsewhere.2 The 33 HCUP QIs included measures for avoidable adverse outcomes,
such as in-hospital mortality and complications of procedures; use of specific inpatient procedures
thought to be overused, underused, or misused; and ambulatory care sensitive conditions.
Although administrative data cannot provide definitive measures of health care quality, they can
be used to provide indicators of health care quality that can serve as the starting point for further
investigation. The HCUP QIs have been used to assess potential quality-of-care problems and to
delineate approaches for dealing with those problems. Hospitals with high rates of poor outcomes on the
HCUP QIs have reviewed medical records to verify the presence of those outcomes and to investigate
potential quality-of-care problems.3 For example, one hospital that detected high utilization rates for
certain procedures refined patient selection criteria for these procedures to improve appropriate
utilization.
Since the original development of the HCUP QIs, the knowledge base on quality indicators has
increased significantly. Risk adjustment methods have become more readily available, new measures
have been developed, and analytic capacity at the State level has expanded considerably. Based on
input from current users and advances to the scientific base for specific indicators, AHRQ funded a
project to refine and further develop the original QIs. The project was conducted by the UCSF-Stanford
EPC.
The major constraint placed on the UCSF-Stanford EPC was that the measures could require
only the type of information found in hospital discharge abstract data. Further, the data elements required
by the measures had to be available from most inpatient administrative data systems. Some State data
systems contain innovative data elements, often based on additional information from the medical record.
Despite the value of these record-based data elements, the intent of this project was to create measures
that were based on a common denominator discharge data set, without the need for additional data
collection. This was critical for two reasons. First, this constraint would result in a tool that could be used
with any inpatient administrative data, thus making it useful to most data systems. Second, this would
enable national and regional benchmark rates to be provided using HCUP data, since these benchmark
rates would need to be calculated using the universe of data available from the States.
2
Ball JK, Elixhauser A, Johantgen M, et al. HCUP Quality Indicators, Methods, Version 1.1: Outcome, Utilization, and
Access Measures for Quality Improvement. (AHCPR Publication No. 98-0035). Healthcare Cost and Utilization
project (HCUP-3) Research notes: Rockville, MD: Agency for Health Care Policy and Research, 1998.
3
Impact: Case Studies Notebook Documented Impact and Use of AHRQ's Research. Compiled by Division of
Public Affairs, Office of Health Care Information, Agency for Healthcare Research and Quality.
The work of the UCSF-Stanford EPC resulted in the AHRQ Quality Indicators, which are being
distributed as three separate modules:
Prevention Quality Indicators. These indicators consist of ambulatory care sensitive conditions,
hospital admissions that evidence suggests could have been avoided through high-quality outpatient
care or that reflect conditions that could be less severe, if treated early and appropriately.
Inpatient Quality Indicators. These indicators reflect quality of care inside hospitals and include
inpatient mortality; utilization of procedures for which there are questions of overuse, underuse, or
misuse; and volume of procedures for which there is evidence that a higher volume of procedures is
associated with lower mortality.
In approaching the task of evaluating patient safety indicators based on administrative data, the
project team developed a conceptual framework and standardized definitions of commonly used terms.
Standardized Definitions
In the literature, the distinctions between medical error, adverse events, complications of care,
and other terms pertinent to patient safety are not well established and are often used interchangeably.In
this report, the terms medical error, adverse events or complications, and similar concepts are defined as
follows:
Case finding indicators. Indicators for which the primary purpose is to identify specific cases in
which a medical error may have occurred, for further investigation.
Complication or adverse event. An injury caused by medical management rather than by the
underlying disease or condition of the patient.4 In general, adverse events prolong the
hospitalization, produce a disability at the time of discharge, or both. Used in this report,
complication does not refer to the sequelae of diseases, such as neuropathy as a complication
of diabetes. Throughout the report, sequelae is used to refer to these conditions.
Medical error. The failure of a planned action to be completed as intended (i.e., error of
execution) or the use of a wrong plan to achieve an aim (i.e., error of planning).1 The definition
includes errors committed by any individual, or set of individuals, working in a health care
organization.5
Patient safety. Freedom from accidental injury, or avoiding injuries or harm to patients from
care that is intended to help them. Ensuring patient safety involves the establishment of
operational systems and processes that minimize the likelihood of errors and maximizes the
likelihood of intercepting them when they occur. 6
Patient safety indicators. Specific quality indicators which also reflect the quality of care inside
hospitals, but focus on aspects of patient safety. Specifically, PSIs screen for problems that
patients experience as a result of exposure to the healthcare system, and that are likely
amenable to prevention by changes at the system or provider level.
4
Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and
negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med 1991;324(6):370-
6.
5
Institute of Medicine, 2000.
6
Envisioning the National Health Care Quality Report. Washington, DC: Institute of Medicine; 2001.
7
Brennan et al., 1991.
Quality. Quality of care is the degree to which health services for individuals and populations
increase the likelihood of desired health outcomes and are consistent with current professional
knowledge. In this definition, the term health services refers to a wide array of services that
affect health(and) applies to many types of health care practitioners (physicians, nurses, and
various other health professionals) and to all settings of care8
Quality indicators. Screening tools for the purpose of identifying potential areas of concern
regarding the quality of clinical care. For the purpose of this report, we focus on indicators that
reflect the quality of care inside hospitals. Quality indicators may assess any of the four system
components of health care quality, including patient safety (see below), effectiveness (i.e.,
providing services based on scientific knowledge to all who could benefit, and refraining from
providing services to those not likely to benefit), patient centeredness, and timeliness (i.e.,
minimizing unnecessary delays").9
Rate based indicators. Indicators for which the primary purpose is to identify the rate of a
complication rather than to identify specific cases.
While the definitions above are intended to distinguish events that are less preventable from
those that are more preventable, the difference is best described as a spectrum. To conceptualize this
spectrum, the project team developed the following three categories of conditions:
2. Conditions that are likely to reflect medical error. These conditions (for example, foreign body
accidentally left during a procedure) are likely to have been caused by medical error. Most of these
conditions appear infrequently in administrative data, and thus rates of events lack the precision to
allow for comparisons between providers. However, these conditions may be the subject of case-
finding indicators.
3. Conditions that conceivably, but not definitively reflect medical error. These conditions (for example,
postoperative DVT or PE) represent a spectrum of preventability between the previous two
categoriesfrom those that are mostly unpreventable to those that are mostly preventable. Because
of the uncertainty regarding the preventability of these conditions and the likely heterogeneity of
cases with the condition, indicators using these conditions are less useful as case-finding indicators.
However, examining the rate of these conditions may highlight potential areas of concern.
8
Measuring the Quality of Health Care: A statement of the National Roundtable on Healthcare Quality Division of
Healthcare Services: National Academy Press; 1999.
9
National Roundtable on Healthcare Quality, 1999.
To evaluate the soundness of each indicator, the project team applied the same framework as
was applied in the technical report10 for the Prevention Quality Indicators (PQIs) and Inpatient Quality
Indicators (IQIs). This included six areas of evidence:
Face validity. Does the indicator capture an aspect of quality that is widely regarded as important
and subject to provider or public health system control? Consensual validity expands face validity
beyond one person to the opinion of a panel of experts.
Minimum bias. Is there either little effect on the indicator of variations in patient disease severity and
comorbidities, or is it possible to apply risk adjustment and statistical methods to remove most or all
bias?
Construct validity. Does the indicator perform well in identifying true (or actual) quality of care
problems?
Fosters real quality improvement. Is the indicator insulated from perverse incentives for providers
to improve their reported performance by avoiding difficult or complex cases, or by other responses
that do not improve quality of care?
Application. Has the measure been used effectively in practice? Does it have potential for working
well with other indicators?
Face validity (consensual validity) was evaluated using a structured panel review, minimum bias was
explored empirically and briefly during the panel review, and construct validity was evaluated using the
limited literature available. A full discussion of this framework is available in the Stanford Technical
report.11
The relative importance of each of these evaluation areas may differ by individual PSIs..
Precision and minimum bias may be less important for indicators that are primarily designed to screen
only for medical error, since these events are relatively rare. In general, these indicators are better used
as case-finding indicators. For these indicators, comparisons between rates are less relevant. However,
for rate-based indicators, concerns of precision and minimum bias remain if indicators are used in any
comparison of rates (comparison to national averages, peer group, etc.).
The literature searches performed in connection with assessing potential AHRQ QIs12 identified
many references relevant to potential PSIs. In addition, the project team performed electronic searches
for articles published before February 2002 followed by hand searching the bibliographies of identified
references. Members of the project team were queried to supplement this list, based on their personal
10
Davies S, Geppert J, McClellan M, McDonald KM, Romano PS, Shojania KG. Refinement of the HCUP Quality
Indicators. Technical Review Number 4. Rockville, MD: (Prepared by UCSF-Stanford Evidence-based Practice
Center under Contract No. 290-97-0013) Agency for Healthcare Research and Quality; 2001. Report No.: 01-0035.
11
Davies et al., 2001.
12
Davies et al., 2001.
The project team identified 326 articles from the Medline search. Articles were screened using
both the titles and abstracts. To qualify for abstraction, an article must have described, evaluated, or
validated a potential indicator of medical errors, patient safety, or potentially preventable complications
based on International Classification for Diseases - Ninth Revision - Clinical Modifications (ICD-9-CM)
coded administrative (hospital discharge or claims) data. Some indicators were also considered if they
appeared to be readily translated into ICD-9-CM, even if the original authors did not use ICD-9-CM codes.
This search was adapted slightly and repeated using the OVID interface with EMBASE14, limited
to articles published from January 1990 through the end of first quarter 2002. The EMBASE search
identified 463 references, and these articles were screened in the same manner. After elimination of
articles that had already been identified using Medline15 and the other approaches described above, only
nine additional articles met the criteria for abstraction.
The project team developed a candidate list of PSIs by first reviewing the literature, then selecting
a subset of indicators to undergo face validity testing by clinician panels.
The literature search located relatively few patient safety indicators that could be defined using
unlinked administrative data. The majority of these indicators were from the Complications Screening
Program (CSP),16 which was developed to identify potentially preventable complications of adult medical
and surgical hospital care using commonly available administrative data. The algorithm uses discharge
abstract dataspecifically ICD-9-CM diagnosis and procedure codes, patient age, sex, diagnosis-related
group (DRG), and date of procedureto identify 28 complications that raise concern about the quality of
care based on the rate of such occurrences at individual hospitals. Each of the complications is applied to
some or all of the following specified risk pools separately: major surgery, minor surgery, invasive
cardiac procedure, endoscopy, medical patients, and all patients. In addition, specified inclusion and
exclusion criteria are applied to each complication to ensure that the complication developed in-hospital,
as opposed to being present on admission, and that the complication was potentially preventable.
Four later studies were designed to test criterion and construct validity by validating the data used
to construct CSP screens, validating the screens as a flag for actual quality problems, and validating the
replicability of hospital-level results using different data sources.17 18 19 20 These studies raised concerns
13
Iezzoni LI, Foley SM, Heeren T, Daley J, Duncan CC, Fisher ES, et al. A method for screening the quality of
hospital care using administrative data: preliminary validation results. QRB Qual Rev Bull 1992;18(11):361-71.
14
EMBASE. In. The Netherlands: Elsevier Science Publishers B.V.
15
MEDLINE [database online]. In. Bethesda (MD): National Library of Medicine.
16
Iezzoni et al., 1992.
17
Lawthers A, McCarthy E, Davis R, Peterson L, Palmer R, Iezzoni L. Identification of in-hospital complications from
claims data: is it valid? Medical Care 2000;38(8):785-795.
18
McCarthy EP, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamael MB, et al. Does clinical evidence support
The project team also reviewed all ICD-9-CM codes implemented in or before 1999 that were
identified by AHRQ as possibly describing medical errors or reflecting the consequences of such errors.21
(This initial set of indicators is referred to as the Miller et al. indicators.) The project team added relevant
codes from the 2000 and 2001 revisions of ICD-9-CM and selected codes from the CSP, such as those
not clearly reflective of medical error, but representing a potentially preventable complication. This
process was guided principally by conceptual considerations. For example, codes for postoperative AMI
(an evaluated indicator that was not included in the final indicator set) were included in the evaluation set
since recent evidence suggests that AMI is a potentially preventable complication.22 A few codes were
also deleted from the initial list based on a review of ICD-9-CM coding guidelines, described in Coding
Clinics for ICD-9-CM and the American Hospital Associations ICD-9-CM Coding Handbook. For example,
the code 2593 for hypoglycemic coma specifically excludes patients with diabetes mellitus, the population
for which this complication is most preventable. This process of updating the Miller et al. PSIs resulted in
a list of over 200 ICD-9-CM codes (valid in 2001) potentially related to medical error.
Codes identified in the CSP and updated from the Miller et. al. PSIs were then grouped into
indicators. Where feasible, codes were compiled as they were in the CSP, or in some cases the Miller et
al. PSIs, depending on which grouping yielded more clinically homogeneous groups. In most cases the
resulting indicators were not identical to the CSP indicators, although they were closely related, as some
of the specific codes included in the original CSP had been eliminated after the teams review of coding
guidelines. The remaining codes were then incorporated into the most appropriate CSP-based indicator,
or were grouped into clinically meaningful concepts to define novel indicators. Exclusion criteria were
added based on CSP methods and clinical judgment. As a result, over 40 patient safety indicators were
defined that, while building on prior work, reflected significantly changed measures to focus more
narrowly on the most preventable complications.
Indicators were defined with both a numerator (complication of interest) and a denominator
(population at risk). Different patient subpopulations have inherently different risks for developing a
complication, with some patients having almost no risk. Thus, the denominator for each indicator
represents the specific population at risk. The intention was to restrict the complication (and consequently
the rate) to a more homogeneous population who are actually at risk for that complication. In general, the
population at risk corresponded to one risk pool (e.g., major surgery) from the CSP, if applicable, or was
defined more narrowly.
Subset Selection
After the project team developed a list of potential indicators, they selected a subset of indicators
First, validation data from previous studies were reviewed and thresholds were set for retaining
CSP-based indicators. Four studies were identified that evaluated the CSP indicators. Three of these
studies, examined the predictive value of each indicator in identifying a complication that occurred in-
hospital, regardless of whether this complication was due to medical error or was preventable. 23 24 25 In a
fourth study, nurses identified specific process failures that may have contributed to complications. In
order to be retained as a potential PSI, at least one of the first three studies needed to demonstrate a
positive predictive value of at least 75%, meaning that 3 out of 4 patients identified by the measure did
indeed have the complication of interest.26 In addition, the positive predictive value of a "process failure"
identified in the fourth study needed to reach or exceed 46%, which was the average rate for surgical
cases that were not flagged by any of the CSP indicators. As a result, only CSP-derived indicators that
were at least somewhat predictive of objectively defined process failures or medical errors were retained.
Second, specific changes to previous definitions or constructs of indicators fell into the following
general categories:
1. Changes to the denominator definitions (inclusion or exclusion criteria), intended to reduce bias
due to the inclusion of atypical patients or to improve generalizability to a broader set of patients
at risk.
2. Elimination of selected ICD-9-CM codes from numerator definitions, intended to focus attention
on more clinically significant complications or complications more likely to result from medical
errors.
4. Division of a single indicator into two or more related indicators, intended to create more clinically
meaningful and conceptually coherent indicators.
A total of 34 indicators, intended to be applied to all age groups, were retained for face validity
testing by clinician panels. Because the primary intent in developing these indicators was to detect
potentially preventable complications related to health care exposure, the final definitions for this set of
indicators represented mostly new measures that built upon previous work.
Coding Review
Experts in ICD-9-CM codes reviewed each code for accuracy of capturing the complication and
population at risk. In some cases, additional codes or other refinements to the indicators were suggested
based on current coding guidelines.
23
Lawthers, et al., 2000.
24
McCarthy, et al., 2000.
25
Weingart et al., 2000.
26
Iezzoni et al., 1999.
The project team conducted a structured review of each indicator to evaluate the face validity
(from a clinical perspective) of the indicators. The methodology for the structured review was adapted
from the RAND/UCLA Appropriateness Method27 and consisted of an initial independent assessment of
each indicator by clinician panelists using an initial questionnaire, a conference call among all panelists,
followed by a final independent assessment by clinician panelists using the same questionnaire. The
review sought to establish consensual validity, which extends face validity from one expert to a panel of
experts who examine and rate the appropriateness of each item.28 The panel process served to refine
definitions of some indicators, add new measures, and dismiss indicators with major concerns from
further consideration.
Eight panels were formed: two panels examined complications of medical care indicators, three
panels examined surgical complications indicators, one panel assessed indicators related to procedural
complications, and two panels examined obstetric complications indicators.
Panelists were presented with four or five indicators, including the standardized text used to
describe each ICD-9-CM code, the specific numeric code, exclusion and inclusion criteria, the clinical
rationale for the indicator, and the specification criteria. For each indicator, panelists completed a 10-item
questionnaire that evaluated the ability of the indicator to screen out conditions present on admission, the
potential preventability of the complication, and the ability of the indicator to identify medical error. In
addition, the questionnaire asked panelists to consider potential bias, reporting or charting problems,
potential for gaming the indicator, and adverse effects of implementing the indicator. Finally, the
questionnaire provided an opportunity for panelists to suggest changes to the indicator.
After the panelists submitted the initial evaluation questionnaires, they participated in a 90-minute
conference call for their panel to discuss the indicators. In general, agenda items for the conference call
focused on points of disagreement among panelists. However, panelists were explicitly told that
consensus was not the goal of discussion. In some cases, panelists agreed on proposed changes to the
indicator definitions, and such consensus was noted and the definition was modified accordingly before
the final round of rating.
Panelists were prompted throughout the process to consider the appropriate population at risk for
each indicator (specifically inclusion and exclusion criteria) in addition to the complication of interest.
However, if panelists wished to discuss other aspects of the indicator, this discussion was allowed within
the time allotted for that indicator (approximately 15 minutes). If time remained at the end of a call, topics
that were not fully addressed previously were revisited.
27
Fitch K, Bernstein J, Aguilar MD, Burnand B, LaCalle JR, Lazaro P, et al. the RAND/UCLA Appropriateness
Method Users Manual: RAND; 2001.
28
Green L, Lewis F. measurement and Evaluation in Health Education and Health Promotion. Mountain View, CA:
Mayfield Publishing Company; 1998.
Results from the final evaluation questionnaire were used to calculate median scores from the 9-
point scale for each question and to categorize the degree of agreement among panelists. Median scores
determined the level of acceptability of the indicator, and dispersion of ratings across the panel for each
applicable question determined the agreement status. Therefore the median and agreement status were
independent measurements for each question. Six criteria were used to identify the panel opinions (i.e.,
median, agreement status category) on the following aspects of the indicator:
The project team used the ratings of the overall appropriateness of each indicator to assess its
overall usefulness as a screen for potential patient safety problems. Indicators were triaged into three
sets: Accepted Indicators (described in this guide), Experimental Indicators, and Rejected Indicators.
The project team conducted empirical analyses to explore the frequency and variation of the
indicators, the potential bias, based on limited risk adjustment, and the relationship between indicators.
The data sources used in the empirical analyses were the 1997 Florida State Inpatient Database (SID) for
initial testing and development and the 1997 HCUP State Inpatient Database for 19 States (referred to in
this guide as the HCUP SID) for the final empirical analyses. The rates presented in the Detailed
Evidence Section of this guide, as well as the means and parameter reference files used by the PSI
software, reflect analyses of the 2000 HCUP SID for 29 states.
All potential indicators were examined empirically by developing and conducting statistical tests
for precision, bias, and relatedness of indicators. Three different estimates of hospital performance were
calculated for each indicator:
1. The raw indicator rate was calculated using the number of adverse events in the numerator divided
by the number of discharges in the population at risk by hospital.
2. The raw indicator was adjusted to account for differences among hospitals in age, gender, modified
DRG, and comorbidities.
Adjacent DRG categories that were separated by the presence or absence of comorbidities or
complications were collapsed to avoid adjusting for the complication being measured. Most of
the super-Major Diagnostic Category (MDC) DRG categories were excluded for the same reason.
APR-DRG risk adjustment was not implemented because removing applicable complications from
each indicator was beyond the scope of this project.
The ICD-9-CM codes used to define comorbidity categories were modified to exclude conditions
likely to represent potentially preventable complications in certain settings.
3. Multivariate signal extraction methods were applied to adjust for reliability by estimating the amount of
noise (i.e., variation due to random error) relative to the amount of signal (i.e., systematic variation
in hospital performance or reliability) for each indicator.
Similar reliability adjustment has been used in the literature for similar purposes.29 30 The project team
constructed a set of statistical tests to examine precision, bias, and relatedness of indicators for all
accepted hospital-level indicators, and precision and bias for all accepted area-level indicators. It should
be noted that rates based on fewer than 30 cases in the numerator or the denominator are not reported.
This exclusion rule serves two purposes:
29
Hofer TP, Hayward RA, Greenfield S, Wagner EH, Kaplan SH, Manning WG. The unreliability of individual
physician report cards for assessing the costs and quality of care of a chronic disease JAMA 1999;281(22):2098-
105.
30
Christiansen CL, Morris CN. Improving the statistical approach to health care provider profiling. Ann Intern Med
1997;127(8 Pt 2):764-8.
This project took a four-pronged approach to the identification, development, and evaluation of
PSIs that included use of literature, clinician panels, expert coders, and empirical analyses. The literature
review and the findings from the clinical panels combined with data analysis provide evidence to suggest
that a number of discharge-based PSIs may be useful screens for organizations, purchasers, and
policymakers to identify safety problems at the hospital level, as well as to document systematic area-
level differences in patient safety problems.
Most adverse events identified by the PSIs have a variety of causes in addition to potential
medical error leading to the adverse event, including underlying patient health and factors that do not vary
systematically. Clinician panelists rated only two of the accepted indicators as very likely to reflect
medical error: (1) transfusion reaction and (2) foreign body left in during a procedure. These indicators
proved to be very rare, with less than 1 per 10,000 cases at risk.
Table 1 summarizes the results of the literature review, clinician panels, and empirical analyses
on the hospital-level PSIs. The table lists each indicator, provides its definition, identifies any concerns
about its validity based on the clinician panels, and summarizes the strength of evidence in the literature
for each indicator.
The following notes about some of the terms in the table are intended to help the reader
understand the context in which they are used.
Validity Concerns. The following concerns, raised during our panel review, are listed if they affect the
validity of the particular indicator:
Rare This indicator is relatively rare and may not have adequate statistical power for some
providers.
Condition definition varies This indicator includes conditions for which diagnosis may be
subjective, depending on the threshold of the physician, and patients with the same clinical state
may not have the same diagnosis.
Underreporting or screening Conditions included in this indicator may not be systematically
reported (leading to an artificially low rate) or may be routinely screened for (leading to a higher
rate in facilities that screen).
Adverse consequences Use of this indicator may have undesirable effects, such as increasing
inappropriate antibiotic use.
Stratification suggested This indicator includes some high risk patient groups and stratification is
recommended when examining rates,
Unclear preventability As compared to other PSIs, the conditions included in this indicator may
be less preventable by the health system.
Heterogeneous severity This indicator includes codes that encompass several levels of severity
of a condition that cannot be ascertained by the codes.
Case mix bias This indicator was felt to be particularly subject to systematic bias, and DRG and
comorbidity risk adjustment may not adequately address the concern.
Denominator unspecific The denominator for this indicator is less than ideal, because the true
population at risk could not be identified using ICD-9-CM codes. Some patients are likely
included who are not truly at risk, or some patients who are at risk are not included.
Empirical Performance. The performance of each indicator is measured for the following:
Rate The rate measures the number of adverse events per 1,000 population at risk. Rates
represent the average rate of the indicator for a nationwide sample of hospitals.
Deviation Standard deviation is an estimate of systematic variation. For the PSIs, standard
deviation is reported between providers.
Bias Bias represents the degree to which the results may be influenced by outside factors. Bias
Strength of Evidence. The following key findings represent a review of the limited literature assessing
the validity of the indicators:
Coding Sensitivity is the proportion of patients who suffered an adverse event, based on detailed
chart review or prospective data collection, for whom that event was coded on a discharge
abstract or Medicare claim. Predictive value is the proportion of patients with a coded adverse
event who were confirmed as having suffered that event, based on detailed chart review or
prospective data collection.
Construct, explicit process Adherence to specific, evidence-based or expert-endorsed
processes of care, such as appropriate use of diagnostic modalities and effective therapies. The
construct is that hospitals that provide better processes of care should experience fewer adverse
events.
Construct, implicit process Adherence to the standard of care for similar patients, based on
global assessment of quality by physician chart reviewers. The construct is that hospitals that
provide better overall care should experience fewer adverse events.
Construct, staffing The construct is that hospitals that offer more nursing hours per patient day,
better nursing skill mix, better physician skill mix, or more experienced physicians should have
fewer adverse events.
Published evidence suggests that the indicator lacks validity in this domain (i.e., less than 50%
sensitivity or predictive value; explicit or implicit process failure rates no more frequent than
among control patients).
The following distinctions were used to summarize the strength of the published evidence for
each indicator:
0 No published evidence regarding this domain of validity.
Published evidence suggests that the indicator may be valid in this domain, but different
studies offer conflicting results (although study quality may account for these conflicts).
+ Published evidence suggests that the indicator is valid, or is likely to be valid, in this domain
(i.e., one favorable study).
++ There is strong evidence supporting the validity of this indicator in this domain (i.e., multiple
studies with consistent results, or studies showing both high sensitivity and high predictive value).
When content validity is exceptionally high, as for transfusion reaction or iatrogenic
pneumothorax, construct validity becomes less important.
A complete description of each PSI is included later in the guide under Detailed Evidence for
Patient Safety Indicators and in Appendix A. Details on the empirical methods can be found in Appendix
B.
a
DRGs that are divided into with complications and comorbidities and without complications and comorbidities are
only included if both divisions have mortality rates below 0.5%.
Many important concerns cannot currently be monitored well using administrative data, such as
adverse drug events, and using these data tends to favor specific types of indicators. For example, the
PSIs evaluated in this report contain a large proportion of surgical indicators, rather than medical or
psychiatric, because medical complications are often difficult to distinguish from comorbidities that are
present on admission. In addition, medical populations tend to be more heterogeneous than surgical,
especially elective surgical populations, making it difficult to account for case-mix. Panelists often
expressed that indicators were more applicable to patient safety when limited to elective surgical
admissions. However, the careful use of administrative data holds promise for screening to target further
Two broad areas of concern also hold true for these data sets.
1. Questions about the clinical accuracy of discharge-based diagnosis coding lead to concerns about
the interpretation of reported diagnoses that may represent safety problems. Specifically:
Administrative data are unlikely to capture all cases of a complication, regardless of the
preventability, without false positives and false negatives (sensitivity and specificity).
When the codes are accurate in defining an event, the clinical vagueness inherent in the
description of the code itself (e.g., hypotension), may lead to a highly heterogeneous pool of
clinical states represented by that code.
Incomplete reporting is an issue in the accuracy of any data source used for identifying patient
safety problems, as medical providers might fear adverse consequences as a result of full
disclosure in potentially public records such as discharge abstracts.
2. The information about the ability of these data to distinguish adverse events in which no error
occurred from true medical errors is limited. A number of factorssuch as the heterogeneity of
clinical conditions included in some codes, lack of information about event timing available in these
data sets, and limited clinical detail for risk adjustmentcontribute to the difficulty in identifying
complications that represent medical error or may be at least in some part preventable.
These factors may exist for other sources of patient safety data as well. For example, they have been
raised in the context of the Joint Commission on Accreditation of Healthcare Organizations (JCAHO)
implementation of a sentinel event program geared at identifying serious adverse events that may be
related to underlying safety problems.
The initial validation evaluations reviewed and performed for the PSIs leave substantial room for
further research with detailed chart data and other data sources. Future validation work should focus on
the following:
The sensitivity and specificity of these indicators in detecting the occurrence of a complication.
The extent to which failures in processes of care at the system or individual level are detected using
these indicators.
The relationship of these indicators with other measures of quality, such as mortality.
Further explorations of bias and risk adjustment.
Enhancements to administrative data are worth exploring in the context of further validation studies that
use data from other sources. For example, as with other quality indicators, the addition of timing variables
may prove particularly useful in identifying whether a complication was present on admission, or whether
it occurred during the hospitalization. While some of the complications that are present on admission may
indeed reflect adverse events of care in a previous hospitalization or outpatient care, many may reflect
comorbidities instead of complications. A second example arealinking hospital data over time and with
outpatient data and other hospitalizationswould allow inclusion of complications that occur after
discharge and likely would increase the sensitivity of the PSIs.
Indicator
Number
Indicator Name Use of External Cause-of-Injury Codes
(used in
software)
15 & 25 Accidental puncture or laceration Required. Used in both the numerator and
denominator definitions.
17 Birth trauma Not used.
1 Complications of anesthesia Required. Used in the numerator definition.
2 Death in low mortality DRGs Not used.
3 Decubitus ulcer Not used.
4 Failure to rescue Not used.
5 & 21 Foreign body left during Required. Used in the numerator definition
procedure although the other ICD-9 CM codes may capture
the same information.
6 & 22 Iatrogenic pneumothorax Not used.
20 Obstetric trauma cesarean Not used.
section
18 Obstetric trauma vaginal with Not used.
instrument
19 Obstetric trauma vaginal without Not used.
instrument
9 Post-operative hemorrhage or Not used.
hematoma
8 Post-operative hip fracture Used as exclusion criteria in denominator
population.
10 Post-operative physiologic and Not used.
metabolic derangements
12 Post-operative pulmonary Not used.
embolism or deep vein
thrombosis
11 Post-operative respiratory failure Not used.
13 Post-operative sepsis Not used.
14 & 24 Post-operative wound dehiscence Not used.
7 & 23 Selected infections due to Not used.
medical care
16 & 26 Transfusion reaction Required. Used in the numerator definition
although the other ICD-9 CM codes may capture
the same information.
The two-page descriptions for each indicator also include a more detailed discussion of the panel review,
the literature review, the source of the indicator, and the results of the empirical analysis, including
information related to adjustments to increase the robustness of the rates:
Reliability. Statistics on the signal standard deviation, signal share, and signal ratio were used to
examine the effect of the reliability adjustment. Multivariate methods were applied to most of the
indicators, and overall the reliability adjustment reduced the hospital-level variation dramatically. In
general, indicators with higher rates tend to perform better on tests of reliability, as a result, obstetric
indicators with high rates tend to do very well relative to other indicators.
Bias. The effect of age, gender, DRG, and comorbidity risk adjustment on the relative ranking of
hospitals compared to no risk adjustment was assessed, if applicable. The presence of high bias
suggests that risk adjustment, using administrative data elements, is necessary to interpret hospital-
level differences in the rates of these indicators.
A full report on the literature review and empirical evaluation can be found in Evidence Report for
Measures of patient Safety Based on Hospital Administrative Data The Patient Safety Indicators by the
UCSF-Stanford EPC, available at http:www.qualityindicators.ahrq.gov/. Detailed coding information
for each PSI is provided in Appendix A.
The software manual Patient Safety Indicators: SAS Software Documentation, Version 2.1 (also available
at http:www.qualityindicators.ahrq.gov) provides detailed instructions on how to use the PSI software
including data preparation, calculation of the PSI rates, and interpretation of output. All hospital level
indicators are expressed as rates per 1,000 discharges. To obtain the standardized rate for each
hospital level PSIs, the output of the software should be multiplied by 1,000. The area level indicators are
expressed as rates per 100,000 population. To obtain the standardized area rate for each area level
PSIs, the output of the software should be multiplied by 100,000.
Source
31
Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher
ES, Duncan C, et al. Identifying complications of care
using administrative data. Med Care 1994;32(7):700-
15.
Definition In-hospital deaths per 1,000 patients in DRGs with less than 0.5% mortality.
Numerator Discharges with disposition of deceased per 1,000 population at risk.
Denominator Patients in DRGs with less than 0.5% mortality rate, based on NIS 1997
low-mortality DRG. If a DRG is divided into without/with complications,
both DRGs must have mortality rates below 0.5% to qualify for inclusion.
Exclude patients with any code for trauma, immunocompromised state, or
cancer.
Type of Indicator Hospital level
Empirical Performance Rate: 0.66 per 1,000 population at risk
Bias: Substantial bias; should be risk-adjusted
Risk Adjustment Age, sex, DRG, comorbidity categories
This indicator is intended to identify in-hospital Based on two-stage implicit review of randomly
deaths in patients unlikely to die during selected deaths, Hannan et al. found that
hospitalization. The underlying assumption is patients in low-mortality DRGs (<0.5%) were 5.2
that when patients admitted for an extremely times more likely than all other patients who died
low-mortality condition or procedure die, a health (9.8% versus 1.7%) to have received care that
care error is more likely to be responsible. departed from professionally recognized
Patients experiencing trauma or having an standards, after adjusting for patient
immunocompromised state or cancer are demographic, geographic, and hospital
excluded, as these patients have higher non- characteristics.32 In 15 of these 26 cases (58%)
preventable mortality. of substandard care, the patients death was
attributed at least partially to that care. The
Panel Review association with substandard care was stronger
for the DRG-based definition of this indicator
The overall usefulness of this indicator was than for the procedure-based definition (5.7%
rated as favorable by panelists. Because the versus 1.7%, OR=3.2). The project team was
denominator includes many heterogeneous unable to find other evidence on the validity of
patients cared for by different services, this this indicator.
indicator should be stratified by DRG type (i.e.,
medical, surgical, psychiatric, obstetric, Empirical Analysis
pediatric) when used as an indicator of quality.
The project team conducted extensive empirical
Panelists noted that hospital case-mix may analyses on the PSIs. Death in low-mortality
affect the rate of death in low mortality DRGs, DRGs generally performs well on several
and that patients referred from skilled nursing different dimensions, including reliability, bias,
facilities, those with certain comorbidities, and relatedness of indicators, and persistence over
older patients may be at higher risk of dying. time.
They advocated risk adjustment for
comorbidities and age. Reliability. The signal ratiomeasured by the
proportion of the total variation across hospitals
Panelists advocated that this indicator not be that is truly related to systematic differences
subject to public reporting because of the
32
potential bias and questions about the extent of Hannan EL, Bernard HR, ODonnell JF, Kilburn H,
preventability. Jr. A methodology for targeting hospital cases for
quality of care record reviews. Am J Public Health
1989;79(4):430-6.
Version 2.1 25 Revision 1 (May 28, 2003)
(signal) in hospital performance rather than
random variation (noise)is high, relative to
other indicators, at 94.2%, suggesting that
observed differences in risk-adjusted rates likely
reflect true differences across hospitals.
Source
33
Hannan et al. 1989.
Definition Cases of decubitus ulcer per 1,000 discharges with a length of stay greater
than 4 days.
Numerator Discharges with ICD-9-CM code of 7070 in any secondary diagnosis field
per 1,000 discharges.
Denominator All medical and surgical discharges defined by specific DRGs.
Include only patients with a length of stay of 5 or more days.
Exclude patients in MDC-9 or patients with any diagnosis of hemiplegia,
paraplegia, or quadriplegia.
Exclude obstetrical patients in MDC 14.
Exclude patients admitted from a long-term care facility.
Type of Indicator Hospital level
Empirical Performance Rate: 22.7 per 1,000 population at risk
Bias: Substantial bias; should be risk-adjusted
Risk Adjustment Age, sex, DRG, comorbidity categories
38 41
Lichtig LK, Knauf RA, Hilholland DK. Some impacts Nursing-Sensitive Quality Indicators for Acute Care
of nursing on acute care hospital outcomes. J Nurs Settings and ANAs Safety & Quality Initiative. In:
Adm 1999;29(2):25-33. American Nurses Association; 1999.
Version 2.1 28 Revision 1 (May 28, 2003)
Failure to Rescue
Definition Deaths per 1,000 patients having developed specified complications of care
during hospitalization.
Numerator Discharges with a disposition of deceased per 1,000 population at risk.
Denominator Discharges with potential complications of care listed in failure to rescue
definition (i.e., pneumonia, DVT/PE, sepsis, acute renal failure,
shock/cardiac arrest, or GI hemorrhage/acute ulcer). Exclusion criteria
specific to each diagnosis.
Exclude patients age 75 years and older.
Exclude neonatal patients in MDC 15.
Exclude patients transferred to an acute care facility.
Exclude patients transferred from an acute care facility.
Exclude patients admitted from a long-term care facility.
Type of Indicator Hospital level
Empirical Performance Rate: 148.4 per 1,000 population at risk
Bias: Substantial bias; should be risk-adjusted
Risk Adjustment Age, sex, DRG, comorbidity categories
Empirical Analysis
Source
48
Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher
ES, Duncan C, et al. Identifying complications of care
using administrative data. Med Care 1994;32(7):700-
15.
49
Johantgen M, Elixhauser A, Bali JK, Goldfarb M,
Harris DR. Quality indicators using hospital discharge
data: state and national applications. Jt Comm J Qual
Improv 1998;24(2):88-105.
50
Miller M, Elixhauser A, Zhan C, Meyer G. Patient
safety indicators: Using administrative data to identify
potential patient safety concerns. Health Services
Research 2001;36(6 Part II):110-132.
Version 2.1 32 Revision 1 (May 28, 2003)
Iatrogenic Pneumothorax
Hospital Level Definition
Definition Cases of iatrogenic pneumothorax per 1,000 discharges.
Numerator Discharges with ICD-9-CM code of 512.1 in any secondary diagnosis field
per 1,000 discharges.
Denominator All discharges.
Exclude patients with any diagnosis of trauma.
Exclude patients with any code indicating thoracic surgery or lung or pleural
biopsy or assigned to cardiac surgery DRGs.
Exclude obstetrical patients in MDC 14.
Type of Indicator Hospital level
Empirical Performance Rate: 0.83 per 1,000 population at risk
Bias: Some bias demonstrated
Risk Adjustment Age, sex, DRG, comorbidity categories
Iatrogenic Pneumothorax
Area Level Definition
Definition Cases of iatrogenic pneumothorax per 100,000 population.
Numerator Discharges with ICD-9-CM code of 512.1 in any diagnosis field (principal or
secondary).
Exclude patients with any diagnosis of trauma.
Exclude patients with any code indicating thoracic surgery or lung or pleural
biopsy or assigned to cardiac surgery DRGs.
Exclude obstetrical patients in MDC 14.
Denominator Population of county or MSA associated with FIPS code of patients
residence or hospital location.
Type of Indicator Area level
Empirical Performance Rate: 8.15 per 100,000 population
Risk Adjustment No risk adjustment
Iatrogenic pneumothorax excludes all trauma Panelists rated the overall usefulness of this
patients because these patients may be more indicator favorably. The denominator of the
Panelists expressed concern that some The signal standard deviation for this indicator is
approaches of placing a central line (e.g., lower than many indicators, at 0.00143,
subclavian) may be more likely to result in indicating that the systematic differences (signal)
pneumothorax than other approaches (e.g., among hospitals is low and less likely
internal jugular). However, other associated with hospital characteristics. The
complicationssuch as complications of the signal share is lower than many indicators, at
carotid arterywould be more common with 0.00183. The signal share is a measure of the
internal jugular approaches. Thus, if providers share of total variation (hospital and patient)
simply change approach, they may have a accounted for by hospitals. The lower the share,
decrease in pneumothorax but an increase in the less important the hospital in accounting for
other unmeasured complications. the rate and the more important other potential
factors (e.g., patient characteristics).
Literature Review
Minimum bias. The project team assessed the
The literature review focused on the validity of effect of age, gender, DRG, and comorbidity risk
complication indicators based on ICD-9-CM adjustment on the relative ranking of hospitals
diagnosis or procedure codes. Results of the compared to no risk adjustment. They
literature review indicate no published evidence measured (1) the impact of adjustment on the
for the sensitivity or predictive value of this assessment of relative hospital performance, (2)
indicator based on detailed chart review or the relative importance of the adjustment, (3) the
prospective data collection. Sensitivity is the impact on hospitals with the highest and lowest
proportion of the patients who suffered an rates, and (4) the impact throughout the
adverse event for whom that event was coded distribution. The detected bias for Iatrogenic
on a discharge abstract or Medicare claim. pneumothorax is moderate, indicating that the
Predictive value is the proportion of patients with measures may or may not be substantially
a coded adverse event who were confirmed as biased based on the characteristics observed.
having suffered that event.
Source
The project team found no published evidence
for this indicator that supports the following This diagnosis code was proposed by Miller et
constructs: (1) that hospitals that provide better al. as one component of a broader indicator
processes of care experience fewer adverse (iatrogenic conditions) in the Patient Safety
events; (2) that hospitals that provide better Indicator Algorithms and Groupings.51 It was
overall care experience fewer adverse events; also included as one component of a broader
and (3) that hospitals that offer more nursing indicator (adverse events and iatrogenic
hours per patient day, better nursing skill mix, complications) in AHRQs Version 1.3 HCUP
better physician skill mix, or more experienced Quality Indicators.
physicians have fewer adverse events.
51
Miller M, Elixhauser A, Zhan C, Meyer G. Patient
Empirical Analysis safety indicators: Using administrative data to identify
potential patient safety concerns. Health Services
Research 2001;36(6 Part II):110-132.
Version 2.1 34 Revision 1 (May 28, 2003)
Selected Infections Due to Medical Care
Hospital Level Definition
Definition Cases of ICD-9-CM codes 9993 or 99662 per 1,000 discharges.
Numerator Discharges with ICD-9-CM code of 9993 or 99662 in any secondary
diagnosis field per 1,000 discharges.
Denominator All medical and surgical discharges defined by specific DRGs.
Exclude patients with any diagnosis code for immunocompromised state or
cancer.
Type of Indicator Hospital level
Empirical Performance Rate: 1.99 per 1,000 population at risk
Bias: Some bias demonstrated
Risk Adjustment Age, sex, DRG, comorbidity categories
Summary infections.
This indicator is intended to capture cases of Coding validity. The original CSP definition had
hemorrhage or hematoma following a surgical a relatively high confirmation rate among major
procedure. This indicator limits hemorrhage and surgical cases (83% by coders review, 57% by
hematoma codes to secondary procedure and physicians review, 52% by nurse-abstracted
diagnosis codes, respectively, to isolate those clinical documentation, and 76% if nurses also
hemorrhages that can truly be linked to a accepted physicians notes as adequate
surgical procedure. documentation).55 56 57 Hartz and Kuhn
estimated the validity of hemorrhage codes
Panel Review using a gold standard based on transfusion
requirement. 58 They identified only 26% of
Panelists noted that some patients may be at
55
higher risk for developing a postoperative Lawthers A, McCarthy E, Davis R, Peterson L,
hemorrhage or hematoma. Specifically, they Palmer R, Iezzoni L. Identification of in-hospital
were concerned about patients with complications from claims data: Is it valid? Med Care
coagulopathies and those on anticoagulants. 2000;38(8):785-795.
They suggested that where possible, this 56
indicator be stratified for patients with underlying McCarthy EP, Iezzoni LI, Davis RB, Palmer RH,
clotting differences. They also noted that Cahalane M, Hamel MB, et al. Does clinical evidence
support ICD-9-CM diagnosis coding of complications?
patients admitted for trauma may be at a higher Med Care 2000;38(8):868-876.
risk for developing postoperative hemorrhage or
may have a hemorrhage diagnosed that 57
Weingart SN, Iezzoni LI, Davis RB, Palmer RH,
occurred during the trauma. They also Cahalane M, Hamel MB, et al. Use of administrative
suggested that this indicator be stratified for data to find substandard care: Validation of the
trauma and non-trauma patients. Complications Screening Program. Med Care
2000;38(8):796-806.
58
Hartz AJ, Kuhn EM. Comparing hospitals that
perform coronary artery bypass surgery: The effect of
Version 2.1 37 Revision 1 (May 28, 2003)
episodes of bleeding (defined as requiring return among hospitals is low and less likely
to surgery or transfusion of at least six units of associated with hospital characteristics. The
blood products) by applying this indicator (9981) signal share is lower than many indicators, at
to Medicare patients who underwent coronary 0.00006. The signal share is a measure of the
artery bypass surgery; the predictive value was share of total variation (hospital and patient)
75%. accounted for by hospitals. The lower the share,
the less important the hospital in accounting for
Construct Validity. Explicit process of care the rate and the more important other potential
failures in the CSP validation study were factors (e.g., patient characteristics).
relatively frequent among major surgical cases
with CSP 24, but not among medical cases Minimum bias. The project team assessed the
(66% and 13%, respectively), after excluding effect of age, gender, DRG, and comorbidity risk
patients who had hemorrhage or hematoma at adjustment on the relative ranking of hospitals
admission.59 Cases flagged on this indicator and compared to no risk adjustment. They
unflagged controls did not differ significantly on measured (1) the impact of adjustment on the
a composite of 17 generic process criteria. assessment of relative hospital performance, (2)
Similarly, cases flagged on this indicator and the relative importance of the adjustment, (3) the
unflagged controls did not differ significantly on impact on hospitals with the highest and lowest
a composite of four specific process criteria for rates, and (4) the impact throughout the
major surgical cases and two specific process distribution. The detected bias for Postoperative
criteria for medical cases in the earlier study of hemorrhage or hematoma is low, indicating that
elderly Medicare beneficiaries.60 the measures are likely not biased based on the
characteristics observed. (It is possible that
Empirical Analysis characteristics that are not observed using
administrative data may be related to the
The project team conducted extensive empirical patients risk of experiencing an adverse event.)
analyses on the PSIs. Postoperative
hemorrhage or hematoma generally performs Source
well on several different dimensions, including
reliability, bias, relatedness of indicators, and This indicator was originally proposed by Iezzoni
persistence over time. et al.61 as part of the Complications Screening
Program (CSP 24, post-procedural hemorrhage
Reliability. The signal ratiomeasured by the or hematoma), although their definition allowed
proportion of the total variation across hospitals either procedure or diagnosis codes. By
that is truly related to systematic differences contrast, the current definition requires a
(signal) in hospital performance rather than hemorrhage or hematoma diagnosis with an
random variation (noise)is lower than most associated procedure to either control the
indicators, at 8.6%, suggesting that observed hemorrhage or drain the hematoma. It was also
differences in risk-adjusted rates may not reflect included as one component of a broader
true differences across hospitals. indicator (adverse events and iatrogenic
complications) in AHRQs original HCUP
The signal standard deviation for this indicator is Quality Indicators.62
lower than most indicators, at 0.00039,
indicating that the systematic differences (signal)
Panelists expressed concern that acute renal Coding validity. No evidence on validity is
failure suffers from the problem of varied available from CSP studies. Geraci et al.70
definition: what one doctor may call acute renal
failure, another may not. To ensure that the only 70
renal failure cases that are picked up are those Geraci JM, Ashton CM, Kuykendall DH, Johnson
that are clinically severe, the panel suggested ML, Wu L. International Classification of Diseases, 9th
Revision, Clinical Modification codes in discharge
that acute renal failure be included only when it
abstracts are poor measures of complication
is paired with a procedure code for dialysis. occurrence in medical inpatients. Med Care
1997;35(6):589-602.
Version 2.1 41 Revision 1 (May 28, 2003)
confirmed only 5 of 15 episodes of acute renal indicators, at 20.9%, suggesting that observed
failure and 12 of 34 episodes of hypoglycemia differences in risk-adjusted rates may not reflect
reported on discharge abstracts of VA patients true differences across hospitals.
hospitalized for CHF, COPD, or diabetes.
Romano reported no false positives in episodes The signal standard deviation for this indicator is
of acute renal failure or hypoglycemia using lower than many indicators, at 0.00054,
discharge abstracts of diskectomy patients.71 indicating that the systematic differences (signal)
ICD-9-CM diagnoses (585 or 7885) had a among hospitals is low and less likely
sensitivity of 8% and a predictive value of 4% in associated with hospital characteristics. The
comparison with the VAs National Surgical signal share is lower than many indicators, at
Quality Improvement Program database, which 0.00033. The signal share is a measure of the
defines renal failure as requiring dialysis within share of total variation (hospital and patient)
30 days after surgery.72 accounted for by hospitals. The lower the share,
the less important the hospital in accounting for
Construct Validity. After adjusting for patient the rate and the more important other potential
demographic, geographic, and hospital factors (e.g., patient characteristics).
characteristics, Hannan et al. reported that
cases with a secondary diagnosis of fluid and Minimum bias. The project team assessed the
electrolyte disorders were no more likely to have effect of age, gender, DRG, and comorbidity risk
received care that departed from professionally adjustment on the relative ranking of hospitals
recognized standards than cases without that compared to no risk adjustment. They
code (2.2% versus 1.7%, OR=1.13).73 However, measured (1) the impact of adjustment on the
these ICD-9-CM codes were omitted from the assessment of relative hospital performance, (2)
accepted AHRQ PSIs. the relative importance of the adjustment, (3) the
impact on hospitals with the highest and lowest
Empirical Evidence rates, and (4) the impact throughout the
distribution. The detected bias for Postoperative
The project team conducted extensive empirical physiologic and metabolic derangements is
analyses on the PSIs. Postoperative physiologic moderate, indicating that the measures may or
and metabolic derangements generally performs may not be substantially biased based on the
well on several different dimensions, including characteristics observed. (It is possible that
reliability, bias, relatedness of indicators, and characteristics that are not observed using
persistence over time. administrative data may or may not be related to
the patients risk of experiencing an adverse
Reliability. The signal ratiomeasured by the event.)
proportion of the total variation across hospitals
that is truly related to systematic differences Source
(signal) in hospital performance rather than
random variation (noise)is lower than many This indicator was originally proposed by Iezzoni
et al.74 as part of the CSP (CSP 20,
postoperative physiologic and metabolic
71
Romano P. Can administrative data be used to
derangements). The University HealthSystem
ascertain clinically significant postoperative Consortium adopted the CSP indicator for major
complications. American Journal of Medical Quality surgery patients (#2945).
Press.
72
Best W, Khuri S, Phelan M, Hur K, Henderson W,
Demakis J, et al. Identifying patient preoperative risk
factors and postoperative adverse events in
administrative databases: Results from the
Department of Veterans Affairs National Surgical
Quality Improvement Program. J Am Coll Surg
2002;194(3):257-266.
73
Hannan EL, Bernard HR, ODonnell JF, Kilburn H,
74
Jr. A methodology for targeting hospital cases for Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher
quality of care record reviews. Am J Public Health ES, Duncan C, et al. Identifying complications of care
1989;79(4):430-6. using administrative data. Med Care 1994;32(7):700-
15.
Version 2.1 42 Revision 1 (May 28, 2003)
Postoperative Pulmonary Embolism or Deep Vein Thrombosis
Definition Cases of deep vein thrombosis (DVT) or pulmonary embolism (PE) per
1,000 surgical discharges.
Numerator Discharges with ICD-9-CM codes for deep vein thrombosis or pulmonary
embolism in any secondary diagnosis field per 1,000 surgical discharges.
Denominator All surgical discharges defined by specific DRGs.
Exclude patients with a principal diagnosis of deep vein thrombosis.
Exclude obstetrical patients in MDC 14.
Exclude patients with secondary procedure code 38.7 when this procedure
occurs on the day of or previous to the day of the principal procedure.
Note: If day of procedure is not available in the input data file, the rate may
be slightly lower than if the information was available.
Type of Indicator Hospital level
Empirical Performance Rate: 9.59 per 1,000 population at risk
Bias: Substantial bias; should be risk-adjusted
Risk Adjustment Age, sex, DRG, comorbidity categories
Summary
Literature Review
This indicator is intended to capture cases of
postoperative venous thromboses and Coding validity. Geraci et al. confirmed only 1 of
embolismspecifically, pulmonary embolism 6 episodes of DVT or PE reported on discharge
and deep venous thrombosis. This indicator abstracts of VA patients for CHF, COPD, or
limits vascular complications codes to secondary diabetes; the sensitivity was 100%.75 Among
diagnosis codes to eliminate complications that Medicare hip fracture patients, by contrast,
were present on admission. It further excludes Keeler et al. confirmed 88% of reported PE
patients who have principal diagnosis of DVT, as cases, and failed to ascertain just 6 cases (65%
these patients are likely to have had PE/DVT sensitivity) using ICD-9-CM codes.76 For DVT,
present on admission. they found just 1 of 6 cases using ICD-9-CM
codes (but no false positive codes). Other
Panel Review studies have demonstrated that ICD-9-CM
codes for DVT and PE have high predictive
Panelists rated the overall usefulness of this value when listed as the principal diagnosis for
indicator relatively highly as compared to other readmissions after major orthopedic surgery
indicators. They noted that preventative (100%) or after inferior vena cava filter
techniques should decrease the rate of this
indicator. This indicator includes pediatric
patients. In the absence of specific thrombophilic 75
Geraci JM, Ashton CM, Kuykendall DH, Johnson
disorders, postoperative thromboembolic ML, Wu L. In-hospital complications among survivors
complications in children are most likely to be of admission for congestive heart failure, chronic
secondary to venous catheters rather than obstructive pulmonary disease, or diabetes mellitus. J
venous stasis in the lower extremities. Gen Intern Med 1995;10(6):307-14.
76
Because the risk for DVT/PE varies greatly Keeler E, Kahn K, Bentow S. Assessing quality of
according to the type of procedure performed, care for hospitalized Medicare patients with hip
fracture using coded diagnoses from the Medicare
panelists suggested that this indicator be
Provider Analysis and Review File. Springfield, VA:
adjusted or stratified according to surgical NTIS;1991.
procedure types.
Definition Cases of acute respiratory failure per 1,000 elective surgical discharges.
Numerator Discharges with ICD-9-CM codes for acute respiratory failure (518.81) in
any secondary diagnosis field per 1,000 discharges (After 1999, include
51884).
Denominator All elective surgical discharges defined by admit type.
Exclude patients with respiratory or circulatory diseases (MDC 4 and MDC
5).
Exclude obstetrical patients in MDC 14.
Type of Indicator Hospital level
Empirical Performance Rate: 3.64 per 1,000 population at risk
Bias: Substantial bias; should be risk-adjusted
Risk Adjustment Age, sex, DRG, comorbidity categories
Literature Review
data to find substandard care: Validation of the
Complications Screening Program. Med Care
Coding Validity. CSP 3 had a relatively high
2000;38(8):796-806.
confirmation rate among major surgical cases in
the FY1994 Medicare inpatient claims files from 86
Geraci JM, Ashton CM, Kuykendall DH, Johnson
California and Connecticut (72% by coders ML, Wu L. In-hospital complications among survivors
review, 75% by physicians review).84 85 Nurse of admission for congestive heart failure, chronic
obstructive pulmonary disease, or diabetes mellitus. J
Gen Intern Med 1995;10(6):307-14.
84
Lawthers a, McCarthy E, Davis R, Peterson L,
87
Palmer R, Iezzoni L. Identification of in-hospital Iezzoni LI, Davis RB, Palmer RH, Cahalane M,
complications from claims data: is it valid? Med Care Hamel MB, Mukamal K, et al. Does the Complications
2000;38(8):785-795. Screening Program flag cases with process of care
problems? Using explicit criteria to judge processes.
85
Weingart SN, Iezzoni LI, Davis RB, Palmer RH, Int J Qual Health Care 1999;11(2):107-18.
Cahalane M, Hamel MB, et al. Use of administrative
Version 2.1 45 Revision 1 (May 28, 2003)
elderly Medicare beneficiaries.88 effect of age, gender, DRG, and comorbidity risk
adjustment on the relative ranking of hospitals
Needleman and Buerhaus found that nurse compared to no risk adjustment. They
staffing was independent of the occurrence of measured (1) the impact of adjustment on the
pulmonary failure among major surgery assessment of relative hospital performance, (2)
patients.89 However, Kovner and Gergen the relative importance of the adjustment, (3) the
reported that having more registered nurse impact on hospitals with the highest and lowest
hours per adjusted patient day was associated rates, and (4) the impact throughout the
with a lower rate of pulmonary compromise distribution. The detected bias for Postoperative
after major surgery.90 respiratory failure is high, indicating that the
measures likely are biased based on the
Empirical Analysis characteristics observed. (It is possible that
characteristics that are not observed using
The project team conducted extensive empirical administrative data may be related to the
analyses on the PSIs. Postoperative respiratory patients risk of experiencing an adverse event.)
failure generally performs well on several Risk adjustment is important for this indicator.
different dimensions, including reliability, bias,
relatedness of indicators, and persistence over Source
time.
This indicator was originally proposed by Iezzoni
Reliability. The signal ratiomeasured by the et al. as part of the CSP (CSP 3, postoperative
proportion of the total variation across hospitals pulmonary compromise).91 Their definition also
that is truly related to systematic differences includes pulmonary congestion, other (or
(signal) in hospital performance rather than postoperative) pulmonary insufficiency, and
random variation (noise)is lower than many acute pulmonary edema, which were omitted
indicators, at 46.6%, suggesting that observed from this PSI. The University HealthSystem
differences in risk-adjusted rates may not reflect Consortium (#2927) and AHRQs original HCUP
true differences across hospitals. Quality Indicators adopted the CSP indicator for
major surgery patients.92 Needleman and
The signal standard deviation for this indicator is Buerhaus identified postoperative pulmonary
lower than many indicators, at 0.00230, failure as an Outcome Potentially Sensitive to
indicating that the systematic differences (signal) Nursing, using the original CSP definition.93
among hospitals is low and less likely
associated with hospital characteristics. The
signal share is lower than many indicators, at
0.00187. The signal share is a measure of the
share of total variation (hospital and patient)
accounted for by hospitals. The lower the share,
the less important the hospital in accounting for
the rate and the more important other potential
factors (e.g., patient characteristics).
88
Hawker GA, Coyte PC, Wright JG, Paul JE,
Bombardier C. Accuracy of administrative data for
91
assessing outcomes after knee replacement surgery. Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher
J. Clin Epidimiol 1997;50(3):265-73. ES, Duncan C, et al. Identifying complications of care
using administrative data. Med Care 1994;32(7):700-
89 15.
Needleman J, Buerhaus PI, Mattke S, Stewart M,
Zelevinsky K. Nurse Staffing and Patient Outcomes in
92
Hospitals. Boston, MA: Health Resources Services Johantgen M, Elixhauser A, Bali JK, Goldfarb M,
Administration; 2001 February 28. Report No.:230-99- Harris DR. Quality indicators using hospital discharge
0021. data: State and national applications. Jt Comm J Qual
Improv 1998;24(2):88-195. Published erratum
90 appears in Jt Comm J Qual Improv 1998;24(6):341.
Kovner C, Gergen PJ. Nurse staffing levels and
adverse events following surgery in U.S. hospitals.
93
Image J Nurs Sch 1998;30(4):315-21. Needleman et al. 2001.
Definition Cases of sepsis per 1,000 elective surgery patients, with length of stay
more than 3 days.
Numerator Discharges with ICD-9-CM code for sepsis in any secondary diagnosis field
per 1,000 elective surgical discharge.
Denominator All elective surgical discharges defined by admit type.
Exclude patients with a principal diagnosis of infection, any code for
immunocompromised state, or cancer.
Include only patients with a length of stay of 4 days or more.
Exclude obstetrical patients in MDC 14.
Type of Indicator Hospital level
Empirical Performance Rate: 10.1 per 1,000 population at risk
Bias: Substantial bias; should be risk-adjusted
Risk Adjustment Age, sex, DRG, comorbidity categories
97 98
Needleman J, Buerhaus PI, Mattke S, Stewart M, Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher
Zelevinsky K. Nurse Staffing and Patient Outcomes in ES, Duncan C, et al. Identifying complications of care
Hospitals. Boston, MA: Health Resources Services using administrative data. Med Care 1994;32(7):700-
Administration; 2001 February 28. Report No.:230-99- 15.
0021.
99
Needleman et al., 2001.
Summary comorbidities.
This indicator is intended to flag cases of wound
dehiscence in patients who have undergone Literature Review
abdominal and pelvic surgery. This indicator is Coding validity. No evidence on validity is
defined both on a hospital level (by including available from CSP studies. Hawker et al. found
cases based on secondary diagnosis associated that the sensitivity and predictive value of wound
with the same hospitalization) and on an area dehiscence were both 100%.100 Faciszewski et
level (by including all cases of wound al. aggregated wound dehiscence with
dehiscence). postoperative hemorrhage or hematoma and
reported a pooled confirmation rate of 17% with
Panel Review 3% sensitivity of coding among patients who
Panelists suggested that postoperative wound underwent spinal fusion.101 In comparison with
disruption be excluded from the indicator and
100
that trauma, cancer, and immunocompromised Hawker BA, Coyte PC, Wright JG, Paul JE,
patients be included. They also reported that Bombardier C. Accuracy of administrative data for
the risk of developing wound dehiscence varies assessing outcomes after knee replacement surgery.
with patient factors such as age and J Clin Epidemiol 1997;50(3):265-73.
101
Faciszewski T, Johnson L, Noren C, Smith MD.
Reliability. The signal ratiomeasured by the This indicator was originally proposed by Iezzoni
proportion of the total variation across hospitals et al. as part of the Complications Screening
that is truly related to systematic differences Program, although unlike the final PSI, its codes
(signal) in hospital performance rather than were split between two CSP indicators (CSP 27,
random variation (noise)is moderately high, technical difficulty with medical care, and
relative to other indicators, at 82.9%, suggesting sentinel events).112 It was also included as one
that observed differences in risk-adjusted rates component of a broader indicator (adverse
most likely reflect true differences across events and iatrogenic complications) in AHRQs
hospitals. original HCUP Quality Indicators.113 The
University HealthSystem Consortium adopted
The signal standard deviation for this indicator is CSP 27 as an indicator for medical (#2806) and
lower than many indicators, at 0.00279, major surgery (#2956) patients. Miller et al. also
indicating that the systematic differences (signal) split this set of ICD-9-CM codes into two broader
among hospitals is low and less likely indicators (miscellaneous misadventures and
associated with hospital characteristics. The E codes) in the original AHRQ PSI Algorithms
signal share is lower than many indicators, at and Groupings.114 Based on expert consensus
0.00241. The signal share is a measure of the panels, McKesson Health Solutions included
share of total variation (hospital and patient) one component of this PSI (Accidental Puncture
accounted for by hospitals. The lower the share, or Laceration) in its CareEnhance Resource
the less important the hospital in accounting for Management Systems, Quality Profiler
the rate and the more important other potential Complications Measures Module.
factors (e.g., patient characteristics).
109 112
Valinsky LJ, Hockey RI, Hobbs MS, Fletcher DR, Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher
Pikora TJ, Parsons RW, et al. Finding bile duct ES, Duncan C, et al. Identifying complications of care
injuries using record linkage: A validated study of using administrative data. Med Care 1994;32(7):700-
complications following cholecystectomy. J Clin 15.
Epidemiol 1999;52(9):893-901.
113
Johantgen M, Elixhauser A, Bali JK, Goldfarb M,
110
Hawker GA, Coyte PC, Wright JG, Paul JE, Harris DR. Quality indicators using hospital discharge
Bombardier C. Accuracy of administrative data for data: State and national applications. Jt Comm J Qual
assessing outcomes after knee replacement surgery. Improv 1998;24(2):88-195. Published erratum
J Clin Epidemiol 1997;50(3):265-73. appears in Jt Comm J Qual Improv 1998;24(6):341.
111 114
Romano P. Can administrative data be used to Miller M, Elixhauser A, Zhan C, Meyer G, Patient
ascertain clinically significant postoperative Safety Indicators: Using administrative data to identify
complications. American Journal of Medical Quality potential patient safety concerns. Health Services
Press. Research 2001;36(6 Part II):110-132.
Version 2.1 52 Revision 1 (May 28, 2003)
Transfusion Reaction
Hospital Level Definition
Definition Cases of transfusion reaction per 1,000 discharges.
Numerator Discharges with ICD-9-CM code for transfusion reaction in any secondary
diagnosis field per 1,000 discharges.
Denominator All medical and surgical discharges defined by specific DRGs.
Type of Indicator Hospital level
Area level
Empirical Performance Rate: 0.01 per 1,000 population at risk
Bias: Did not undergo empirical testing of bias
Risk Adjustment No risk adjustment
Transfusion Reaction
Area Level Definition
Definition Cases of transfusion reaction per 100,000 population.
Numerator Discharges with ICD-9-CM code for transfusion reaction in any diagnosis
field (principal or secondary ) of all medical and surgical discharges defined
by specific DRGs.
Denominator Population of county or MSA associated with FIPS code of patients
residence or hospital location.
Type of Indicator Area level
Empirical Performance Rate: 0.05 per 100,000 population
Risk Adjustment No risk adjustment
This indicator is intended to flag cases of major The project team was unable to find evidence on
reactions due to transfusions (ABO and Rh). validity from prior studies, most likely because
This indicator is defined both on a hospital level this complication is quite rare.
(by including cases based on secondary
diagnosis associated with the same Empirical Analysis
hospitalization) and on an area level (by
including all cases of transfusion reactions). The project team conducted extensive empirical
analyses on the PSIs. Given the low rates or
Panel Review occurrences for Transfusion reaction, the project
team did not measure reliability or minimum
The overall usefulness of this indicator was bias. The indicator could not be risk-adjusted
rated as very favorable by panelists. This due to the small number of numerator cases.
indicator includes only those events that result in Users of the PSI software should note the output
additional medical care. Some minor reactions will only contain observed rates for Transfusion
may be missed, although the panel suggested reaction.
that these minor reactions are less clearly due to
medical error than the Rh or ABO reactions Source
included in the indicator.
This indicator was originally proposed by Iezzoni
115
Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher
ES, Duncan C, et al. Identifying complications of care
using administrative data. Med Care 1994;32(7):700-
15.
116
Johantgen M, Elixhauser A, Bali JK, Goldfarb M,
Harris DR. Quality indicators using hospital discharge
data: State and national applications. Jt Comm J Qual
Improv 1998;24(2):88-195. Published erratum
appears in Jt Comm J Qual Improv 1998;24(6):341.
117
Miller M, Elixhauser A, Zhan C, Meyer G, Patient
safety indicators: Using administrative data to identify
potential patient safety concerns. Health Services
Research 2001;36(6 Part II):110-132.
Version 2.1 54 Revision 1 (May 28, 2003)
Birth TraumaInjury to Neonate
The overall usefulness of this indicator was Reliability. The signal ratiomeasured by the
rated as favorable by panelists proportion of the total variation across hospitals
that is truly related to systematic differences
Literature Review (signal) in hospital performance rather than
random variation (noise)is high, relative to
Coding validity. A study of newborns who had a other indicators, at 97.0%, suggesting that
discharge diagnosis of birth trauma found that observed differences in risk-adjusted rates
only 25% had sustained a significant injury to reflect true differences across hospitals.
the head, neck, or shoulder.118 The remaining
patients either had superficial injuries or injuries The signal standard deviation for this indicator is
inferior to the neck. The project team was also high, relative to other indicators, at 0.04128,
unable to find other evidence on the validity of indicating that the systematic differences (signal)
this indicator. Towner et al. linked California among hospitals is high and more likely
maternal and infant discharge abstracts from associated with hospital characteristics. The
1992 through 1994, but they used only infant signal share is also high, relative to other
discharge abstracts to describe the incidence of indicators, at 0.13603. The signal share is a
neonatal intracranial injury, and they did not measure of the share of total variation (hospital
report the extent of agreement between the two and patient) accounted for by hospitals. The
118 119
Hughes C, Harley E, Milmoe G, Bala R, Martorella Towner D, Castro MA, Eby-Wilkens E, Gilbert
A. Birth trauma in the head and neck. Arch WM. Effect of mode of delivery in nulliparous women
Otolaryngol Head Neck Surg 1999;125:193-199. on neonatal intracranial injury. N Engl J Med
1999;341(23):1709-14.
Version 2.1 55 Revision 1 (May 28, 2003)
lower the share, the less important the hospital
in accounting for the rate and the more
important other potential factors (e.g., patient
characteristics).
Source
120
Miller M, Elixhauser A, Zhan C, Meyer G, Patient
Safety Indicators: Using administrative data to identify
potential patient safety concerns. Health Services
Research 2001;36(6 Part II):110-132.
121
Johantgen M, Elixhauser A, Bali JK, Goldfarb M,
Harris DR. Quality indicators using hospital discharge
data: State and national applications. Jt Comm J Qual
Improv 1998;24(2):88-195. Published erratum
appears in Jt Comm J Qual Improv 1998;24(6):341.
Version 2.1 58 Revision 1 (May 28, 2003)
Obstetric TraumaVaginal Delivery with Instrument
Summary
Reliability. The signal ratiomeasured by the
This indicator is intended to flag cases of proportion of the total variation across hospitals
potentially preventable trauma during vaginal that is truly related to systematic differences
delivery with instrument. (signal) in hospital performance rather than
random variation (noise)is moderately high,
Panel Review relative to other indicators, at 69.9%, suggesting
that observed differences in risk-adjusted rates
The overall usefulness of an Obstetric trauma likely reflect true differences across hospitals.
indicator was rated as favorable by panelists.
After initial review, the indicator was eventually The signal standard deviation for this indicator is
split into three separate Obstetric trauma also high, relative to other indicators, at 0.09794,
indicators: Vaginal delivery with instrument, indicating that the systematic differences (signal)
Vaginal delivery without instrument, and among hospitals is high and more likely
Cesarean delivery. associated with hospital characteristics. The
signal share is high, relative to other indicators,
Literature Review at 0.05539. The signal share is a measure of the
share of total variation (hospital and patient)
Coding validity. In a stratified probability sample accounted for by hospitals. The lower the share,
of vaginal and Cesarean deliveries, the weighted the less important the hospital in accounting for
sensitivity and predictive value of coding for the rate and the more important other potential
third- and fourth-degree lacerations and factors (e.g., patient characteristics).
vulvar/perineal hematomas (based on either
diagnosis or procedure codes) were 89% and Minimum bias. The bias for Obstetric
90%, respectively.158 The authors did not report traumavaginal delivery with instrument was
coding validity for third- and fourth-degree not measured, since adequate risk adjustment
lacerations separately. The project team was was not available..
unable to find other evidence on validity from
prior studies. Source
122
Johantgen M, Elixhauser A, Bali JK, Goldfarb M,
Harris DR. Quality indicators using hospital discharge
data: State and national applications. Jt Comm J Qual
Improv 1998;24(2):88-195. Published erratum
appears in Jt Comm J Qual Improv 1998;24(6):341.
Version 2.1 60 Revision 1 (May 28, 2003)
Obstetric TraumaVaginal Delivery without Instrument
This indicator is intended to flag cases of Reliability. The signal ratiomeasured by the
potentially preventable trauma during a vaginal proportion of the total variation across hospitals
delivery without instrument. that is truly related to systematic differences
(signal) in hospital performance rather than
Panel Review random variation (noise)is high, relative to
other indicators, at 86.4%, suggesting that
The overall usefulness of an Obstetric trauma observed differences in risk-adjusted rates
indicator was rated as favorable by panelists. reflect true differences across hospitals.
After initial review, the indicator was split into
three separate Obstetric trauma indicators: The signal standard deviation for this indicator is
Vaginal delivery with instrument, Vaginal also high, relative to other indicators, at 0.04314,
delivery without instrument, and Cesarean indicating that the systematic differences (signal)
delivery. among hospitals is high and more likely
associated with hospital characteristics. The
Literature Review signal share is lower than many other indicators,
at 0.02470. The signal share is a measure of the
Coding validity. In a stratified probability sample share of total variation (hospital and patient)
of vaginal and Cesarean deliveries, the weighted accounted for by hospitals. The lower the share,
sensitivity and predictive value of coding for the less important the hospital in accounting for
third- and fourth-degree lacerations and the rate and the more important other potential
vulvar/perineal hematomas (based on either factors (e.g., patient characteristics).
diagnosis or procedure codes) were 89% and
90%, respectively.158 The authors did not report Minimum bias. The bias for Obstetric
coding validity for third- and fourth-degree traumavaginal delivery without instrument was
lacerations separately. The project team was not measured, since adequate risk adjustment
unable to find other evidence on validity from was not available.
prior studies.
Source
Empirical Analysis
An overlapping subset of this indicator (third- or
The project team conducted extensive empirical fourth-degree perineal laceration) has been
analyses on the PSIs. Obstetric adopted by the Joint Commission for the
traumavaginal delivery without instrument Accreditation of Healthcare Organizations
generally performs well on several different (JCAHO) as a core performance measure for
dimensions, including reliability, relatedness of pregnancy and related conditions (PR-25).
123
Johantgen M, Elixhauser A, Bali JK, Goldfarb M,
Harris DR. Quality indicators using hospital discharge
data: State and national applications. Jt Comm J Qual
Improv 1998;24(2):88-195. Published erratum
appears in Jt Comm J Qual Improv 1998;24(6):341.
Version 2.1 62 Revision 1 (May 28, 2003)
References
Ball JK, Elixhauser A, Johantgen M, et al. HCUP Quality Indicators, Methods, Version 1.1: Outcome,
Utilization, and Access Measures for Quality Improvement. (AHCPR Publication No. 98-0035).
Healthcare Cost and Utilization project (HCUP-3) Research notes: Rockville, MD: Agency for Health
Care Policy and Research, 1998.
Barbour GL. Usefulness of a discharge diagnosis of sepsis in detecting iatrogenic infection and quality of
care problems. Am J Med Qual 1993;8(1):2-5.
Berlowitz D, Brand H, Perkins C. Geriatric syndromes as outcome measures of hospital care: Can
administrative data be used? JAGS 1999;47:692-696.
Best W, Khuri S, Phelan M, Hur K, Henderson W, Demakis J, et al. Identifying patient preoperative risk
factors and postoperative adverse events in administrative databases: Results from the Department
of Veterans Affairs National Surgical Quality Improvement Program. J Am Coll Surg 2002;194(3):257-
266.
Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events
and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J
Med 1991;324(6):370-6.
Christiansen CL, Morris CN. Improving the statistical approach to health care provider profiling. Ann Intern
Med 1997;127(8 Pt 2):764-8).
Davies S, Geppert J, McClellan M, McDonald KM, Romano PS, Shojania KG. Refinement of the HCUP
Quality Indicators. Technical Review Number 4. Rockville, MD: (Prepared by UCSF-Stanford
Evidence-based Practice Center under Contract No. 290-97-0013) Agency for Healthcare Research
and Quality; 2001. Report No.: 01-0035.
Envisioning the National Health Care Quality Report. Washington, DC: Institute of Medicine; 2001.
Faciszewski T, Johnson L, Noren C, Smith MD. Administrative databases complication coding in anterior
spinal fusion procedures. What does it mean? Spine 1995;20(16):1783-8.
Fitch K, Bernstein J, Aguilar MD, Burnand B, LaCalle JR, Lazaro P, et al. the RAND/UCLA
Appropriateness Method Users Manual: RAND; 2001.
Geraci JM, Ashton CM, Kuykendall DH, Johnson ML, Wu L. In-hospital complications among survivors of
admission for congestive heart failure, chronic obstructive pulmonary disease, or diabetes mellitus. J
Gen Intern Med 1995;10(6):307-14.
Geraci JM, Ashton CM, Kuykendall DH, Johnson ML, Wu L. International Classification of Diseases, 9th
Revision, Clinical Modification codes in discharge abstracts are poor measures of complication
occurrence in medical inpatients. Med Care 1997;35(6):589-602.
Green L, Lewis F. measurement and Evaluation in Health Education and Health Promotion. Mountain
View, CA: Mayfield Publishing Company; 1998.
Hannan EL, Bernard HR, ODonnell JF, Kilburn H, Jr. A methodology for targeting hospital cases for
quality of care record reviews. Am J Public Health 1989;79(4):430-6.
Hartz AJ, Kuhn EM. Comparing hospitals that perform coronary artery bypass surgery: The effect of
outcome measures and data sources. Am J Public Health 1994;84(10):1609-14.
Hofer TP, Hayward RA, Greenfield S, Wagner EH, Kaplan SH, Manning WG. The unreliability of
individual physician report cards for assessing the costs and quality of care of a chronic disease
JAMA 1999;281(22):2098-105.
Hughes C, Harley E, Milmoe G, Bala R, Martorella A. Birth trauma in the head and neck. Arch Otolaryngol
Head Neck Surg 1999;125:193-199.
Iezzoni L, Lawthers A, Petersen L, McCarthy E, Palmer R, Cahalane M, et al. Project to validate the
Complications Screening Program: Health Care Financing Administration; 1998 March 31. Report No:
HCFA Contract 500-94-0055.
Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher ES, Duncan C, et al. Identifying complications of care
using administrative data. Med Care 1994;32(7):700-15.
Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel MB, Mukamal K, et al. Does the Complications
Screening Program flag cases with process of care problems? Using explicit criteria to judge
processes. Int J Qual Health Care 1999;11(2):107-18.
Iezzoni LI, Foley SM, Heeren T, Daley J, Duncan CC, Fisher ES, et al. A method for screening the quality
of hospital care using administrative data: preliminary validation results. QRB Qual Rev Bull
1992;18(11):361-71.
Impact: Case Studies Notebook Documented Impact and Use of AHRQ's Research. Compiled by
Division of Public Affairs, Office of Health Care Information, Agency for Healthcare Research and
Quality.
Institute of Medicine. To Err is Human: Building a Safer Health System. Kohn LT, Corrigan JM, Donaldson
MS (eds.) Washington DC: National Academy Press, 2000.
Johantgen M, Elixhauser A, Bali JK, Goldfarb M, Harris DR. Quality indicators using hospital discharge
data: state and national applications. Jt Comm J Qual Improv 1998;24(2):88-105.
Johantgen M, Elixhauser A, Bali JK, Goldfarb M, Harris DR. Quality indicators using hospital discharge
data: state and national applications. Jt Comm J Qual Improv 1998;24(2):88-195. Published erratum
appears in Jt Comm J Qual Improv 1998;24(6):341.
Keeler E, Kahn K, Bentow S. Assessing quality of care for hospitalized Medicare patients with hip fracture
using coded diagnoses from the Medicare Provider Analysis and Review file. Springfield, VA: NTIS;
1991.
Kovner C, Gergen PH. Nurse staffing levels and adverse events following surgery in U.S. hospitals.
Image J Nurs Sch 1998;30(4):315-21.
Lichtig LK, Knauf RA, Hilholland DK. Some impacts of nursing on acute care hospital outcomes. J Nurs
Adm 1999;29(2):25-33.
McCarthy EP, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamael MB, et al. Does clinical evidence
support ICD-9-CM diagnosis coding of complications? Med Care 2000;38(8);868-876.
Measuring the Quality of Health Care: A statement of the National Roundtable on Healthcare Quality
Miller M, Elixhauser A, Zhan C, Meyer G, Patient Safety Indicators: Using administrative data to identify
potential patient safety concerns. Health Services Research 2001;36(6 Part II):110-132.
Needleman J, Buerhaus PI, Mattke S, Stewart M, Zelevinsky K. Nurse Staffing and Patient Outcomes in
Hospitals. Boston, MA: Health Resources Services Administration; 2001 February 28. Report No.:
230-88-0021.
Nursing-Sensitive Quality Indicators for Acute Care Settings and ANAs Safety & Quality Initiative. In:
American Nurses Association; 1999.
Romano P. Can administrative data be used to ascertain clinically significant postoperative complications.
American Journal of Medical Quality Press.
Shojania KG, Duncan BW, McDonald KM, Wachter RM. Making health care safer: A critical analysis of
patient safety practices. Evidence Report/Technology Assessment No. 43 (Prepared by the University
of California at San Francisco-Stanford Evidence-based Practice Center under Contract No. 290-97-
0013). Rockville, MD: Agency for Healthcare Research and Quality; 2001. Report No.: AHRQ
Publication No. 01-E058.
Silber J, Rosenbaum P, Ross R. Comparing the contributions of groups of predictors: Which outcomes
vary with hospital rather than patient characteristics? J Am Stat Assoc 1995;90:7-18.
Silber JH, Rosenbaum PR, Williams SV, Ross RN, Schwartz JS. The relationship between choice of
outcome measure and hospital rank in general surgical procedures: Implications for quality
assessment. Int J Qual Health Care 1997;9(3):193-200.
Silber JH, Williams SV, Krakauer H, Schwartz JS. Hospital and patient characteristics associated with
death after surgery. A study of adverse occurrence and failure to rescue. Med Care 1992;30(7):615-
29.
Taylor B. Common bile duct injury during laparoscopic cholecystectomy in Ontario: Does ICD-9 coding
indicate true incidence? CMAJ 1998;158(4):481-5.
Towner D, Castro MA, Eby-Wilkens E, Gilbert WM. Effect of mode of delivery in nulliparous women on
neonatal intracranial injury. N Engl J Med 1999;341(23):1709-14.
Valinsky LJ, Hockey RI, Hobbs MS, Fletcher DR, Pikora TJ, Parsons RW, et al. Finding bile duct injuries
using record linkage: A validated study of complications following cholecystectomy. J Clin Epidemiol
1999;52(9):893-901.
Weingart SN, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel MB, et al. Use of administrative data
to find substandard care: Validation of the Complications Screening Program. Med Care
2000;38(8):796-806.
White RH, Romano P, Zhou H, Rodrigo J, Barger W. Incidence and time course of thromboembolic
outcomes following total hip or knee arthroplasty. Arch Intern Med 1998;158(14):1525-31.
Complications of Anesthesia
Numerator:
Discharges with ICD-9-CM diagnosis codes for anesthesia complications in any secondary
diagnosis field per 1,000 discharges.
Anesthesia Complications
ICD-9-CM diagnosis codes:
Adverse effects in therapeutic use, other central nervous system depressants and anesthetics:
E9381 Halothane
E9382 Other gaseous anesthetics
E9383 Intravenous anesthetics
E9384 Other and unspecified general anesthetics
E9385 Surface and infiltration anesthetics
E9386 Peripheral nerve and plexus blocking anesthetics
E9387 Spinal anesthetics
E9389 Other and unspecified local anesthetics
Denominator:
All surgical discharges defined by specific DRGs.
Surgical Discharges
DRGs:
Exclude:
Patients with codes for poisoning due to anesthetics (E8551, 9681-4, 9687) and any diagnosis
code for active drug dependence, active nondependent abuse of drugs, or self-inflicted injury.
Self-Inflicted Injury
ICD-9-CM diagnosis codes:
E956 Suicide and self inflicted injury by cutting and piercing instrument
Exclude:
Patients with any code for trauma, immunocompromised state, or cancer.
Trauma
ICD-9-CM diagnosis codes (includes 4th and 5th digits):
DRGs:
Immunocompromised States
ICD-9-CM diagnosis codes:
Cancer
ICD-9-CM diagnosis codes (includes 4th and 5th digits):
DRGs:
Decubitus Ulcer
Numerator:
Discharges with ICD-9-CM code of 7070 in any secondary diagnosis field per 1,000 discharges.
Denominator:
All medical and surgical discharges defined by specific DRGs (see denominator for
Complications of Anesthesia for surgical discharges).
Medical Discharges
DRGs:
Failure to Rescue
Numerator:
All discharges with a disposition of deceased per 1,000 population at risk.
Denominator:
Discharges with potential complications of care listed in failure to rescue (FTR) definition (e.g.,
pneumonia, DVT/PE, sepsis, acute renal failure, shock/cardiac arrest, or GI hemorrhage/acute
ulcer). Exclusion criteria specific to each diagnosis.
Exclude principal diagnosis of acute renal failure, abortion-related renal failure, acute myocardial
infarction, cardiac arrest, cardiac arrhythmia, hemorrhage, GI hemorrhage, shock, or trauma.
Cardiac Arrhythmia
ICD-9-CM diagnosis codes (when principal diagnosis):
Cardiac Arrest
ICD-9-CM diagnosis code (when principal diagnosis):
Hemorrhage:
ICD-9-CM diagnosis codes (when principal diagnosis):
Shock
ICD-9-CM diagnosis codes (when principal diagnosis):
FTRDVT/PE
Include ICD-9-CM diagnosis codes:
Exclude principal diagnosis of pulmonary embolism or deep vein thrombosis, abortion related
and postpartum obstetric pulmonary embolism.
FTRPneumonia
Include ICD-9-CM diagnosis codes:
Exclude principal diagnosis code for pneumonia or 997.3, any diagnosis code for viral pneumonia,
MDC 4, and any diagnosis of immunocompromised state.
Viral pneumonia
ICD-9-CM diagnosis codes (includes 4th and 5th digits) (when principal diagnosis):
Immunocompromised States
ICD-9-CM diagnosis codes (when principal diagnosis):
FTRSepsis
Include ICD-9-CM diagnosis codes:
Immunocompromised States
ICD-9-CM diagnosis codes (when principal diagnosis):
124
Note: The length of stay exclusion criteria has been corrected in this version of the PSI Guide. The
first version noted length of stay of 4 or more days which was incorrect.
Version 2.1 93 Revision 1 (May 28, 2003)
Failure to Rescue
3352 Bilateral lung transplantation
336 Combined heart-lung transplantation
375 Heart transplantation
410 Operations on bone marow and spleen
4100 Bone marrow transplant, NOS
4101 Autologous bone marrow transplant without purging
4102 Allogenic bone marrow transplant with purging
4103 Allogenic bone marrow transplant without purging
4104 Autologous hematopoietic stem cell transplant without purging
4105 Allogeneic hematopoietic stem cell transplant without purging
4106 Cord blood stem cell transplant
4107 Autologous hematopoietic stem cell transplant with purging
4108 Allogeneic hematopoietic stem cell transplant with purging
4109 Autologous bone marrow transplant with purging
5051 Auxiliary liver transplant
5059 Liver transplant, NEC
5280 Pancreatic transplant, NOS
5281 Reimplantation of pancreatic tissue
5282 Homotranplant of pancreas
5283 Heterotransplant of pancreas
5285 Allotransplantation of cells of islets of Langerhans
5286 Transplantation of cells of islets of Langerhans, NOS
5569 Other kidney transplantation
Infection
ICD-9-CM diagnosis codes (when principal diagnosis):
DRGs:
Exclude MDC 4 and 5, principal diagnosis of shock or cardiac arrest, abortion-related shock,
hemorrhage, trauma, GI hemorrhage.
Abortion-related Shock
ICD-9-CM diagnosis codes (when principal diagnosis):
Gastric ulcer:
53130 Acute without mention of hemorrhage or perforation without mention of obstruction
53131 Acute without mention of hemorrhage or perforation with obstruction
53190 Unspecified as acute or chronic, without mention of hemorrhage or perforation without mention of
obstruction
53191 Unspecified as acute or chronic, without mention of hemorrhage or perforation with obstruction
Duodenal ulcer:
53230 Acute without mention of hemorrhage or perforation without mention of obstruction
53231 Acute without mention of hemorrhage or perforation with obstruction
53290 Unspecified as acute or chronic, without mention of hemorrhage or perforation without mention of
obstruction
53291 Unspecified as acute or chronic, without mention of hemorrhage or perforation with obstruction
Peptic ulcer:
53330 Site unspecified acute without mention of hemorrhage and perforation without mention of obstruction
53331 Site unspecified acute without mention of hemorrhage and perforation with obstruction
53390 Site unspecified as acute or chronic, without mention of hemorrhage or perforation without mention of
obstruction
53391 Unspecified as acute or chronic, without mention of hemorrhage or perforation with obstruction
Gastrojejunal ulcer:
53430 Acute without mention of hemorrhage or perforation without mention of obstruction
53431 Acute without mention of hemorrhage or perforation with obstruction
53190 Unspecified as acute or chronic, without mention of hemorrhage or perforation without mention of
obstruction
53491 Unspecified as acute or chronic, without mention of hemorrhage or perforation with obstruction
5307 Gastroesophageal laceration-hemorrhage syndrome
53082 Esophageal hemorrhage
Gastric ulcer:
53100 Acute with hemorrhage without mention of obstruction
53101 Acute with hemorrhage with obstruction
53110 Acute with perforation without mention of obstruction
53111 Acute with perforation with obstruction
53120 Acute with hemorrhage and perforation without mention of obstruction
53121 Acute with hemorrhage and perforation with obstruction
53130 Acute without mention of hemorrhage or perforation without mention of obstruction
Duodenal ulcer:
Peptic ulcer:
53300 Site unspecified acute with hemorrhage without mention of obstruction
53301 Site unspecified acute with hemorrhage with obstruction
53310 Site unspecified acute with perforation without mention of obstruction
53311 Site unspecified acute with perforation with obstruction
53320 Site unspecified acute with hemorrhage and perforation without mention of obstruction
53321 Site unspecified acute with hemorrhage and perforation without mention of obstruction
Gastrojejunal ulcer:
53400 Acute with hemorrhage without mention of obstruction
53401 Acute with hemorrhage with obstruction
53410 Acute with perforation without mention of obstruction
53411 Acute with perforation with obstruction
53420 Acute with hemorrhage and perforation without mention of obstruction
53421 Acute with hemorrhage and perforation with obstruction
53430 Acute without mention of hemorrhage or perforation - without mention of obstruction
Alcoholism
ICD-9-CM diagnosis codes (when principal diagnosis):
Denominator:
All medical and surgical discharges defined by specific DRGs (see denominators for Decubitus
Ulcer for medical discharges and Complications of Anesthesia for surgical discharges).
Iatrogenic Pneumothorax
Numerator:
Discharges with ICD-9-CM code of 5121 in any secondary diagnosis field per 1,000 discharges.
Denominator:
All medical and surgical discharges defined by specific DRGs (see denominators for Decubitus
Ulcer for medical discharges and Complications of Anesthesia for surgical discharges).
Exclude:
Patients with any diagnosis of trauma.
Patients with any code indicating thoracic surgery, lung or pleural biopsy, or cardiac surgery.
Obstetrical patients in MDC 14 (Pregnancy, Childbirth and the Puerperium).
Trauma
ICD-9-CM diagnosis codes (includes 4th and 5th digits):
DRGs:
Lung transplant:
3350 Lung transplantation, NOS
3351 Unilateral lung transplantation
3352 Bilateral lung transplantation
336 Combined heart-lung transplantation
3392 Ligation of bronchus
3393 Puncture of lung
3398 Other operations on bronchus
3399 Other operations on lung
3329 Other diagnostic procedure on lung and bronchus
3333 Pneumoperitoneum for collapse of lung
3401 Incision of chest wall
3402 Exploratory thoracotomy
3403 Reopening of recent thoracotomy site
3405 Creation of pleuroperitoneal shunt
3409 Other incision of pleura
341 Incision of mediastinum
Operations on diaphragm:
3481 Excision of lesion or tissue of diaphragm
3482 Suture of laceration of diaphragm
3483 Closure of fistula of diaphragm
3484 Other repair of diaphragm
3485 Implantation of diaphragmatic pacemaker
3489 Other operations on diaphragm
3493 Repair of pleura
3499 Other operations on thorax, other
Esophagotomy:
4201 Incision of esophageal web
4209 Other incision of esophagus
4210 Esophagostomy, NOS
4211 Cervical esophagostomy
4212 Exteriorization of esophageal pouch
4219 Other external fistulization of esophagus
4221 Operative esophagoscopy by incision
4225 Open biopsy of esophagus
4231 Local excision of esophageal diverticulum
4232 Local excision of other lesion or tissue of esophagus
4239 Other destruction of lesion or tissue of esophagus
Excision of esophagus:
4240 Esophagectomy, NOS
4241 Partial esophagectomy
4242 Total esophagectomy
Antesternal anastomosis
4261 Antesternal esophagoesophagostomy
4262 Antesternal esophagogastrostomy
4263 Antesternal esophageal anastomosis with interposition of small bowel
4264 Other antesternal esophagoenterostomy
4265 Antesternal esophageal anastomosis with interposition of colon
4266 Other antesternal esophagocolostomy
4268 Other antesternal esophageal anastomosis with interposition
4269 Other antesternal anastomosis of esophagus
427 Esophagomyotomy
Cardiac Surgery
DRGs:
Immunocompromised States
ICD-9-CM diagnosis codes:
Cancer
ICD-9-CM diagnosis codes (include 4th and 5th digits):
DRGs:
Postoperative Hematoma
ICD-9-CM diagnosis code:
Postoperative Hemorrhage
ICD-9-CM diagnosis code:
Drainage of Hematoma
ICD-9-CM procedure codes:
Denominator:
Hip Fracture
ICD-9-CM diagnosis codes (includes all 5th digits):
Denominator:
All surgical discharges defined by specific DRGs (see denominator for Complications of
Anesthesia).
Exclude:
Patients who have diseases and disorders of the musculoskeletal system and connective tissue
(MDC 8).
Patients with principal diagnosis codes for seizure, syncope, stroke, coma, cardiac arrest,
poisoning, trauma, delirium and other psychoses, or anoxic brain injury.
Patients with any diagnosis of metastatic cancer, lymphoid malignancy or bone malignancy, or
self-inflicted injury.
Obstetrical patients in MDC14 (Pregnancy, Childbirth and the Puerperium).
Patients 17 years of age and younger.
Seizure
ICD-9-CM diagnosis codes:
Syncope
ICD-9-CM diagnosis codes:
Stroke
ICD-9-CM diagnosis codes:
Coma
ICD-9-CM diagnosis codes:
Cardiac Arrest
ICD-9-CM diagnosis code:
Poisoning
ICD-9-CM diagnosis codes (includes 4th and 5th digits):
Trauma
ICD-9-CM diagnosis codes (includes 4th and 5th digits):
DRGs:
Metastatic Cancer
ICD-9-CM diagnosis codes (includes 4th and 5th digits):
Lymphoid Malignancy
ICD-9-CM diagnosis codes (includes 4th and 5th digits):
Bone Malignancy
ICD-9-CM diagnosis code (includes 4th and 5th digits):
Self-Inflicted Injury
ICD-9-CM diagnosis codes:
Denominator:
All elective surgical discharges defined by admission type and specific DRGs (see denominator
for Complications of Anesthesia for surgical discharges).
Elective
Diabetes
ICD-9-CM diagnosis codes (includes 4th and 5th digits):
Cardiac Arrhythmia
ICD-9-CM diagnosis codes:
DRGs:
Cardiac Arrest
ICD-9-CM diagnosis code:
Shock
ICD-9-CM diagnosis codes:
Hemorrhage
ICD-9-CM diagnosis codes:
Pulmonary Embolism
ICD-9-CM diagnosis codes:
Denominator:
All surgical discharges defined by specific DRGs (see denominator for Complications of
Anesthesia).
Exclude:
Patients with a principal diagnosis of deep vein thrombosis.
Obstetrical patients in MDC 14 (Pregnancy, Childbirth and the Puerperium).
Patients with secondary procedure code 387 when this procedure occurs on the day of or
previous to the day of the principal procedure.
Elective
Postoperative Sepsis
Numerator:
Discharges with ICD-9-CM code for sepsis in any secondary diagnosis field per 1,000 elective
surgical discharges.
Sepsis
ICD-9-CM diagnosis codes:
0380 Streptococcal septicemia
03810 Staphylococcal septicemia, unspecified
03811 Staphylococcus aureus septicemia
03819 Other staphylococcal septicemia
0382 Pneumococcal septicemia (streptococcus pneumoniae septicemia)
0383 Septicemia due to anaerobes
Denominator:
All elective surgical discharges (see denominator for Complications of Anesthesia for surgical
discharges).
Elective
Infection
ICD-9-CM diagnosis codes:
Immunocompromised States
ICD-9-CM diagnosis codes:
Cancer
ICD-9-CM diagnosis codes (includes 4th and 5th digits):
DRGs:
Abdominopelvic
ICD-9-CM procedure codes:
Denominator:
Transfusion Reaction
Numerator:
Discharges with ICD-9-CM codes for transfusion reaction in any secondary diagnosis field per
1,000 discharges.
Transfusion Reaction
ICD-9-CM diagnosis codes:
Denominator:
All medical and surgical discharges defined by specific DRGs (see denominators for
Complications of Anesthesia for surgical discharges and Decubitus Ulcer for medical
discharges).
Birth Trauma
ICD-9-CM diagnosis codes:
7670 Subdural and cerebral hemorrhage (due to trauma or to intrapartum anoxia or hypoxia)
7673 Injuries to skeleton (excludes clavicle)
7674 Injury to spine and spinal cord
7677 Other cranial and peripheral nerve injuries
7678 Other specified birth trauma
7679 Birth trauma, unspecified
Denominator:
All liveborn infants.
Liveborn
DRGs:
AND
Exclude:
Infants with a subdural or cerebral hemorrhage (subgroup of birth trauma coding - 7670) and any
diagnosis code of pre-term infant (denoting a birth weight of less than 2,500 grams and less than
37 weeks gestation, or 34 weeks gestation or less).
Infants with injury to skeleton (7673, 7674) and any diagnosis code of osteogenesis imperfecta
(75651).
Preterm infant
ICD-9-CM diagnosis codes:
Obstetric Trauma
ICD-9-CM diagnosis codes:
66430,1,4 Trauma to perineum and vulva during delivery, fourth degree perineal laceration
66530,1,4 Other obstetrical trauma, laceration of cervix
66540,1,4 Other obstetrical trauma, high vaginal lacerations
66550,1,4 Other obstetrical trauma, other injury to pelvic organs
Denominator:
All cesarean delivery discharges.
Cesarean Delivery
DRGs:
Obstetric Trauma
ICD-9-CM diagnosis codes:
66430,1,4 Trauma to perineum and vulva during delivery, fourth degree perineal laceration
66530,1,4 Other obstetrical trauma, laceration of cervix
Denominator:
All vaginal delivery discharges with any procedure code for instrument-assisted delivery.
Vaginal Delivery
DRGs:
Instrument-Assisted Delivery
ICD-9-CM procedure codes:
Obstetric Trauma
ICD-9-CM diagnosis codes:
66430,1,4 Trauma to perineum and vulva during delivery, fourth degree perineal laceration
66530,1,4 Other obstetrical trauma, laceration of cervix
66540,1,4 Other obstetrical trauma, high vaginal lacerations
66550,1,4 Other obstetrical trauma, other injury to pelvic organs
Denominator:
All vaginal delivery discharge patients.
Vaginal Delivery
DRGs:
Instrument-Assisted Delivery
ICD-9-CM procedure codes
Empirical analyses were conducted to provide additional information about the indicators. These
analyses were intended not as decision making tools, but rather explorations into the characteristics of
the indicators. Specifically, these analyses explore the frequency and variation of the indicators, the
potential bias, based on limited risk adjustment, and the relationship between indicators.
Analysis Approach
Data sources. The data sources used in the empirical analyses were the 1997 Florida State
Inpatient Database (SID) (for initial testing and development; 1995-1997 used for persistence analysis)
and the 1997 State Inpatient Databases (SID) for 19 HCUP participating States, referred to in this report
as the National SID (for the final empirical analysis). The Florida SID consists of about 2 million
discharges from over 200 hospitals, and was chosen because Florida is a large diverse State. The
National SID consists of about 19 million discharges from over 2,300 hospitals. The National SID
contains all-payer data on hospital inpatient stays from participating States (Arizona, California, Colorado,
Connecticut, Florida, Illinois, Iowa, Kansas, Maryland, Massachusetts, Missouri, New Jersey, New York,
Oregon, Pennsylvania, South Carolina, Tennessee, Washington, and Wisconsin). All discharges from
participating States community hospitals are included in the SID database, which defines community
hospitals as non-Federal, short-term, general, and other specialty hospitals, excluding long-term hospitals
and hospital units of long-term care institutions, psychiatric hospitals, and alcoholism and chemical
dependency treatment facilities.
A complete description of the content of the SID, including details of the participating States
discharge abstracts, can be found on the Agency for Healthcare Research and Quality Web site
(www.ahrq.gov/data/hcup/hcupsid.htm). Because the Florida SID was used only for initial testing and
development, the empirical results reported are from the National SID. Descriptive results from the
Florida SID are reported for comparison to ensure that the hospital-level results were similar in both data
sources. Differences between Florida and national results are pointed out in the text. The National SID
data were also used for the construction of area measures, with data from the U.S. Census Bureau used
to construct the denominator of these rates.
Reported patient safety indicators. Three sets of patient safety indicators were examined. First,
the Accepted patient safety indicators met the face validity criteria established through the literature
review and clinician panel review. Second, the Experimental patient safety indicators did not meet those
criteria, but appeared to warrant further testing and evaluation. Third, several Accepted patient safety
indicators were modified into area indicators, which were designed to assess the total incidence of the
adverse event within geographic areas. For example, the project team constructed an indicator for
Transfusion reaction at both the hospital and area levels. Transfusion reactions that occur after
discharge from a hospitalization would result in a readmission. The area-level indicator includes these
cases, while the hospital level restricts the number of transfusion reactions to only those that occur during
the same hospitalization that exposed the patient to this risk.
All potential indicators were examined empirically by developing and conducting statistical tests
for precision, bias, and relatedness of indicators. For each indicator, the project team calculated five
different estimates of hospital performance:
1. The raw indicator rate was calculated using the number of adverse events in the numerator divided
by the number of discharges in the population at risk by hospital. For the area indicators, the
denominator is the population of the Metropolitan Statistical Area (MSA), New England County
Metropolitan Area (for the New England States) or county (for non-MSA areas) of the hospital.
2. The raw indicator was adjusted using a logistic regression to account for differences among hospitals
(and areas) in demographics (specifically, age and gender). Age was modeled using a set of dummy
variables to represent 10-year categories except for young children, whose age categories are
Version 2.1 139 Revision 1 (May 28, 2003)
narrower (i.e., less than 1, 1-4, 5-14, 15-24, 25-34, 35-44, 45-54, 55-64, 65-74, 75-84, and 85 or more
years), along with a parallel set of age-gender interactions. Because of sparse cells, certain age
categories were combined or omitted for selected indicators, such as the obstetric indicators.
3. The raw indicator was adjusted to account for differences among hospitals in age, gender and
modified DRG category (as described below).
4. The raw indicator was adjusted to account for differences among hospitals in age, gender, modified
DRG, and comorbidities (defined using an adaptation of the AHRQ comorbidity software) of patients.
5. Multivariate signal extraction (MSX) methods were applied to adjust for reliability by estimating the
amount of noise (i.e., variation due to random error) relative to the amount of signal (i.e.,
systematic variation in hospital performance or the reliability) for each indicator. This or similar
reliability adjustment has been used in the literature for similar purposes.125 126 Mutlivariate methods
(taking into account correlations among indicators to extract additional signal) were applied to most of
the accepted indicators. The exceptions were Death in Low Mortality DRGs and Failure to Rescue.
Only univariate signal extraction methods (smoothing) were applied to these two indicators and to the
experimental indicators, because these indicators possibly cover broader clinical concepts.
Correlations between these indicators and other indicators may not reflect correlations due to quality
of care, and thus inclusion of these indicators may adversely affect the MSX approximations.
For additional details on the empirical methods, refer to the companion EPC HCUP Quality Indicator
Report, published by AHRQ (http://www.ahrq.gov/data/hcup/qirefine.htm). Additional details on the
modifications made to the DRG and comorbidity categories are described below.
Hospital Fixed Effects. In the risk-adjustment models, hospital fixed effects were calculated using
the standard method with logistic models of first estimating the predicted value for each discharge, then
subtracting the actual outcome from the predicted, and averaging the difference for each hospital to get
the hospital fixed effect estimate. In the Quality Indicator Report,127 linear regression models were used
with hospital fixed effects included, arguing that the logistic approach yielded biased estimates due to the
omission of a variable (the hospital) correlated with both the dependent (e.g., in-hospital mortality) and
the independent (e.g., age, gender, APR-DRG) variables in the model. Given the rare occurrence of
many of the PSIs, however, the logistic approach may be more appropriate for this application. Linear
methods assume that the error term is normally distributed. This assumption is violated when the
outcome is dichotomous.
The QI means were generally an order of magnitude higher than the PSI means, so the
assumption was not as problematic. However, the most appropriate method depends on the particular
characteristics of each indicator, whether QI or PSI. To the extent that bias is a concern, accounting for
the clustering of patients by using a hospital fixed effect is advantageous. To the extent that extreme
values are a concern, imposing structure on the error term with logistic methods is advantageous. In the
end, the two approaches can be compared in terms of how much difference it makes in the relative
assessment of provider performance. This issue warrants further analysis to better understand the trade-
offs and limitations of each approach, and under what conditions and for what indicators each approach
might best apply.
Ykit is the kth PSI for patient i in year t (i.e., whether or not the event associated with the indicator
occurred on that discharge).
Zit is a vector of patient covariates for patient i in year t (i.e., the patient-level measures used as risk
adjusters).
kt is a vector of parameters in each year t, giving the effect of each patient risk adjuster on indicator k
(i.e., the magnitude of the risk adjustment associated with each patient measure).
In the second step, the hospital effect was estimated by subtracting the resulting predictions from
this patient-level regression from the actual observed patient-level outcomes, and taking the mean of this
difference for each hospital. That is, for each hospital j (j=1,,J),
Mkjt is the raw adjusted measure for indicator k for hospital j in year t (i.e., the hospital fixed effect in
the patient-level regression).
Zit is the vector of patient covariates for patient i in year t estimated in Step 1.
In addition to age, sex, and age*sex interactions as adjusters in the model, the project team also
included a modified DRG and comorbidity category for the admission.
Modified DRG Categories. Two modifications were made to the Centers for Medicare and
Medicaid Services (CMS, formerly Health Care Financing Administration) DRGs. First, adjacent DRG
categories that were separated by the presence or absence of comorbidities or complications were
collapsed. For example, DRGs 076 (Other Resp System Operating Room Procedures w CC) and 077
(Other Resp System Operating Room Procedures w/o CC) were grouped into one category. The purpose
was to avoid adjusting for the complication the team was trying to measure. Second, most of the super-
MDC DRG categories were excluded from the logistic models. Excluding these categories also avoids
adjusting for the complications the team was trying to measure. For example, tracheostomies (DRG 482-
483) often result from potentially preventable respiratory complications that require long-term mechanical
ventilation. Similarly, operating room procedures unrelated to the principal diagnosis (DRG 468, 477)
often result from potentially preventable complications that require surgical repair (i.e., fractures,
lacerations).
In the companion technical report on quality indicators, the risk adjustment method implemented
All Patient Refined (APR)-DRGs, a refinement of DRGs to capture different levels of complications.
However, patient safety indicators, designed to detect potentially preventable complications, require a risk
adjustment approach that does not inherently remove the differences between patients based on their
complications. The APR-DRGs could be modified to remove applicable complications, on an indicator-by-
indicator basis, but implementation of such an approach was beyond the scope of the current project. In
this report, APR-DRG risk adjustment was not implemented.
Modified Comorbidity Software. To adjust for comorbidities, the project team used an updated
adaptation of AHRQ Comorbidity Software (http://www.ahrq.gov/data/hcup/comorbid.htm). The ICD-9-
CM codes used to define the comorbidity categories were modified to address four main issues.
1. Comorbidity categories were excluded in the current software that include conditions likely to
represent potentially preventable complications in certain settings, such as after elective surgery.
Specifically, three DRG categories (cardiac arrhythmia, coagulopathy, and fluid/electrolyte disorders)
were removed from the comorbidity adjustment.
3. The comorbidity definitions did not include obstetric comorbidity codes, which are relevant for the
obstetric indicators. Codes, when available, for these comorbidities in obstetric patients were added.
Low Mortality DRGs. In order to be included in the Low Mortality DRG indicator, the DRG had
to have an overall in-hospital mortality rate (based on the National SID sample) of less than 0.5%. In
addition, if a DRG category was split based on the presence of comorbidities or complications, then the
category was included only if both DRGs (with and without comorbidities or complications) met the
mortality threshold. Otherwise, the category was not included in the Low mortality DRG PSI. The
indicator is reported as a single measure and stratified into medical (adult and pediatric), surgical (adult
and pediatric), neonatal, obstetric and psychiatric DRGs.
Using these methods, the project team constructed a set of statistical tests to examine precision,
bias, and relatedness of indicators for all accepted hospital-level indicators, and precision and bias for all
accepted area-level and experimental indicators. Each of the key statistical test results was summarized
and explained in the overview section of the companion HCUP Quality Indicator report.128 Tables B-1
through B-3 provide a summary of the statistical analyses and their interpretation.
Table B-1. Precision Tests
128
Davies et al., 2001.