Journal of the American Medical Informatics Association Volume 8 Number 6 Nov / Dec 2001 527

White Paper ■

Clinical Decision Support

Systems for the Practice of
Evidence-based Medicine

IDA SIM, MD, PHD, PAUL GORMAN, MD, ROBERT A. GREENES, MD, PHD,
R. BRIAN HAYNES, MD, PHD, BONNIE KAPLAN, PHD,
HAROLD LEHMANN, MD, PHD, PAUL C. TANG, MD

A b s t r a c t Background: The use of clinical decision support systems to facilitate the practice
of evidence-based medicine promises to substantially improve health care quality.
Objective: To describe, on the basis of the proceedings of the Evidence and Decision Support
track at the 2000 AMIA Spring Symposium, the research and policy challenges for capturing
research and practice-based evidence in machine-interpretable repositories, and to present
recommendations for accelerating the development and adoption of clinical decision support
systems for evidence-based medicine.
Results: The recommendations fall into five broad areas—capture literature-based and practice-
based evidence in machine-interpretable knowledge bases; develop maintainable technical and
methodological foundations for computer-based decision support; evaluate the clinical effects and
costs of clinical decision support systems and the ways clinical decision support systems affect and
are affected by professional and organizational practices; identify and disseminate best practices
for work flow–sensitive implementations of clinical decision support systems; and establish public
policies that provide incentives for implementing clinical decision support systems to improve
health care quality.
Conclusions: Although the promise of clinical decision support system–facilitated evidence-based
medicine is strong, substantial work remains to be done to realize the potential benefits.
■ J Am Med Inform Assoc. 2001;8:527–534.

Affiliations of the authors: University of California–San Francisco, Clinical decision support systems (CDSSs) have been
California (IS); Oregon Health and Science University, Portland, hailed for their potential to reduce medical errors1
Oregon (PG); Harvard Medical School (RAG); McMaster
University, Hamilton, Ontario, Canada (RBH); Yale University
and increase health care quality and efficiency.2 At
School of Medicine, Hamden, Connecticut (BK); Johns Hopkins the same time, evidence-based medicine has been
University School of Medicine, Baltimore, Maryland (HL); Palo widely promoted as a means of improving clinical
Alto Medical Foundation, Palo Alto, California (PCT). outcomes, where evidence-based medicine refers to
This work was supported in part by a United States Presidential the practice of medicine based on the best available
Early Career Award for Scientists and Engineers awarded to Dr. scientific evidence. The use of CDSSs to facilitate evi-
Sim and administered through grant LM-06780 of the National
Library of Medicine. dence-based medicine therefore promises to substan-
Correspondence and reprints: Ida Sim, MD, PhD, Department of
tially improve health care quality.
Medicine and Program in Biological and Medical Informatics, The Evidence and Decision Support track of the 2000
University of California–San Francisco, 400 Parnassus Avenue, AMIA Spring Symposium examined the challenges
Room A-405, San Francisco, CA 94143-0320; e-mail: <sim@medi-
cine.ucsf.edu>. in realizing the promise of CDSS-facilitated evidence-
Received for publication: 2/19/01; accepted for publication:
based medicine. This paper describes the activities of
7/11/01. this track and summarizes discussions in specific
528 SIM ET AL., Decision Support Systems for Evidence-based Medicine

research and policy recommendations for accelerat- Process

ing the development and adoption of CDSSs for evi-
dence-based medicine. The speakers for the Evidence and Decision Support
track are listed at the end of this paper. The track con-
Definitions sisted of three panels and two break-out discussion
sessions.
We introduce a new term, “evidence-adaptive CDSSs,”
The first panel addressed the role of information tech-
to distinguish a type of CDSS that has technical and
nology in the dissemination and critical appraisal of
methodological requirements that are not shared by
research evidence, the technical challenges and oppor-
CDSSs in general. To clarify this distinction between
tunities of evidence-adaptive computerized decision
evidence-adaptive and other CDSSs, we define the fol-
support, and the organizational and workflow issues
lowing terms as they are used in this paper:
that arise when effecting practice change through
■ Evidence-based medicine. Evidence-based medicine information technology (Haynes, Tang, and Kaplan,
is the management of individual patients through respectively).
individual clinical expertise integrated with the The second panel presented two case studies of evi-
conscientious and judicious use of current best dence-based quality improvement projects (Packer,
evidence from clinical care research.3 This Stone) and summarized the status of the GuideLine
approach makes allowances for missing, incom- Interchange Format (GLIF), a developing founda-
plete, or low-quality evidence and requires the tional technology for distributed evidence-adaptive
application of clinical judgment. CDSSs (Greenes). Finally, a commentator panel
The scientific literature is the major source of evi- expanded on some of the pitfalls to changing practice
dence for evidence-based medicine, although liter- through technology (Gorman) and on the informa-
ature-based evidence should often be comple- tion-technology funding agenda of the Agency for
mented by local, practice-based evidence for indi- Healthcare Research and Quality (Burstin).
vidual and site-specific clinical decision making. Interspersed with these panel presentations were two
Evidence-based medicine is conducted by the moderated break-out sessions, in which participants
health care provider and may or may not be com- worked to identify the research and policy needs and
puter-assisted. priorities for effective computer-supported practice
change.
■ Clinical decision support system (CDSS). In this paper,
we define clinical decision support systems to be All conference sessions were audiotaped. Using these
software that designed to be a direct aid to clinical audiotapes, we distilled five central areas of activity
decision-making, in which the characteristics of an that are essential to the goal of increased adoption of
individual patient are matched to a computerized CDSSs for evidence-based medicine.
clinical knowledge base and patient-specific assess-
ments or recommendations are then presented to ■ Capture of both literature-based and practice-
the clinician or the patient for a decision. based research evidence into machine-inter-
pretable formats suitable for CDSS use
■ Evidence-adaptive CDSS. This paper focuses on a
subclass of CDSSs that are evidence-adaptive, in ■ Establishment of a technical and methodological
which the clinical knowledge base of the CDSS is foundation for applying research evidence to indi-
derived from and continually reflects the most up- vidual patients at the point of care
to-date evidence from the research literature and
practice-based sources. For example, a CDSS for ■ Evaluation of the clinical effects and costs of
cancer treatment is evidence-adaptive if its knowl- CDSSs, as well as how CDSSs affect and are affect-
edge base is based on current evidence and if its ed by professional and organizational practices
recommendations are routinely updated to incor- ■ Promotion of the effective implementation and use
porate new research findings. Conversely, a CDSS of CDSSs that have been shown to improve clini-
that alerts clinicians to a known drug–drug inter- cal performance or outcomes
action is evidence-based but not evidence-adap-
tive if its clinical knowledge base is derived from ■ Establishment of public policies that provide in-
scientific evidence, but no mechanisms are in place centives for implementing CDSSs to improve
to incorporate new research findings. health care quality
Journal of the American Medical Informatics Association Volume 8 Number 6 Nov / Dec 2001 529

The Role of Evidence in research for automated updating of their knowledge

Evidence-adaptive CDSSs bases. The Trial Bank project is a collaboration with
the Annals of Internal Medicine and JAMA to capture
the design and results of randomized trials directly
Clinical decision support systems can be only as effec-
into structured knowledge bases17 and is a first step
tive as the strength of the underlying evidence base.
toward the transformation of text-based literature
That is, the effectiveness of CDSSs will be limited by
into a shared, machine-interpretable resource for evi-
any deficiencies in the quality or relevance of the
dence-adaptive CDSSs.
research evidence. Therefore, one key step in develop-
ing more effective CDSSs is to generate not simply Practice-based Evidence
more clinical research evidence, but more high-quali-
ty, useful, and actionable evidence that is up-to-date, Although the research literature serves as the foun-
easily accessible, and machine interpretable. dation for evidence-based practice, it is not uncom-
mon that local, practice-based evidence is required
Literature-based Evidence for optimizing health outcomes. For example, ran-
domized trials have shown that patients with symp-
Only about half the therapeutic interventions used in
tomatic carotid artery stenosis have fewer strokes if
inpatient4,5 and outpatient6 care in internal and family
they receive a surgery called carotid endarterecto-
medicine are supported in the research literature with
my.18 If complication rates from the surgery are
evidence of efficacy. The other half of the interventions
greater than about 6 percent, however, the benefits
either have not been studied or have only equivocal
are nullified.19 Despite this, only 19 percent of physi-
supportive evidence. Several problems exist with
cians know the CEA complication rates of the hospi-
using the research literature for evidence-based medi-
tals in which they operate or to which they refer
cine. First, the efficacy studies of clinical practice that
patients.20 For clinical problems with locally variable
form the basis for evidence-based medicine constitute
parameters, therefore, developers of CDSSs should
only a small fraction of the total research literature.7
place a high priority on obtaining local practice-
Furthermore, this clinical research literature has been
based evidence to complement the literature-based
beset for decades with study design and reporting
evidence.
problems8,9—problems that still exist in the recent ran-
domized trial,10 systematic review,11,12 and guide- Practice-based evidence may also be useful for the
lines13 literature. As the volume of research publica- development of practice guidelines. Although the
tion explodes while quality problems persist, it is not evidentiary support for individual decision steps in a
surprising that most clinicians consider the research guideline comes primarily from literature-based evi-
literature to be unmanageable14 and of limited appli- dence, as discussed above, a guideline’s process flow
cability to their own clinical practices.15,16 is usually constructed on the basis of expert opinion
only. With more practice-based information on clini-
The full promise of CDSSs for facilitating evidence-
cal processes and events, however, guideline devel-
based medicine will occur only when CDSSs can
opers may be able to improve the way they design
“keep up” with the literature—that is, when evi-
process flows.
dence-adaptive CDSSs can monitor the literature for
new relevant studies, identify those that are of high As useful as practice-based evidence may be, it is
quality, and then incorporate the best evidence into often not easy to come by. The informatics communi-
patient-specific assessments or practice recommen- ty can foster this much-needed research by develop-
dations. Automation of these tasks remains an open ing information technologies for practice-based
area of research. In the meantime, the best electronic research networks to automatically capture clinical
resources for evidence-based medicine include the processes and events in diverse outpatient settings.
Cochrane Library, Best Evidence, and Clinical Many research and policy issues concerning these
Evidence, resources that cull the best of the literature research networks—from the standardization of data
to provide an up-to-date solid foundation for evi- items to data ownership and patient privacy—are
dence-based practice. The drawback to these active areas of inquiry.21–24
resources is that their contents are textual and thus
not machine-interpretable by present-day CDSSs. Patient-directed Evidence

In contrast, if the research literature were available as The Internet and other sources of research evidence
shared, machine-interpretable knowledge bases, then have provided patients with many more options for
CDSSs would have direct access to the newest obtaining health information but have also increased
530 SIM ET AL., Decision Support Systems for Evidence-based Medicine

the potential for patients to misinterpret or become ■ Continue to develop better methods for synthesiz-
misinformed about research results.24,25 As a result, ing results from a wide variety of study designs,
patients are now less dependent on clinicians for from randomized trials to observational studies .
information, but still trust clinicians the most for help
■ Develop shareable, machine-interpretable reposi-
with selecting, appraising, and applying a profusion
tories of up-to-date evidence of multiple types
of information to health decisions.26 Clinical decision
(e.g., from clinical trials, systematic reviews, deci-
support systems can support this growing involve-
sion models).
ment of patients in clinical decision making through
interactive tools that allow patients to explore rele- ■ Develop shareable, machine-readable repositories
vant information that can foster shared decision mak- of executable guidelines that are linked to up-to-
ing.27,28 Systems that provide both patients and clini- date evidence repositories.
cians with valid, applicable, and useful information
may result in care decisions that are more concordant ■ Define and build standard interfaces among these
with current recommendations, are better tailored to repositories, to allow evidence to be linked auto-
individual patients, and ultimately are associated matically among systems for systematic review-
with improved clinical outcomes. The actual effects ing, decision modeling, and guideline creation
of these CDSSs on care decisions and outcomes and maintenance.
should be evaluated. ■ Develop an informatics infrastructure for prac-
tice-based research networks to collect practice-
Recommendations based evidence.

The gap between the current state of CDSSs and the Establishment of a Technical and
full promise of CDSSs for evidence-based medicine Methodological Foundation
suggests a research and development agenda. On the
basis of the expert panels and discussion sessions at Figure 1 depicts the informatics architecture that we
the Congress, we recommend the following steps for suggest is needed for CDSSs to facilitate evidence-
researchers, developers, and implementers to take in based practice. In this architecture, CDSSs are situat-
the five areas of activity essential to increasing adop- ed in a distributed environment that comprises mul-
tion of evidence-adaptive CDSSs. tiple knowledge repositories as well as the electronic
medical record. Vocabulary and interface standards
Capture of Literature-based and will be crucial for interoperation among these sys-
Practice-based Evidence tems. To provide patient-specific decision support at
the point of care, CDSSs need to interface with the
If clinical research is to improve clinical care, it must electronic medical record to retrieve patient-specific
be relevant, of high quality, and accessible. The data and, increasingly, also to effect recommended
research should provide evidence of efficacy, effec- actions through computerized order entry. Evidence-
tiveness, and cost-effectiveness for typical inpatient adaptive CDSSs also need to interface with up-to-
and outpatient practice settings.29 If CDSSs are to date repositories of clinical research knowledge. No
help translate this research into practice, CDSSs must longer should CDSSs be thought of as stand-alone
have direct machine-interpretable access to the expert systems.
research literature, so that automated methods can be
brought to bear on the myriad tasks involved in In addition to establishing standardized communica-
“keeping up with the literature.” Thus, the establish- tion among CDSSs, electronic medical records, and
ment of shared, machine-interpretable knowledge knowledge repositories, we also need better models
bases of research and practice-based evidence is a of individualized patient decision making in real-
critical priority. On the basis of discussions at the world settings. Formal models of decision making
conference, we identify six specific recommendations such as decision analysis are not commonly used;
for action: much methodological work remains to be done on
mapping real-world decision-making challenges to
Recommendations for Clinical and Informatics tractable computational approaches.
Researchers
We identify several additional priorities for evi-
■ Conduct better quality clinical research on the effi- dence-adaptive CDSSs in particular. These priorities
cacy, effectiveness, and efficiency of clinical inter- include the development of methods for adjusting for
ventions, particularly in primary care settings. the quality of the evidence base, and efficient, sus-
Journal of the American Medical Informatics Association Volume 8 Number 6 Nov / Dec 2001 531

tainable methods for ensuring that CDSS recommen-

dations reflect up-to-date evidence.

Recommendations for Researchers

and Developers
■ Continue development of a comprehensive, ex-
pressive clinical vocabulary that can scale from
administrative to clinical decision support needs.
■ Continue to develop shareable computer-based
representations of clinical logic and practice
guidelines.
■ Develop tools for knowledge editors to incorpo-
rate new literature-based evidence into CDSS
knowledge bases; specify the clinical context in
which that knowledge is applicable (e.g., that a
rule is for the treatment of stable outpatient dia-
betic patients only); and customize the literature- F i g u r e 1 Architecture for the capture and use of litera-
based evidence for local conditions (e.g., factoring ture-based and practice-based evidence, showing the dis-
in local surgical complication rates). tributed nature of the knowledge and functionality
involved in the use of CDSSs to support evidence-based
■ Explore and develop automatic methods for updat- medicine. Vocabulary and interface standards are needed
ing CDSS knowledge bases to reflect the current for inter-operation of the various systems.
state and quality of the literature-based evidence.
■ Develop more flexible models of decision making ■ Explicitly describe the care delivery setting and
that can accommodate clinical evidence of varying clinical scenarios for which the CDSS is applicable
methodological strength and relevance, so that (e.g., that a CDSS for diabetes treatment is intend-
evidence from randomized trials (Level I evidence ed for the management of stable outpatient diabet-
by U.S. Preventive Services Task Force criteria30) is ics only).
accorded more weight than evidence from case
■ Integrate CDSSs with electronic medical records
reports or expert opinion (Level III evidence).
and other relevant systems using appropriate
■ Develop models of decision making that can interoperability standards (e.g., HL-7).
simultaneously accommodate the beliefs, perspec-
■ Develop more CDSSs for outpatient settings.
tives, and values of multiple decision makers,
Approximately 60 percent of U.S. physicians prac-
including those of physicians and patients.
tice in outpatient settings, where an aging popula-
■ Develop methods for constructing and selecting tion is requiring increasingly complex diagnostic,
among decision models of scalable granularity treatment, and supportive services.
and specificity that are neither too general nor too
specific for the case at hand. Recommendation for Policy Makers,
Organizations, and Manufacturers
Recommendations for
■ Fund development and demonstration of inter-
Current CDSS Developers
organizational sharing of evidence-based knowl-
■ Adopt and use standard vocabularies and stan- edge and its application in diverse CDSSs.
dards for knowledge representation (e.g., GLIF) as
they become available. Evaluation of Clinical Decision Support Systems
■ We consider it axiomatic that CDSSs must be Despite the promise of CDSSs for improving care,
based on the best available evidence. Incorporate formal evaluations have shown that CDSSs have only
into the CDSS knowledge base the current best lit- a modest ability to improve intermediate measures
erature-based and practice-based evidence, and such as guideline adherence and drug dosing accura-
either provide mechanisms for keeping the knowl- cy.31–34 The effect of CDSSs on clinical outcomes
edge base up-to-date or explain why keeping up remains uncertain.32 Thus, more evaluations of
with the evidence is not applicable. CDSSs are needed to produce valid and generaliz-
532 SIM ET AL., Decision Support Systems for Evidence-based Medicine

able findings on the clinical and organizational evaluated to establish the presence or absence of
aspects of CDSS use. A wide variety of evaluation clinical benefits. Any randomized clinical trials
methods are available,35–37 and both quantitative and that are conducted should have sufficient sample
qualitative methods should be used to provide com- sizes to detect clinically meaningful outcomes,
plementary insight into the use and effects of CDSSs. should randomize physicians or clinical units
All types of evaluation studies, not just randomized rather than patients, and should be analyzed using
trials, deserve increased attention and funding.38,39 methods appropriate for cluster randomization
studies.
In light of the current focus on errors in medicine, a
special class of evaluation study deserves particular ■ Establish partnerships between academic groups
mention. These studies are ongoing, iterative reeval- and community practices to conduct evaluations.
uations and redesigns of CDSSs that identify and
amplify system benefits while identifying and miti- Promotion of the Implementation of CDSSs
gating unanticipated system errors or dangers. The
rationale for these types of studies is that automation Relatively few examples of CDSSs can be found in
in other industries has not always been beneficial, practice. In part, this limited adoption may be
and indeed, automation can interfere with and because CDSSs are as much an organizational as a
degrade overall organizational performance.40 technical intervention, and organizational, profes-
Woods and Patterson 41 offer a cautionary note from sional, and other challenges to implementing CDSSs
the transportation industry: may be as daunting as the technical challenges.

Despite the fact that these systems are often justified Recommendations for CDSS Implementers
on the grounds that they would help offload work
from harried practitioners, we find that they in fact ■ Establish a CDSS implementation team composed
create new additional tasks, force the user to adopt of clinicians, information technologists, managers,
new cognitive strategies, require more knowledge and evaluators to work together to customize and
and more communication, at the very times when the
implement the CDSS.
practitioners are most in need of true assistance .
■ Develop a process for securing clinician agree-
Clinicians and health care managers must be contin-
ment regarding the science underlying the recom-
uously vigilant against unforeseen adverse effects of
mendations of a CDSS. For evidence-adaptive
CDSS use.
CDSSs, a process is also needed for maintaining
Recommendations for Evaluators clinician awareness of and agreement with any
changes in CDSS recommendations that may
■ Evaluate CDSSs using an iterative approach that result from new evidence.
identifies both benefits and unanticipated prob-
lems related to CDSS implementation and use: all ■ Plan explicitly for work flow re-engineering and
CDSSs can benefit from multiple stages and types other people, organizational, and social issues and
of testing, at all points of the CDSS life cycle. incorporate change management techniques into
system development and implementation. For
■ Conduct more CDSS evaluations in actual practice example, a CDSS that recommends immediate
settings, including ambulatory settings. angioplasty instead of thrombolysis as a new treat-
■ Use both quantitative and qualitative evaluation ment option for acute coronary syndromes will
methodologies to assess multiple dimensions of necessitate a major restructuring of the hospital’s
CDSS use and design (e.g., the correctness, relia- resource use and work practices.
bility, and validity of the CDSS knowledge base;
the congruence of system-driven processes with Establishment of Public Policies That Provide
clinical roles and work routines in actual practice; Incentives for Implementing CDSSs
and the return-on-investment of system imple- Significant financial and organizational resources are
mentation). Qualitative studies should incorporate often needed to implement CDSSs, especially if the
the expertise of ethnographers, sociologists, orga- CDSS requires integration with the electronic med-
nizational behaviorists, or other field researchers ical record or other practice systems. In a competitive
from within and without the medical informatics health care marketplace, financial and reimburse-
community, as applicable. ment policies can therefore be important drivers both
n If preliminary testing suggests that a CDSS could for and against the adoption of effective CDSSs. As
improve health outcomes, the CDSS should be more evaluation studies become available, policy
Journal of the American Medical Informatics Association Volume 8 Number 6 Nov / Dec 2001 533

makers will be better able to tailor these policies to ACP J Club. 1993;Nov–Dec:A16.
promote only those CDSSs that are likely to improve 8. Schor S, Karten I. Statistical evaluation of medical journal
manuscripts. JAMA. 1966;195(13):1123–8.
health care quality. 9. Fletcher RH, Fletcher SW. Clinical research in general medical
journals: a 30-year perspective. N Engl J Med. 1979;301(4):180–3.
Recommendations for Policy Makers 10. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh
S. Assessing the quality of randomized controlled trials: an
■ Develop financial and reimbursement policies that
annotated bibliography of scales and checklists. Control Clin
provide incentives for health-care providers to Trials. 1995;16(1):62–73.
implement and use CDSSs of proven worth. 11. Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis:
an update. Mt Sinai J Med. 1996;63(3–4):216–24.
■ Develop and implement financial and reimburse- 12. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF.
ment policies that reward the attainment of meas- Improving the quality of reports of meta-analyses of ran-
urable quality goals, as might be achieved by domised controlled trials: the QUOROM statement—Quality of
CDSSs. Reporting of Meta-analyses. Lancet. 1999;354(9193):1896–900.
13. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines
■ Promote coordination and leadership across the following guidelines? The methodological quality of clinical
health care and clinical research sectors to leverage practice guidelines in the peer-reviewed medical literature.
JAMA. 1999;281(20):1900–5.
informatics promotion and development efforts 14. Williamson JW, German PS, Weiss R, Skinner EA, Bowes FD.
by government, industry, AMIA, and others. Health science information management and continuing edu-
cation of physicians: a survey of U.S. primary care practitioners
Conclusions and their opinion leaders. Ann Intern Med. 1989;110(2):151–60.
15. McAlister FA, Graham I, Karr GW, Laupacis A. Evidence-
based medicine and the practicing clinician. J Gen Intern Med.
The coupling of CDSS technology with evidence- 1999;14(4):236–42.
based medicine brings together two potentially pow- 16. Greer AL. The state of the art vs. the state of the science: the
erful methods for improving health care quality. To diffusion of new medical technologies into practice. Int J
realize the potential of this synergy, literature-based Technol Assess Health Care. 1988;4(1):5–26.
17. Sim I, Owens DK, Lavori PW, Rennels GD. Electronic trial
and practice-based evidence must be captured into banks: a complementary method for reporting randomized tri-
computable knowledge bases, technical and method- als. Med Decis Making. 2000;20(4):440–50.
ological foundations for evidence-adaptive CDSSs 18. Cina CS, Clase CM, Haynes BR. Refining the indications for
must be developed and maintained, and public poli- carotid endarterectomy in patients with symptomatic carotid
stenosis: a systematic review. J Vasc Surg. 1999;30(4):606–17.
cies must be established to finance the implementa-
19. Chassin MR. Appropriate use of carotid endarterectomy [edi-
tion of electronic medical records and CDSSs and to torial]. N Engl J Med. 1998;339(20):1468–71.
reward health care quality improvement. 20. Goldstein LB, Bonito AJ, Matchar DB, et al. U.S national sur-
vey of physician practices for the secondary and tertiary pre-
The authors thank the many discussion participants whose anony- vention of ischemic stroke: design, service availability, and
mous comments were included in this paper. They also thank common practices. Stroke. 1995;26(9):1607–15.
Patricia Flatley Brennan for her helpful comments on an earlier 21. Nutting PA. Practice-based research networks: building the
draft of this manuscript, and Amy Berlin for her assistance in infrastructure of primary care research. J Fam Pract. 1996;
preparing the manuscript. 42(2):199–203.
22. Nutting PA, Baier M, Werner JJ, Cutter G, Reed FM, Orzano
References ■ AJ. Practice patterns of family physicians in practice-based
research networks: a report from ASPN. Ambulatory Sentinel
1. Bates DW, Cohen M, Leape LL, et al. Reducing the frequency Pratice Network. J Am Board Fam Pract. 1999;12(4):278–84.
of errors in medicine using information technology. J Am Med 23. van Weel C, Smith H, Beasley JW. Family practice research
Inform Assoc. 2001;8:299–308. networks: experiences from 3 countries. J Fam Pract. 2000;
2. Teich JM, Wrinn MM. Clinical decision support systems come 49(10):938–43.
of age. MD Comput. 2000;17(1):43–6. 24. Kaplan B, Brennan PF. Consumer informatics supporting
3. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson patients as co-producers of quality. J Am Med Inform Assoc.
WS. Evidence-based medicine: what it is and what it isn’t [edi- 2001;8:309–16.
torial]. BMJ. 1996;312(7023):71–2. 25. Jadad AR, Haynes RB, Hunt D, Browman GP. The Internet
4. Nordin-Johansson A, Asplund K. Randomized controlled tri- and evidence-based decision-making: a needed synergy for
als and consensus as a basis for interventions in internal med- efficient knowledge management in health care. CMAJ. 2000;
icine. J Intern Med. 2000;247(1):94–104. 162(3):362–5.
5. Suarez-Varela MM, Llopis-Gonzalez A, Bell J, Tallon-Guerola 26. Harris Interactive. Ethics and the Internet: Consumers vs.
M, Perez-Benajas A, Carrion-Carrion C. Evidence-based gen- Webmasters, Internet Healthcare Coalition, and National
eral practice. Eur J Epidemiol. 1999;15(9):815–9. Mental Health Association. Oct 5, 2000.
6. Ellis J, Mulligan I, Rowe J, Sackett DL. Inpatient general med- 27. Morgan MW, Deber RB, Llewellyn-Thomas HA, Gladstone P,
icine is evidence based. A-Team, Nuffield Department of Clin- O’Rourke K, et al. Randomized, controlled trial of an interac-
ical Medicine. Lancet. 1995;346(8972):407–10. tive videodisc decision aid for patients with ischemic heart
7. Haynes RB. Where’s the meat in clinical journals [editorial]? disease. J Gen Intern Med. 2000;15(10):685–93.
534 SIM ET AL., Decision Support Systems for Evidence-based Medicine

28. Health Dialog Videos. Available at: http://www.healthdia- 35. Friedman CP, Wyatt JC. Evaluation methods in medical infor-
log.com/video.htm. Accessed Feb 20, 2001. matics. New York: Springer-Verlag, 1997.
29. Haynes B. Can it work? Does it work? Is it worth it? The test- 36. Anderson JG, Aydin CE. Evaluating the impact of health care
ing of healthcare interventions is evolving [editorial]. BMJ. information systems. Int J Technol Assess Health Care. 1997;
1999;319(7211):652–3. 13(2):380–93.
30. United States Preventive Services Task Force. Guide to Clinical 37. Kaplan B. Evaluating informatics applications: social interac-
Preventive Services. 2nd ed. Appendix A: Task Force Ratings. tionism and call for methodological pluralism. Int J Med Inf.
1996. Available at: http://text.nlm.nih.gov/ftrs/directBrowse. 2001;84:39–56.
pl?dbName=cps&href=APPA. Accessed Jun 18, 2001. 38. Tierney WM, Overhage JM, McDonald CJ. A plea for con-
31. Kaplan B. Evaluating informatics applications: review of the trolled trials in medical informatics [editorial]. J Am Med
clinical decision support systems evaluation literature. Int J Inform Assoc. 1994;1(4):353–5.
Med Inf. 2001;64:15–37. 39. Stead WW, Haynes RB, Fuller S, Friedman CP, Travis LE, Beck
32. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of comput- JR, et al. Designing medical informatics research and library:
er-based clinical decision support systems on physician per- resource projects to increase what is learned. J Am Med
formance and patient outcomes: a systematic review. JAMA. Inform Assoc. 1994;1(1):28–33.
1998;280(15):1339–46. 40. Woods D. Testimony of David Woods, Past President, Human
33. Balas E, Austin S, Mitchell J, Ewigman B, Bopp K, Brown G. Factors and Ergonomics Society. In: National Summit on
The clinical value of computerized information services. Arch Medical Errors and Patient Safety Research. Washington, DC:
Fam Med. 1996;5:271–8. Quality Interagency Coordination Task Force, Sep 11, 2000.
34. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 41. Woods DD, Patterson ES. How unexpected events produce an
randomized controlled trials to evaluate computer-based clin- escalation of cognitive and coordinative demands. In:
ical reminder systems for preventive care in the ambulatory Hancock PA, Desmond PA (eds). Stress Workload and
setting. J Am Med Inform Assoc. 1996;3:399–409. Fatigue. Hillsdale, NJ: Erlbaum, 2001.

Appendix
PANELISTS AND GROUP LEADERS

Keynote Panelists: Expert Commentator Panelists:

R. Brian Haynes, MD, PhD, Chief, Health Information Paul Gorman, MD, Assistant Professor, Oregon Health and
Research Unit, McMaster University Science University
Paul Tang, MD, Medical Director of Clinical Informatics, Helen Burstin, MD, MPH, Director, Center for Primary
Palo Alto Medical Foundation Care Research, Agency for Healthcare Research and
Quality
Bonnie Kaplan, PhD, Yale Center for Medical Informatics
and President, Kaplan Associates
Discussion Group Leaders:
Case Studies and Guidelines Panelists: Gordon D. Brown, PhD, Health Management and
Informatics, University of Missouri
Marvin Packer, MD, Harvard Pilgrim Health Care
Richard Bankowitz, MD, MBA, University Health System
Tamara Stone, MBS, PhD, Assistant Professor of Health
Consortium
Management, University of Missouri
Harold Lehmann, MD, PhD, Director of Medical
Robert Greenes, MD, PhD, Director, Decision Sciences
Informatics Education, Johns Hopkins University School of
Group, Brigham and Women’s Hospital, Boston,
Medicine
Massachusetts