Evaluating a Complex Educational Intervention in a

Complex Setting:
Assessing the Impact of a Morbidity, Mortality, and
Improvement Conference on Patient Care
DOM-ACE Meeting | May 14, 2015

Lori Bakken, Curtis Olson, Jonathan Ross, Mary Turco, Lisa Jackson, Wendy Murphy,
Daphne Ellis, Kathryn Kirkland
Context
• Phase I evaluation of MM&I completed
• Part of the Dartmouth Aligning Education with Quality
(ae4Q/AAMC) initiative
• Sample evaluation questions (8 total):
– What types of cases have been discussed and why were they selected?
– What values do the MMIC discussions espouse and how do participants’
perceive them?
– To what extent, or in what ways, is critical thinking engaged as part of
the clinical reasoning process?
– In what ways do participants’ knowledge or practice change in terms of
treating and managing specific diseases?
• In what ways and how is MM&I contributing to improving patient
care?

2
QI and Interprofessional Education

Quality ? IPEMM&I
Improvement

3
Educationally salient characteristics
• Resident-led, weekly case conference
• Interprofessional, multigenerational
• Addresses clinical, interpersonal, systems issues
• Highly participatory, longitudinal
• Addresses how to respond to uncertain
situations
• Embodies respectful communication and
professional accountability
• “Safe” environment; legally protected

4
Complexity in Cases
• Cases often:
– Are complex, difficult to diagnose; experts often
disagree
– Involve rapid changes in patient’s condition
– Involve patient/family communication around
end-of-life, goals of care
• All provide opportunity for JIT teaching, critical
reflection on clinical reasoning/team functioning

5
Other aspects of complexity
• Hospital/Organization
– Multiple, interdependent providers
– Self-forming teams
• Educational Intervention
– Topic/presenter/audience variability
– Adaptive, emergent educational approach

6
Methodology

7
MM&I Systems Logic Model

8
Developmental evaluation
• Initial exploration of linkages between MM&I and
improved care
– Is there evidence that MM&I is producing the
intended outcomes through the four pathways?
– Are there other (ie, unanticipated) pathways that link
MM&I with improved care?

Patton MQ. Developmental evaluation: applying complexity concepts to enhance

innovation and use. New York, NY: Guilford Press; 2011.
9
 Data Collection

Interviews Focus Guided Values Structured

Groups Observation Survey Audit

Participants:
• Residents
• Specialists
• Community Physicians   
• Med Students
• Allied HCPs

Live MM&I events 

Video recordings of
past MM&Is 
Archival documents 

10
Findings

11
 Pathway 1: Individual changes in clinical practice *

• Direct application of new/reactivated medical

knowledge and clinical approaches

*Outcomes reported in this presentation are illustrative, not comprehensive.

12
• I think the most important stuff that I get out of
[MM&I] is hearing the “why” and the “how” questions
thought through by somebody who’s in a specialty
other than mine. Honestly, I can tell you that I do feel
myself channeling those people in my head when I’m
caring for patients. I will hear a conversation in that
conference, a particular cardiologist talking about
how he thought through diuresis in somebody with
tubular dysfunction, as I’m seeing somebody with that
condition. –General Attendee (non-resident)

13
Pathway 1: Individual changes in clinical practice *

• Direct application of new/reactivated medical

knowledge/clinical approaches
• Increased attention to patient and family goals
and concerns

*Outcomes reported in this presentation are illustrative, not comprehensive.

14
• [Sometimes family] members are on completely
different wavelengths about what they expect. For me,
it was important to learn how to make that experience
for them as comfortable [as possible]… because that’s
the experience that they have to live with for their rest
of their lives. It’s so important because their loved one is
going to die. . . And I think that sometimes in this new
world of trying to push people in and out of the
hospitals, we forget that the family’s experience is just
as important as our care for the patient. –Resident-
presenter

15
 Pathway 1: Individual changes in clinical practice *

• Direct application of new/reactivated medical

knowledge/clinical approaches
• Increased attention to patient and family goals
and concerns
• Increased awareness and use of hospital services

*Outcomes reported in this presentation are illustrative, not comprehensive.

16
• I would say it’s probably heightened my decision to
get palliative care involved earlier. [MM&I has]
highlighted how [palliative care is] not necessarily
hospice care; it’s a team that can help patients
make better decisions for what they consider
important. And I definitely think of that earlier –
including for people who are young, or who
hopefully aren’t going to die but just have an awful
lot on their plate. –General Attendee (non-
resident)

17
QI Pathways (2-4)
• Very limited evidence to show these were
active
• However, there was evidence of a 5th pathway

18
MM&I Systems Logic Model

A 5th Pathway:
• Professionalism
• Affective change
• Teaching

19
Professionalism Outcomes
• Acquiring greater humility by being confronted
with one’s own limitations

20
• There’s . . . being confronted by what you don’t know.
And so that shock, I think, it takes physicians off their
pedestal and we say, “Oh, that’s a person and that
person had a future that perhaps wasn’t in my crystal
ball.” That is a nice message of humility; we have had
that a couple of times. There was a young man with
HIV who, for all intents and purposes, was written off
as dead [during the] conference [and] who also
attended the conference. At the end, [he was] looking
like a million bucks! –General Attendee (non-resident)

21
Professionalism Outcomes
• Acquiring greater humility by being confronted
with one’s own limitations
• Grasping the value of reflection on practice

22
• Another component is reflection for the
improvement piece. Knowing… that decisions
that are made aren’t final, in a sense. They
happen, but it is revisited. That we think about
the way we are treating patients. It’s not just
[that] the patient’s discharged, and that’s that.
But there is a process for it, and that’s part of
being a good provider. Going forward and
knowing that reflection is required. –
Medical Student

23
Professionalism Outcomes
• Acquiring greater humility by being confronted
with one’s own limitations
• Grasping the value of reflection on practice
• Developing a greater sense of accountability

24
• You’re doing it on a stage… Other people are
also looking at you and saying, “Why did you
do that?,” and, “Why didn’t you choose this
imaging versus that?” You have to defend your
choices, to a certain extent. But I think that’s a
good process… because you will always think
in the future that, if I had to defend this
decision, what would I say? I think it’s a self-
bettering process. –Medical Student

25
Affective Outcomes
• Obtaining resolution and a greater sense of
closure

26
They really do kind of argue both sides, when
you’re in [MM&I]. That was really, really interesting
and helpful to me, and kind of gave me some
emotional closure on the patient who I had known .
. .. It was nice to have that emotional and medical
clinical closure on the case in a room full of all the
attendings that we respect, as well as our
colleagues.
Resident presenter

27
Teaching Outcomes
• Having cases available as teaching tools
• Having shared reference points to refer back to
• Identifying and addressing faculty and trainee
educational needs

28
At one [session] one of the doctors said, “Well, I called the
STEMI alert.” He was a hospitalist.
It was like, “Just so you know, you don’t call a STEMI alert;
you call the cardiology fellows, stat, and a cardiology fellow
will call the STEMI alert because… this is how the system
works.”
“Well, I just thought the cardiologist would come . . .”
So that gave me the idea . . . that we need to educate the
housestaff, not just the cardiology fellows. So that was done,
and we haven’t had one since.
–Nurse

29
Limitations
• Subject to bias from recall, selection, attribution
• Does not show how frequent or widespread
attributable outcomes are
• Results are only suggestive

30
Implications
• Results such as these cannot be understood
solely in terms of the quality improvement
framework
• MM&I may be making an important and
complementary contribution to quality of care
• How to conceptualize the contribution?

31
Improving quality of care
Quality techniques such as Six Sigma:
“. . . act in their own characteristic ways to
improve quality, and it is important to accept
that in linear environments (systems they were
designed for) they do a very good job of it . . .”

Chapman W, Hutchinson C, Bialek D. Medical quality systems: the elusive goal of quality in complex systems. Wes Chapman’s Blog. 2011.
http://www.mwestonchapman.com/medical-quality-systems-the-elusive-goal-of-quality-in-complex-medical-systems/. Accessed December 5, 2014.

32
However,
“. . . none of these techniques can
independently succeed in improving the quality
of healthcare delivery because . . . healthcare is
a complex, non-linear system, fundamentally
different from the linear processes from which
the underlying quality concepts were derived.”*

*Chapman W, Hutchinson C, Bialek D. Medical quality systems: the elusive goal of quality in complex systems. Wes Chapman’s Blog. 2011.
http://www.mwestonchapman.com/medical-quality-systems-the-elusive-goal-of-quality-in-complex-medical-systems/. Accessed December 5, 2014.

33
The impact of uncertainty
Lower undertainty

-Routine, standardized More emphasis on

work processes -Provider relationships,
-Are amenable to QI

Higher uncertainty
communication
approaches -How providers make
sense of what is
happening, solve
problems, improvise,
and learn with/from
each other

Leykum LK, Lanham HJ, Pugh JA, et al. Manifestations and implications of uncertainty for improving healthcare systems: an analysis of
observational and interventional studies grounded in complexity science. Implementation Science. 2014;9(1):165.

34
IPEMM&I a complement to QI?

y Improved
Quality of
int
rta

Care
e
Unc
gh

etc
Hi

QI
E,

P
CM

roj
ects
PI

Low
uncertainty

“Education”,
“Training”, Quality
Collaboration
what
what is
is Improvement
“taught”
“caught”

Developed from: Leykum LK, Lanham HJ, Pugh JA, et al. Manifestations and implications of uncertainty for improving healthcare systems: an analysis of
observational and interventional studies grounded in complexity science. Implementation Science. 2014;9(1):165.

35
Next steps
• Dissemination
– Several presentations to date
– Manuscript in preparation for Acad Med
– Workshop at the AMEE/GAME meeting in
Glasgow, Sept 2015
• Phase II underway to get a more micro-view of
impact and catalogue facilitator behaviors

36
Contact
Curtis Olson, PhD
[email protected] | (608) 335-3773

Jonathan Ross, MD
[email protected] | (603) 653-9571

Mary Turco, EdD

[email protected] | (603) 653-6608

Lisa Jackson, MPH

[email protected] | (603) 653-6631

37