EMR Implementation

Readiness Assessment and

Patient Satisfaction
Kasey Parker, Evaluation Lead and Report Writer
Catherine Campbell, Human Factors Lead
Anne-Marie Parent, Change Management Lead
Dr. Jim King, Physician and CMIO
 Objectives
During EMR implementation we evaluated:

 Readiness and technical

adoption

 Provider adoption, use and

satisfaction

 Patient/family satisfaction

No Conflict of Interest to declare

Thursday, July 9, 2015 2

 Children’s Hospital of Eastern Ontario

 167 bed tertiary care pediatric hospital

 Referral Base: ~ 2 million
o Pediatric Care to National Capital Region and Eastern Nunavut
o Regional trauma center and Level III NICU
o 194,000 outpatient visits to 63 specialty clinics
 Enterprise Epic implementation
o EpicCare Ambulatory
o Epic lab: Beaker
o Patient Access and Revenue Cycle
o Patient Health Portal

3
Thursday, July 9, 2015 3
  Enterprise suite

Thursday, July 9, 2015 4

 Phase 1: Ambulatory
Wave 4 &
Wave 1 Wave 2 Wave 3 Post Upgrade
(Oct 2013) (Apr 2014) (Oct 2014) (Apr-Oct 2015)
Infectious Disease C1 Nephrology Gastroenterology Neonatal Follow-Up
Pediatric Medicine Urology Child and Youth Protection
Neurology
Rheumatology
Urodynamics Complex Care
Neurosurgery
ENT
Adolescent Health Chronic Pain
Respirology
Discharge Planning
Endocrinology Orthopedics
Genetics Cardiology
Infectious Disease C10 Opthalmology
Audiology
Cardiovascular Surgery
Autism General Surgery
Physiotherapy
Occupational Therapy (Rehab)
Psychology
Metabolics Plastics

Mental Health Rehab Medicine (Rehab) Neuromuscular

Social Work

Clinical Nutrition MH Eating Disorder Speech/Language (Rehab) Spina Bifida

Increasing Complexity
*100% patients are registered and scheduled 
* 83% (33/40) clinics are ‘Live’ for order entry and clinical documentation
Thursday, July 9, 2015 5
 ‘s Benefits are being Realized

System    Providers     Patients
134 pt/proxy accessed their  BedBoard supporting patient 
record through MyChart flow

65,559 medication orders 1,059 unique users/mo 
~ 1‐2% rapid change/mo accessed 331,180 records 

> 1 million  procedure orders 
(e.g. lab, DI) > 1 million less lines of 
~ 1‐2% duplicates/mo transcription in 2014

Thursday, July 9, 2015

Up to date as of Apr, 2015 6
 Benefits Evaluation Framework
 Overarching question => “What 
are the benefits (clinical and/or 
operational) of implementing 
an EMR system in a pediatric 
out‐patient hospital based 
setting?”
 Multiple evidenced based 
frameworks and EMR best 
practices
 Aligned Epic and CHI’s key 
performance indicators to 
CHEO’s strategic objectives

Thursday, July 9, 2015 7

 Epic and CHI Performance Indicators by Net Domain (N=12)

Improved capacity to 
System,  information, 
Improved office/clinic  assess performance/ 
service quality and  Improved patient access
efficiency  conduct quality 
use/user satisfaction 
improvement
Percent result messages marked   Encounter cycle time: check‐in/out  
completed within 24 hours (hh:mm)
Percent visits with a printed after 
Impact on transcription lines  visit summary (AVS)
(n‐lines) Percent of charts returned the 
same day as the patient's 
Provider adoption, use and  appointment (WinRecs)*
Impact on transcription costs 
satisfaction~ 
(CAD currency)
(EHR  Readiness Assessment ) Percent of encounters closed same 
day  of patients appointment 
Impact on chart pulls (n‐pulled) (Epic)^
Patient Satisfaction~

Time to third available  Percent open closed encounters 
appointment (days) within 5 days

*Charts can be returned regardless if clinical documentation is complete
^Charts are returned/closed when: 1) diagnoses (or suspected) recorded; 2) charges dropped; and  3) progress note written (no restrictions)
~Qualitative metric collected via questionnaire

* Based on CHI BE Framework (Lau et al 2009 ‐‐ adapted from the DeLone and McLean information system success model )
Thursday, July 9, 2015 8
 Epic EHR Readiness Assessment: Overview of Methods
and Questionnaire
Section 1: Background (11 questions)
•e.g. role, department, education activities,
computer sophistication

Section 2: Likert Scale (57 questions)

•scale: strongly agree to strongly disagree (5-
point scale +1 n/a option)
•9 themes: e.g. leadership accountability,
training, Epic EHR usefulness and
friendliness

Section 3: Open Text (2 questions)

•opportunity to address concerns and benefits

*Run period: 3 weeks with up to 2 reminders
Thursday, July 9, 2015 9
 Epic EHR Readiness Assessment: Design Components
 Derived from questionnaires previously validated and applied in a physician 
based healthcare setting

 Three (3) main sources were utilized  to acquire:
 Contextual/technical acceptance1;
 Readiness assessment2; and
 Relational (technical/physician) factors3

 Pilot tested in a diverse group of project team 
members for face‐validity:
 Application Coordinators 
 Change Management Lead 
 EHR Research Team members
 Human Factors BSA
 Physician end user
1Morton et al 2008,  2CHI Change Implementation Success Factors: Risk and Readiness Assessment, 2012 and 3Davis et al 1989 and 

Chutter et al 2009 
Thursday, July 9, 2015 10
 Patient Satisfaction: Methods and Questionnaire
 Engaged and collaborated with 
Patient/Family Centred Care team

 Added  19‐question insert into the existing 
CHEO “Bear” patient satisfaction 
questionnaire (REB 13/96x)

 Collected pre (6 weeks) and post 
(3 months) Go‐Live for each Wave

 Run period = 5 weeks

 Sent questionnaire to ALL patients/families 
that completed an appointment 

* Questions adopted from Zurovac et al 2012 and Lelievre et al 2010 
Thursday, July 9, 2015 11
 Epic EHR Readiness Assessment: Flow Diagram

Thursday, July 9, 2015 12

 Epic EHR Readiness Assessment: Background Results
Gender • 85% female; 14% male; 1% missing value 

Age • 26% 30 ‐ 39 years; 28% 40 ‐ 49 years; 27% 50 ‐ 59 years

• 32% Health Professional / Trainee; 22% Nurse / Nurse 
Healthcare Role Practitioner; 12% Physician

Years of Service • 50% more than 15 years

Relationship to CHEO • 69% full time employee / full‐time hours at CHEO

Computer Sophistication • 66% general skills

*Wave 1 (n=96) ; Wave 2 (n=75) and Wave 3 (n=43)
Thursday, July 9, 2015 13
 Epic EHR Readiness Assessment: Key Findings
•98% of end users agreed that they understand the reasons for implementing the Epic 
Case for Change EHR, including improving patient safety and quality of care (2%)

•71% of end users agreed the vision for the Epic EHR creates understanding and 
Vision Clarity and Strength excitement about the changes in workflow and practice that need to be made (5%)

User Engagement and  •68% of end users agreed the communication about the project is clear and consistent, 

Involvement regardless of who is delivering the message (3%)

Percent (%) Agreement (all Waves N=214) Wave 1 (n=96) Wave 2 (n=75) Wave 3 (n=43)

Question Pre  Post %Change P‐value^ Pre Post    %Change    P‐value^    Pre   Post   %Change      P‐value^  
1. I understand the reasons for implementing the Epic EHR, including improving 
96.9 98 1.1 0.22 98.6 97.3 ‐1.3 0.68 92.9 97.6 4.7 0.071
patient safety and quality of care

6. The vision for the Epic EHR creates understanding and excitement about the 
72.3 64.9 ‐7.4 0.01* 75.3 68.5 ‐6.8 0.08 81 81 0 0.66
changes in workflow and practice that need to be made 

16. The communication about the project is clear and consistent, regardless of 
62.6 52.7 ‐9.9 0.01* 70.8 70.8 0 0.58 81.4 81.4 0 0.19
who is delivering the message 

^Marginal Homogeneity test; *p<0.05; **p<0.001
Thursday, July 9, 2015 14
 Epic EHR Readiness Assessment: Key Findings
Provider‐Patient  •End users did not feel that using the Epic EHR increased patient satisfaction with 
the quality of health care he/she receives (29%)
Relationship
•End users did not feel that the Epic EHR: improved the quality of care they 
EHR Usefulness delivered to their patients (28%), increased  productivity (16%) or improved 
performance (14%)

Percent (%) Agreement (all Waves N=214) Wave 1 (n=96) Wave 2 (n=75) Wave 3 (n=43)

Question Pre  Post %Change P‐value^ Pre Post    %Change    P‐value^    Pre   Post   %Change      P‐value^  

27. Using the Epic EHR has increased patient satisfaction with the quality of 
50 16.7 ‐33.3 <0.001** 40.6 15.6 ‐25 <0.001** 47.5 17.5 ‐30 0.001*
health care he/she receives

29. Using the Epic EHR for clinical documentation has improved the quality 
66.7 31.8 ‐34.9 <0.001** 56.1 33.3 ‐22.8 <0.001** 46.3 19.5 ‐26.8 0.001*
of care I deliver to my patients

35. Using the Epic EHR has increased my productivity  37.8 19.5 ‐18.3 <0.001** 44.6 27.7 ‐16.9 <0.001** 47.2 33.3 ‐13.9 0.002*

37. Using the Epic EHR has improved my performance  36.4 19.5 ‐16.9 <0.001** 36.9 30.8 ‐6.1 0.001* 31.6 13.2 ‐18.4 <0.001**

^Marginal Homogeneity test; *p<0.05; **p<0.001
Thursday, July 9, 2015 15
 Epic EHR Readiness Assessment: Key Findings
• 76% of end users agreed they received the training  they need 
Training to be able to understand and use the Epic EHR (5%)

EHR User Friendliness /  • 53%  of end users agreed it is easy for them to remember how 

Usability to perform tasks using the Epic EHR (15%)

Overall Attitude about the  • 74% of end users agreed that overall, their attitude about Epic 
Epic EHR EHR use is positive (7%)

Percent (%) Agreement (all Waves N=214) Wave 1 (n=96) Wave 2 (n=75) Wave 3 (n=43)

Question Pre  Post %Change P‐value^ Pre Post    %Change    P‐value^    Pre   Post   %Change      P‐value^  

23.  I have received the training that I need to be able to understand and 
58.6 60.9 2.3 0.27 76.1 76.1 0 0.88 76.9 89.7 12.8 0.24
use the Epic EHR 

48. It is easy for me to remember how to perform tasks using the Epic EHR  36.4 50.6 14.2 0.04* 41 57.4 16.4 0.12 37.1 51.4 14.3 0.041*

57. Overall, my attitude about Epic EHR use is positive 79.8 65.2 ‐14.6 <0.001 81.7 77.5 ‐4.2 0.27 81 78.6 ‐2.4 0.19

^Marginal Homogeneity test; *p<0.05; **p<0.001
Thursday, July 9, 2015 16
 Patient Satisfaction: Key Findings
Survey sent to 16,998 with 2,253 responses (13%) 
 93% of patients/families are familiar with computers 

57% of patients/families have noticed their Providers 
using a computer during their visit  to enters notes, 
share information and to look up test results (20%).
Patient 
satisfaction  97% of patients/families did not notice a change in the 
attention they were receiving from their Provider 
with their  (1%).  
CHEO visit
95% of patients/families did not find a difference 
in the ease of talking to their Provider (‐).

Thursday, July 9, 2015 17

 Conclusion
 Benefits Evaluation (BE) ongoing

 Readiness and technical adoption
o Providers were ready and adapted well
 strong Change Management and BE
 trend improving over time with subsequent waves

 Provider adoption, use and satisfaction
o Providers felt EMR impacted quality of care , performance and productivity
 trend not improving over time with subsequent waves
 but still supported the implementation 
 patient
 Patient/family satisfaction
o Did not notice a change in the attention or ease of talking  to their Provider

Thursday, July 9, 2015 18

 Patient versus Provider Perspective

Patients Providers

Results are
expected at this
early stage. We
anticipate that it
will take 6-12
months post Go-
Live to see a true
reflection of user
experience.

*Anticipate that the ‘balance board’ will shift in a positive direction 
Thursday, July 9, 2015 19
 QUESTIONS

Any questions, please contact Kasey Parker [email protected]

Thursday, July 9, 2015 20