Measurement

Module Objectives

 By the end of this module, participants

will be able to:
 Explain the importance of data/measurement
 Define an indicator
 Design quality improvement indicators
 Link measurement and aim statement
 Describe a run chart
 Construct a run chart
 Interpret a run chart using the run chart rules
Large Group Discussions

 Reflection
questions
 What is data?
 Why are we
measuring?
 What is the use of
measurement?
 Time: 10 minutes
 What is data?

The Healthcare Data Guide, p26

 Why Measure?

 Assess progress on aim

 Learn from the impact of the change
 Assess whether the system as a whole is
improving
 Helps to make informed decision on what
to do next
 Improvement is about making
changes, it doesn’t come from
measurement
Institute for Healthcare Improvement
 Purposes of Data

 Data is
not for the purposes of
judgment (for deciding whether or
not to buy or to accept or reject) but

 For the purposes of learning

(such as from experiment, from others, or
from history)
 Displaying Data

 Learning comes from understanding

patterns in data
 Patterns are easier to recognize when the
data are plotted over time
 Displaying data on graph over time
 allows us to see the impact of our changes
on the system
 Maximizes the learning from data
 Visual Display of Data

Cycle Time (min.)

100
90
80

Cycle Time Results for 70

60
Unit 1
Units 1, 2 and 3 50
40
30
20 Change
Made
10
0

Apr

Aug
Mar

May
date

Jun

Nov
Feb
Jan

Sep

Oct

Dec
Jul
100

Cycle Time (min.)

90
80
80
70
70
60 Unit 2
70 50
40
60 30
Cycle Time (min.)

20
50 Change
Made
10
40 35 0

Apr

Aug
Jun
Mar
date

Feb

May

Nov
Jan

Jul

Oct
Sep

Dec
30 100
Cycle Time (min.)

90
20 80
70
10 60
50
Unit 3
0 40
Avg Before Avg After 30
20 Change
Change Change 10 Made

0
Apr

Aug
Jun
Mar
date

Feb

Nov
Jan

May

Jul

Oct
Sep

Dec
 Planning for Data Collection

Questions to ask when planning for data collection:

 What are my indicators?
 How do I collect the data?
 How frequent (i.e. daily, weekly, monthly, etc.) is
it collected?
 Who is responsible for the collection of the data?
 Is it routinely collected/documented?
 What is the data source?
 Do I have to develop any tool(s) to collect the
data?
 How do I want to present the data?
 Data Collection Tools

• Registers
• Reporting forms
• Checklist
• Observational
guide
• Questionnaire
• Chart audit
• More…
 An Indicator
 What is an indicator?
 According to OECD/DAC, an indicator is: "A
quantitative or qualitative factor or
variable that provides a simple and reliable
means to measure achievement, to reflect
changes connected to an intervention, or to
help assess the performance of a
development actor".
(OECD/DAC Glossary of Key Terms in Evaluation and Results Based Management,
2002)
An Indicator

 What is an
indicator?
 According to the
definition adopted by
USAID, an indicator
is: "A quantitative
or qualitative
variable that
provides reliable
means to measure
a particular
phenomenon or
attribute". (
 Components of an Indicator
Numerator
(data element)

number tested for HIV at first visit

Proportion of patients __________________________________
= …x100
HIV tested at first ANC number needing an HIV test at first
visit visit

For percentage……

Denominator
(data element)
Key Tips For Defining An Indicator
 In stating an indicator it should include
words like:
 Number of…
 Average…
 Proportion of…
 Percentage …
 Rate of…
 In stating an indicator it doe not include
words like:
 Improve
 Reduce
 Increase
Small Group Exercise Instructions

 Create groups of four to six participants

 Each group should be assigned with one QI
topic
 Task: State the indicator, numerator,
denominator, factor/ multiplier and data
source if possible
 Each group must be prepared to present
their work at the plenary session
 Time: 15 minutes + 60 minutes
 Design quality improvement
indicators
Small Group Exercise: Design a quality
improvement indicator for a QI team
working on the following:
 Reducing clients waiting time at OPD from 5
hours to 3 hours from February 2018 to July
2018 – Group 1
 Improving adherence to partograph from 40%
to 80% from February 2018 to July 2018 –
Group 2
 Reducing Emergency Caesarian response time
(time doctor was called and the time of
incision) from 2 hours to 45 minutes from
February 2018 to May 2018 – Group 3
 Design quality improvement
indicators
Small Group Exercise: Design a quality
improvement indicator for a QI team
working on the following:
 Reducing stock out for oxygen from 20% to 5%
from February 2018 to April 2018 – Group 4
 Improving adherence to malaria protocol from
40% to 90% from February 2018 to July 2018 –
Group 5
 Reducing average cervical dilatation of a
laboring pregnant woman on arrival at the
facility from 7cm to 5cm from February 2018 to
December 2018 – Group 6
 Linking measurement and aim
Statement

What are we trying to accomplish?

Indicators/
Measures
What change can we make that will result in
an improvement?
 Outcome
measure(s)
How will we know a change is an
A improvement?
P
 Process
Act Plan measure(s)
S D
Study Do
 Balancing
measure(s)
 It’s not Just Measurement..

“You
can’t
fatten a
cow by
weighing
 Types of Measures
 Outcome measure
 It measures if you are moving towards achieving your aim of
the project
 It is linked to “aim statement”

 Process measure
 It measures to what extent the interventions (changes) are
being implemented
 It is linked to “changes”

 Balancing measure
 it measure if the new intervention (change) has unintended
negative or positive consequences
 It assesses the whole system, including areas out side the
scope of the specific QI project
 It is linked with the “potential risks and assumptions”
 Large Group Exercise

Aim: I aim to reduce my weight from 75 kg to

65
kg from 1st January, 2018 to 31st March,
2018

Change: Go swimming twice a week for 1 hour

What is/are the outcome measure and process

measure(s)? What can be a balancing measure?

Time: 2 minutes
 Large Group Exercise

Aim: Hospital X aims to reduce TB treatment

defaulters from 37 to 10 percent
from January 2018 to July 2018
Change: Community care workers/volunteers to
follow up patients on TB treatment at
home when they are 5 days late in
returning to the hospital for renewal of
prescription
What is/are the outcome measure and process
measure(s)? What can be a balancing measure?

Time: 3 minutes
 Measures of Central Tendency

Mean
 Arithmetic average of data
Median
 Middle value of ordered data
 When n is an even number? How do we calculate the
median?
 When n is an odd number? How do we calculate the
median?
Mode
 Most frequently occurring value
Small Group Exercise Instructions
 Create groups of four to six participants
 Each group should be assigned one
question
 Task: Calculate (i) Mean,
(ii) Median and
(iii) Mode
 You will be given 10 minutes to work on
your assigned question
 Each group must be prepared to present
their work at the plenary session
 Time: 10 minutes + 20 minutes
 Measures of Central Tendency
Compute the following: Mean, Median and
Mode
Data set Data set Data set Data set
1 2 3 4
5 48 2 0
19 6 9 19
3 57 7 29
0 39 5 14
16 52 24 2
14 25 3 7
8 33 7 22
7 37 7 25
4 41 6 9
3 6 7
Group 1 Group 2 2
Group 3 Group 4
 Study Data over Time - Run Chart

 A graphical display
of data plotted in
some type of order
(data over time)
 Run Chart is also
called Trend Chart or
Time Series Chart
 A Run Chart is a Line
Graph with a
median line
 Uses of Run Charts in Improvement
Initiatives
1. Make process performance visible
Current Process Performance: Isolated Femur Fractures
1200
Minutes ED to OR per

1000

800
Patient

600
3. Determine if holding gain
400
Median 429
200 Holding the Gain: Isolated Femur Fractures
1200
0

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 1000
Sequential Patients

Minutes ED to OR per
800
v.1protocol
2. Determine if change resulted

Patient
600 v.2 3 4 5 Implement
in an improvement 400

Process Improvement: Isolated Femur Fractures 200

1200
Minutes ED to OR per

0
1000

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64
800 v.1protocol Sequential Patients
Patient

600

400
v.2 3 4 5
200

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64
Sequential Patients

The Healthcare Data Guide, p86

 Is this a Line Graph or a Run
chart?
% of babies who weigh less than 2.5kg at
20.0birth in Woreda X - July 2015 to May 2017

15.0
PERCENTAGE

10.0

5.0

0.0
5 5 5 6 6 6 6 6 6 7 7 7
'l 1 p'1 v'1 n'1 r'1 y'1 l'1 p'1 v'1 n'1 r'1 y'1
Ju Se No Ja Ma Ma Ju Se No Ja Ma Ma
 Is this a Line Graph or a Run
chart?

% of ANC Registrants (ANC1) at Reg-

istration per Expected Pregnancies
in Woreda Y-July 2015 to April 2017
100.0
80.0
PERCENTAGE

60.0
40.0
20.0
0.0
5 Se o 16 a a 6 Se o 17 a
'l 1 N
n ' M M 'l 1 N
n ' M
Ju Ja Ju Ja
 Constructing Run Chart

 Steps to construct a Run Chart

1) State the question that the run chart will answer
and obtain data necessary to answer this
questions
2) Develop the horizontal axis. This is usually is in a
time scale (days, weeks, months, quarters, years
etc.)
3) Develop the vertical axis
4) Plot the data points
5) Label the graph completely with a useful title
6) Calculate and place the median of the data on the
run chart
7) Add additional information to the chart
 Group Exercise Instructions

 Create groups of four to six participants

 Each group should be assigned with a
scenario
 Task: Construct a run chart using both flip
chart and MS Excel. The chart should be
well labelled.
 Each group must be prepared to present
their work at the plenary session
 Time: 20 minutes + 40 minutes
 Scenario 1-Waiting Time in the
Facility
Your facility collected the following data over
the past 15 weeks on waiting time (time of
entry and time of exit) of your patients. Use
the data below to construct a run chart using
both the flip chart and MS Excel.
Wk Wk Wk Wk Wk Wk Wk Wk Wk Wk Wk Wk Wk Wk Wk
Weeks 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Total waiting
time (minutes) 3350 4176 3260 4160 3110 2750 2628 2910 2860 2655 3050 2772 2920 4245 4470
Number of 10 12 10 13 10 10 9 10 10 9 10 9 10 15 15
patients
 Scenario 2- Adherence to Malaria
Protocol
Your facility collected data on adherence to
malaria protocol for the past 9 months. Use
the data below to construct a run chart
using both the flip chart and MS Excel.

Jan- Feb- Mar- Apr- May- Jun- Jul- Aug- Sep-

Month 17 17 17 17 17 17 17 17 17
Adherence to
Malaria Protocol 3 5 8 4 7 8 8 5 6
Number of folders
sampled 10 10 10 10 10 10 9 10 10
Scenario 3- Institutional Neonatal
Mortality Rate

Below the neonatal deaths and live births

from Woreda X. Use the data below to
construct a run chart using both the flip
chart and MS Excel.
Jan- Feb- Mar- Apr- May- Jun- Aug- Sep-
Month Jul-17
17 17 17 17 17 17 17 17
Number of
Neonatal 3 0 1 4 3 6 2 5 6
deaths
Number of
200 189 170 180 268 291 255 280 295
Live births
 Signals of Improvement

 Identify non-random signals in a Run Chart,

if one or more of the circumstances depicted
by the four Run Chart rules.
 Direction of the non-random signal in
combination with the indicator presented,
will determine whether this signal is an
improvement or not.
 Rules 1 and 3 are violations of random
patterns that are based on a probability of
less than 5% chance of occurring just by
chance with no change. The Healthcare Data Guide, p78
 4 Rules for Interpreting Run
Charts

A Shift: A Trend
6 or more 5 or more

Too many or An astronomical

too few runs data point

The run chart, Rocco J. Perla, Lloyd P. Provost and Sandy K. Murray, 2011
 Rule 1 - Shift

Rule 1
Rule 1 – Shift:
25
Six or more consecutive
20 points either all above or
all below the median.
Measure or Characteristic

15

10
Skip values on the median
5 and continue counting
0
points. Values on the
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 median DO NOT make or
break a shift.

The Healthcare Data Guide, p77-84

 Rule 2 - Trend

Rule 2 Rule 2 – Trend:

25
Five or more consecutive
20 points all going up or all
going down.
Measure or Characteristic

15

10 If the value of two or

5 more successive points
is the same, count all of
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 them as one point; like
values do not make or
break a trend
The Healthcare Data Guide, p77-84
 Rule 3 – Too Many or Too Few Runs

Rule 3 – Runs
Rule 3 A run is a series of points in a
10 data points not on median.
1 crossing +1= 2 Runs = Too few runs row on one side of the median
25
20 Step 1: Count the number of data
15 points that do not fall on the
median line
Measure or Characeristic

10
Step 2a: Count the number of times
5 the performance line cuts/crosses
0 the median line
1 2 3 4 5 6 7 8 9 10 Step 2b: Add one to the number
from step 2a
The Healthcare Data Guide, p77-84 Step 3: Make reference to
the reference table
 Reference Table – Rule 3
Total no. of data points that do not Lower limit for no. of runs (<this Upper limit for no. of runs (>this
fall on the median no. of runs is “too few” no. of runs is “too many”

10 3 9
11 3 10
12 3 11
13 4 11
14 4 12
15 5 12
16 5 13
17 5 13
18 6 14
19 6 15
20 6 16
… … …

Source: Swed, Frieda S. and Eisenhart, C. (1943) “Tables for Testing Randomness of Grouping in a
Sequence of Alternatives.” Annals of Mathematical Statistics. Vol. XIV, pp. 66-87, Tables II and III.
 Rule 3 – Too Many or Too Few Runs

Rule 3 – Runs
8 Rule 3
7 A run is a series of points in a
6 row on one side of the median
5 Median 3.66
4
Step 1: Count the number of data
3
points that do not fall on the
2
median line
1 20 data points not on median.
18 crossings +1= 19 Runs= Too many runs Step 2a: Count the number of times
0
J- F M A M J J A S O N D J- F M A M J J A S
the performance line cuts/crosses
03 04 the median line
Step 2b: Add one to the number from
step 2a
The Healthcare Data Guide, p77-84
Step 3: Make reference to
the reference table
 Reference Table – Rule 3
Total no. of data points that do not Lower limit for no. of runs (<this Upper limit for no. of runs (>this
fall on the median no. of runs is “too few” no. of runs is “too many”

10 3 9
11 3 10
12 3 11
13 4 11
14 4 12
15 5 12
16 5 13
17 5 13
18 6 14
19 6 15
20 6 16
… … …

Source: Swed, Frieda S. and Eisenhart, C. (1943) “Tables for Testing Randomness of Grouping in a
Sequence of Alternatives.” Annals of Mathematical Statistics. Vol. XIV, pp. 66-87, Tables II and III.
 Rule 4 – Astronomical Data Point

Rule 4
25 Rule 4 – Astronomical
20
Data Point:
15
For detecting unusually
Measurement or Characteristic

10 large or small numbers

5

Data that is Blatantly

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Obvious different value

The Healthcare Data Guide, p77-84

ANC Data from 6 Health Facilities in
Ethiopia

Percentage of pregnant women attending

ANC clinics tested for syphilis
Quality Improvement Project Work

Section III - Measurement

 Building on the above small group
exercise, section I (problem identification,
prioritization and aim statement on
module 4) and section II (developing
changes on module 5) exercise of your
quality improvement project, now design
of quality improvement indicators and
measures.
 Use the project workbook
 Time: 20 minutes
 Summary

 Obtaining data for improvement involves some

effort and resources
 Our goal in obtaining data for improvement is
to aim at usefulness, not perfection
 “Perfection is the enemy of the good.”
 The main difference between a line graph and a
run chart is a line graph does not have a
median while the run chart has a median line
plotted on the graph
 Remember:
 “Some is not a number, soon is not a time.”
Summary – Run Chart Rules

A Shift: A Trend
6 or more 5 or more

Too many or An astronomical

too few runs data point

The run chart, Rocco J. Perla, Lloyd P. Provost and Sandy K. Murray, 2011