Observer Agreement for Respiratory Signs and Oximetry in

Infants Hospitalized with Lower Respiratory Infections1- 3

ELAINE E. L. WANG, RUTH A. MILNER, L1SSETTE NAVAS, and HELEN MAJ

Introduction
T he evaluation of pulmonary findings SUMMARY Todetermine observer agreement for a clinical score and oximetry in lower respiratory
infection in children less than 2 yr of age, a convenience sample of 56 infants hospitalized with
is an important component of physical bronchiolitis or pneumonia was assessed Independently by two observers. A total of 12 infants had
diagnosis. There has been little research chronic lung disease of prematurity or congenital heart disease. Infants In whom oxygen supplemen-
into the agreement among independent tation could not be discontinued for at least 5 min were excluded. A severity score was assigned
observers for the presence or absence of for each of four categories (respiratory rate, retractions, wheeze, and general appearance). A total
findings or on their severity when pres- for each patient was obtained by summing the score for each category. Oxygen saturation was mea-
ent. Previous publications on the subject sured using a Nellcor oximeter. Agreement beyond chance was measured using the kappa statistic.
included patients with airways obstruc- The relationship between observers for total score and oXimetry and the mean total score and mean
tion (1), emphysema (2), and various re- oximetry value for each patient was expressed as a Pearson correlation coefficient. A total of 56
spiratory disorders (3). They suggest that infants and children were studied: 2 had pneumonia, 11 had an exacerbation of pUlmonary signs
and symptoms with their underlying cardiac or pulmonary disease, and 43 had bronchiolitis. Kappa
agreement for these signs is poor, usually
was 0.48 for general assessment, 0.38 for respiratory rate, 0.31 for wheeze, and 0.25 for retractions.
falling in the range of less than halfway All values were statistically significantly greater than 0 at p < 0.01. Correlations for total score and
between agreement expected by chance for oximetry were 0.68 and 0.88, respectively. The median difference between oximetry readings
alone and maximal agreement possible. was 1. The correlation coefficient between total score and oXimetry was -0.04. The limited agree-
More recent studies have found observer ment for clinical signs makes comparison of patient illness severity between studies difficult. Fur-
agreement on physical signs to be of the thermore, oximetry should be performed on all patients with lower respiratory distress to determine
same order, although agreement on di- oxygen needs because there is a poor correlation between clinical findings and oximetry.
agnosis was higher (4,5). All these studies AM REV RESPIR DIS 1992; 145:106-109

were conducted only in adults. All assess-

ments referred only to a dichotomous
presence or absence of a finding, but did
not address the severitywhen the sign was hypothermia, oximetry is thought to be tion was a convenience sample obtained dur-
present. a reliable indicator of arterial oxygen ing weekdays when two observers were avail-
Respiratory scores incorporating some saturation (14). However, measurements able to examine them. They were assessed af-
of these signs have been developed in as- may be difficult to obtain in infants who ter parental consent was obtained without
prior review of the patient chart by either as-
sessing illness severity and response to are excessively active, irritable, or un-
sessor. In some cases, however, the admitting
therapy in infants with bronchiolitis cooperative. There is no published infor- diagnosis was known to the assessors. No pa-
(6-11). In addition, scores of physical mation or examiner agreement for ox- tients were assessed on more than one occa-
findings on respiratory examination have imetry findings. sion. The study was approved by the Human
been used as part of an algorithm for the The purpose of this study was to de- Subjects Review Committee at the Hospital
management of lower respiratory illness termine the interobserver agreement of for Sick Children.
in children (12). However, these studies four commonly used clinical signs as well
have been criticized for the lack of infor- as oximetry in the assessment of infants Observers
mation on observer agreement and with bronchiolitis and pneumonia. All patients were assessed by a pediatric in-
reproducibility, given the subjective na- fectious disease consultant (EW). The other
ture of the score (13). Methods observer for some patients was a research
Oximetry is a noninvasive measure of Patient Population
oxygen saturation (14-18). Its ease of per- Patients who were admitted to Hospital for
formance has led to its widespread use Sick Children with a diagnosis of bronchioli- (Received in original form April 4, 1991 and in
in intensive care, including neonatal units tis or pneumonia who were less than 2 yr of revised form June 26, 1991)
(15-18), during anesthesia (19), and in the age were eligible for study. Patients requiring
management of a variety of respiratory 35070 or more of inspired oxygenby head hood I From the Division of Infectious Diseases and

disorders (20). A recent study has sug- or its equivalent wereexcludedbecause of con- Clinical Epidemiology Unit, Department of Pedi-
cerns that they may be unable to tolerate lack atrics, the Hospital for Sick Children, Toronto, On-
gested that pulse oximetry should be used tario, Canada.
of oxygen supplementation for a period of
in addition to the available clinical in- 2 Supported in part by Grant No. 02183 from the
5 min, the minimum period required for per-
formation in deciding on hospitalization forming oximetry. Patients with chronic lung Ontario Ministry of Health.
3 Correspondence and requests for reprints
of children with wheezing-associated re- disease of prematurity or underlying cardiac should be addressed to Elaine E. L. Wang, M.D.,
spiratory illness (21). Aside from patients disease were included since they make up a ClinicalEpidemiology Unit, Hospital for Sick Chil-
who have poor peripheral perfusion or significant proportion of patients admitted dren, 555 University Avenue, Toronto, Ontario,
hypotension or patients with anemia or with lower respiratory disease. The popula- Canada M5G IX8.
106
OBSERVER AGREEMENT FOR RESPIRATORY SIGNS AND OXIMETRY IN INFANTS 107

TABLE 1 The Pearson correlation between the

DEFINITION OF CATEGORIES FOR CLINICAL ASSESSMENT two observer pairs for the overall clini-
Score cal assessment, or total clinical score, was
0.68. The scattergram for this relation-
o 2 3 ship is displayed in figure 1. The correla-
Respiratory rate < 30 31-45 46-60 > 60 tion for oximetry was 0.88 (figure 2). The
Wheezing None Terminal expiration or Entire expiration or Inspiration and expiration absolute difference between the two ox-
only with stethoscope audible on expiration without stethoscope imetry readings was most commonly 0,
without stethoscope
Retractions None Intercostal only Tracheosternal Severe with nasal flaring
the median difference was 1, and the
General condition Normal Irritable, lethargic, poor maximum difference was 6. There was
feeding a poor correlation between overall clini-
cal assessment and oxygen saturation as
determined by oximetry (r value - 0.04)
(figure 3).
nurse (HM) and for the others was a pedi- Statistical Analysis
atric infectious disease fellow (LN). Before Observed agreement includes two compo- Discussion
the start of the study, both 0 bservers reviewed nents: change agreement and agreement be- Poor to moderate interobserver agree-
the findings in a patient with bronchiolitis. yond chance. The kappa statistic is an expres- ment has been observed in studies of re-
This review was conducted with two differ- sion of the amount of agreement beyond spiratory findings conducted in adults
ent patients for the two observer pairs. Each chance (22). Thus, the degree of observer (1-5). Of the signs examined, only club-
patient was assessed independently, and ob- agreement for the clinical observations was
servations were recorded on a printed form
bing, tachypnea, and decreased percus-
represented using a kappa statistic. This cal- sion note demonstrated agreement great-
identified by the patient initials and medical culation took into account the combination
record number. The form for each patient was of results from different observer pairs (22).
er than 50% beyond chance (3). Since
kept separate so each observer would not be A scattergram and Pearson correlation were clinical scores used in studying bronchio-
influenced by the other's observations. The used to compare the overall clinical score and litis and croup are based on an assess-
paired observations were performed within pulse oximetry recorded independently by two ment of severity of various clinical find-
20 min of each other. To avoid an order ef- observers. A Pearson correlation coefficient ings (6-11), examination of observer
fect, the sequence of evaluation by the ob- was also used to express the relationship be- agreement for such assessments in chil-
serverswas haphazard: typically, one observer tween the averaged values for the total clini- dren has been suggested (12). In this
examined one patient while the other observer cal score and oximetry. study, moderate agreement was achieved
examined another. They would then switch
patients.
between two observers independently ex-
Results amining a group of infants with lower
Measurement A total of 56 patients were enrolled in respiratory tract infections for respiratory
Wewishedto examineobserver agreement un- the study during two separate 3-month rate and general appearance. However,
der different clinical conditions. Thus, pa- periods, one in 1988 and one in 1990: 2 there was poor agreement on the degree
tients were assessed at different periods after patients had pneumonia (mean age 49.3 of retractions and wheezing. The poorer
feeding or bronchodilator therapy. However, wk) and 43 patients had bronchiolitis agreement in the two latter findings may
no patients were examined within an hour of (mean age 27.4 wk). In addition, 9 pa- be because judging their severity is more
bronchodilator therapy to avoid a potential tients with bronchopulmonary dysplasia subjective despite the provision of defi-
dynamic state in which improvement or de- (mean age 30.3 wk) and 2 patients with nitions. The correlation for the overall
terioration in pulmonary function could oc- congenital heart disease(mean age 17wk)
cur. The patients had been hospitalized for
score was 0.68. However, this is a mea-
varying durations at the time of the assess-
who had wheezing, pulmonary deterio- sure of association between the two vari-
ments. All measurements were taken with the ration, and fever were included. ables, not agreement.
child in an awake, noncrying state. Each pa- The agreement beyond chance for all Gjorup and coworkers did not observe
tient was assessed according to four signs: (1) four clinical signs was below 500/0. Al- an improvement in agreement when the
respiratory rate, (2) presence of wheezing, (3) though the kappa values are low, they are history and working diagnosis was known
presence of retractions, and (4) general ap- statistically significantly different from to observers compared with the agree-
pearance. Each finding was categorized ac- no agreement beyond chance. The agree- ment of observers to whom this infor-
cording to the definitions provided to the as- ment for assessment of general condition mation was not available (4). However,
sessor (table 1). Respiratory rate was deter- (kappa = 0.48) and respiratory rate (kap- others have observed that although there
mined by counting respirations for 30 s. A pa = 0.38) was greater than that for as- may be poor agreement for a single clin-
total score was calculated from the sum of
the individual scores in the four categories.
sessment of wheeze (kappa = 0.31) and ical assessment, there is much better
Oxygen saturation was determined for each retractions (kappa = 0.25) (table 2). agreement on assessment of improve-
patient from a 5-min recording using a Nell
cor pulse oximeter (Medilogic, Rexdale, On-
tario, Canada). In all patients, this recording TABLE 2
was performed in room air. Oxygen satura- OBSERVER AGREEMENT FOR FOUR CLINICAL SIGNS
tion was recorded at the same time as patients
Standard Level of
werebeing observed for the clinical score. The Category Mean Kappa Deviation Kappa Significance
oxygen saturation was noted as the value at
which the recording had been stable for at General condition 0.48 0.13 < 0.001
least 2 min. The oximetry recording was tak- Respiratory rate 0.38 0.09 < 0.001
ing place as the patient was being observed Wheeze 0.31 0.08 < 0.001
for clinical signs. Retractions 0.25 0.09 < 0.01
108 WANG, MILNER, NAVAS, AND MAJ

12 100
* x xx x x2 *
2
Xl
x
x
x

• 98
~x ** xx ~2 x x x
x
10 96
x H1 x
~
tu • ~ 94 x x~x x
£ 8 • • • v
.~ 92
x x x
E
"~ • • • .2 • Fig. 1. Scattergram of the overall clin- e
~
;;
90
x

g6 • ~ .3 • cal assessments, EW versus HM and

LN. The value indicates the number of
P-
e 88 x
«l
• .3 .2 • .2 paired observations if the point repre- ~ 86
] 4 .2 .2 .2 sents more than one paired observation.
84
<3 • .2 • .2 • x
2
• .3 82

.3 • • • 4 6 8 10 12
0 Mean total clinicalscore
0 2 4 6 8 10 12
Fig. 3. Mean total clinical score versus mean pulse ox-
Overall assessment by HM or LN
imetry. The value is the number of paired results if point
represents more than one pair of observations.

100 • •3.2
• •5.
98 • • 2.2.2.
• • • 2.3
ing this observation to patients with very
~
'"
£
96

94 .2. .• •
.2 • •2
.3 e4 • •

•
low oxygenation. The exclusion of pa-
tients with low oxygenation caused the
clustering of oxygen saturation measure-
"
~
92 Fig. 2. Scattergram of pulse oximetry
~ 90 • • results, EWversus HM and LN. The val- ments above 880/0. This reduces the pos-
E • sibility of correlations to be observed.
j;' 88
• ue indicates the number of paired ob-
servations if the point represents more The lack of correlation between blood
E 86
.;; than one paired observation.
0
gas measurements and clinical assessment
84 in lower respiratory tract infections, how-
82
• ever, was previously reported by Simp-
son and Flenley (26). In their study of
80
80 82 84 86 88 90 92 94 96 98 100 32 children aged less than 3 yr hospital-
Oximetry measured by HM or LN ized with severe lower respiratory illness,
level of consciousness and other clinical
signs did not predict the level of oxygen-
ation. Of the clinical findings, only pres-
ment or deterioration when patients are curricula. Ensuring that respiratory rate ence of cyanosis was able to identify pa-
followed longitudinally (23, 24). The in- is counted for a full 60 s or having video- tients with oxygen saturation below 85%.
vestigators in Rochester used a global tapes of patients with differing severities Similarly, Hall and coworkers observed
clinical analog scale to assess overall clin- of clinical findings are examples of pos- no association between clinical findings
ical status rather than a clinical scoring sible measures. and measurements of arterial oxygen
system because the former was thought Pulse oximetry has been used in a va- saturation obtained with an ear oximeter
to be more reliable and easier to ad- riety of conditions in pediatrics (14). in patients with lower respiratory tract
minister (24, 25). Since we wished to de- Aside from factors relating to patient infection (36). This is not surprising if
termine the accuracy of assessments at changes, such as hypothermia and chang- one considers that the clinical score may
a single point, not as measures of change es in peripheral circulation (14), fluores- represent ventilatory effort, but this may
over time, we did not assess this aspect cent lighting has also been shown to not be associated with hypoxia, particu-
in the study. decrease the accuracy of oximetry (26). larly in patients with bronchiolitis who
It is likely that clinical scores for mea- The short duration between assessments have obstructive lung disease. It is neces-
suring respiratory illness severity are makes it unlikely that patient changes or sary to consider both clinical score and
widely used in clinical practice given that lighting changes contributed to the vari- oximetry separately for sample size cal-
similar scoring systems are used as out- ation. The differences in the observations culations in the planning of clinical trials.
come measures in numerous prognostic are small and of questionable clinical sig- The lack of correlation between clinical
studies and clinical trials (13, 24, 28-35). nificance. They reflect the variation that assessment and oxygenation emphasizes
The poor agreement for these clinical occurs with instantaneous monitoring of the importance of measuring oxygenation
findings limits the ability to compare pa- a dynamic steady state rather than a static rather than basing decisions on oxygen
tients across centers if one must base this equilibrium. The small differences in ox- supplementation on clinical evaluation.
strictly on clinical findings. Their poor imetry results in this study confirm the
agreement must be considered when reliability of this measurement.
References
deciding whether they should be includ- A poor correlation was observed be-
ed in courses of physical diagnosis. Ad- tween oximetry and clinical score in this 1. Godfrey S, Edwards RHT, Campbell ElM, Ar-
mitage P, Oppenheimer EA. Repeatability of phys-
ditionally, practical measures to reduce study. That the patients who were sick- ical signs in airways obstruction. Thorax 1969;
observer disagreement should be stud- est were excluded because of oxygen re- 24:4-9.
ied and incorporated in medical school quirement prevents one from generaliz- 2. Fletcher eM. The clinical diagnosis of pulmo-
OBSERVER AGREEMENT FOR RESPIRATORY SIGNS AND OXIMETRY IN INFANTS 109

nary emphysema-an experimental study. Proc R rin: a case of premature adjudication. 1 Pediatr orescent light interferes with pulse oximetry. 1 Clin
Soc Med 1952; 45:577-84. 1988; 112:154-8. Monit 1989; 5:135-6.
3. Smyllie HC, Blendis LM, Armitage P. Observ- 14. Bowes WA, Corke BC, Hulka 1. Pulse oxime- 27. Simpson H, Flenley DC. Arterial blood-gas
er disagreement in physical signs of the respiratory try: a review of the theory, accuracy, and clinical tensions and pH in acute lower-respiratory-tract in-
system. Lancet 1965; 2:412-3. applications. Obstet Gynecol 1989; 74:541-6. fections in infancy and childhood. Lancet 1967;
4. Gjorup T, BuggePM, lensen AM. Interobserver 15. Durand M, Ramanathan R. Pulse oximetry 1:7-12.
variation in assessment of respiratory signs. Physi- for continuous oxygen monitoring in sick newborn 28. Bierman CW, Pierson WE. The pharmaco-
cians' guesses as to interobserver variation. Acta infants. 1 Pediatr 1986; 109:1052-6. logic management of status asthmaticus in chil-
Med Scand 1984; 216(1):61-6. 16. Fanconi S, Doherty P, Edmonds lF, Barker dren. Pediatrics 1974; 54:245.
5. Spiteri MA, Cook DG, Clarke SW. Reliability GA, Bohn Dl. Pulse oximetry in pediatric inten- 29. Campbell H, Forgie1M, Lloyd-EvansN, Byass
of eliciting physical signs in examination of the sive care: comparison with measured saturations P, O'NeiII KP, Greenwood BM. Trial of co-trim ox-
chest. Lancet 1988; 1:873-5. and transcutaneous oxygentension. 1 Pediatr 1985; azole versus procaine peniciIIin with ampiciIIin in
6. Conrad DA, Christenson lC, Waner Jl., Marks 107:362-6. treatment of community-acquired pneumonia in
MI. Aerosolized ribavirin treatment of respiratory 17. lennis MS, Peabody lL. Pulse oximetry: an young Gambian children. Lancet 1988; 2:1182-4.
syncytial virus infection in infants hospitalized dur- alternative method for the assessment of oxygena- 30. Conrad DA, Christenson lC, Waner lL, Marks
ing an epidemic. Pediatr Infect Dis 1 1987;6:152-8. tion in newborn infants. Pediatrics 1987;79:524-8. MI. Aerosolized ribavirin treatment ofrespiratory
7. Rodriguez Wl, Kim HW, Brandt CD, et af. 18. Ramanathan R, Durand M, Larrazabal C. syncytial virus infection in infants hospitalized dur-
Aerosolized ribavirin in the treatment of patients Pulse oximetry in very low birth weight infants with ing an epidemic. Pediatr Infect Dis 1 1987;6:152-8.
with respiratory syncytialvirus disease. Pediatr 1nfec acute and chronic lung disease. Pediatrics 1987; 31. Groothius lR, Woodin KA, Katz R, et af. Early
Dis 1 1987; 6:159-63. 79:612-7. ribavirin treatment of respiratory syncytial viral in-
8. Taber LH, Knight V, Gilbert BE, et al. Ribavi- 19. Cote Cl, Goldstein EA, Cote MA, Hoaglin fection in high-risk children. 1 Pediatr 1990; 117:
rin aerosol treatment of bronchiolitis associated DC, Ryan lE A single-blind study of pulse oxime- 792-8.
with respiratory syncytial virus infection in infants. try in children. Anesthesiology 1988; 68:184-8. 32. Keeton GR, Kehoe B, Phillips SW, Daya H.
Pediatrics 1983; 72:613-8. 20. Mulholland EK, OIinsky A, Shann FA. Clini- Ceftazidime and cefamandole in the treatment of
9. Dabbous lA, Tkachyk rs, Stamm Sl. A dou- cal findings and severityof acute bronchiolitis. Lan- pneumonia. 1 Antimicrob Chemother 1983; 12
ble blind study on the effects of corticosteroids in cet 1990; 335:1259-61. (Suppl A:27-30).
the treatment of bronchiolitis. Pediatrics 1966; 21. Rosen LM, Yamamoto LG, Wiebe RA. Pulse 33. Rodriguez Wl, Kim HW, Brandt CD, et al.
37:477-84. oximetry to identify a high-risk group of children Aerosolized ribavirin in the treatment of patients
10. Maayan C, Itzhaki T, Bar-Yishay E, Gross S, with wheezing. Am 1 Emerg Med 1989; 7:567-70. with respiratory syncytial virus disease. Pediatr In-
Tal A, Godfrey S. The functional response of in- 22. Fleiss lL. Statistical methods for rates and fect Dis 1 1987; 6:159-63.
fants with persistent wheezing to nebulized beclo- proportions. 2nd ed. NewYork:John Wiley & Sons, 34. Shaw KN, Bell LM, Sherman NH. Outpatient
methasone dipropionate. Pediatr Pulmonol 1986; 1981. assessment of infants with bronchiolitis. Am 1 Dis
2:9-14. 23. Guyatt G. Measuring health status in chronic Child 1991; 145:151-5.
11. Tal A, Bavilski C, Yohai D, Bearman rs, airflow limitation. Eur Respir 1 1988; 1:560-4. 35. Wood DW, Downes JJ, Leeks HI. A clinical
Gorodischer R, Moses SW. Dexamethasone and 24. Hall CB, McBride rr, Gala CL, Hildreth SW, scoring system for the diagnosis of respiratory fail-
salbutamol in the treatment of acute wheezing in Schnabel KC. Ribavirin treatment of respiratory ure. Preliminary report on childhood status asth-
infants. Pediatrics 1983; 71:13-8. syncytial viral infection in infants with underlying maticus. Am 1 Dis Child 1972; 123:227-8.
12. Shann F, Hart K, Thomas D. Acute lower re- pulmonary disease. lAMA 1985; 254:3047-51. 36. Hall CB, Hall wr, Speers DM. Clinical and
spiratory tract infections in children: Possible criter- 25. McBride IT. Study design considerations for physiological manifestations of bronchiolitis and
ia for selection of patients for antibiotic therapy ribavirin: efficacy studies. Pediatr Infect Dis 1 1990; pneumonia. Outcome of respiratory syncytial vi-
and hospital admission. BullWHO 1984; 62:749-53. 9:S74-8. rus. Am 1 Dis Child 1979; 133:798-802.
13. Wald ER, Dashefsky B, Greem M. In re ribavi- 26. Amar D, Neidswski 1, Wald A, Finck AD. Flu-