Validating the Use of the Evaluation Tool of Childrens

HandwritingManuscript to Identify Handwriting

Difficulties and Detect Change in School-Age Children
Marie Brossard-Racine, Barbara Mazer, Marilyse Julien,
Annette Majnemer

KEY WORDS
 discriminant analysis
 handwriting
 occupational therapy
 child
 validation studies

In this study we sought to validate the discriminant ability of the Evaluation Tool of Childrens Handwriting
Manuscript in identifying children in Grades 23 with handwriting difficulties and to determine the
percentage of change in handwriting scores that is consistently detected by occupational therapists.
Thirty-four therapists judged and compared 35 pairs of handwriting samples. Receiver operating characteristic (ROC) analyses were performed to determine (1) the optimal cutoff values for word and letter
legibility scores that identify children with handwriting difficulties who should be seen in rehabilitation and
(2) the minimal clinically important difference (MCID) in handwriting scores. Cutoff scores of 75.0% for
total word legibility and 76.0% for total letter legibility were found to provide excellent levels of accuracy.
A difference of 10.0%12.5% for total word legibility and 6.0%7.0% for total letter legibility were found
as the MCID. Study findings enable therapists to quantitatively support clinical judgment when evaluating
handwriting.
Brossard-Racine, M., Mazer, B., Julien, M., & Majnemer, A. (2012). Validating the use of the Evaluation Tool of Childrens
HandwritingManuscript to identify handwriting difficulties and detect change in school-age children. American
Journal of Occupational Therapy, 66, 414421. http://dx.doi.org/10.5014/ajot.2012.003558

Marie Brossard-Racine, PhD, OT, is Postdoctoral

Fellow, Advanced Paediatric Brain Imaging Research
Laboratory, Diagnostic Imaging and Radiology/Fetal and
Transitional Medicine, Childrens National Medical Center,
Washington, DC. At the time of the study, she was a PhD
candidate at the School of Physical and Occupational
Therapy, McGill University, Montreal, Quebec.
Barbara Mazer, PhD, is Assistant Professor, School of
Physical and Occupational Therapy, McGill University,
Montreal, Quebec, and Research Associate, Centre for
Interdisciplinary Research in Rehabilitation of Greater
Montreal, Jewish Rehabilitation Hospital, 3205 Alton
Goldbloom, Laval, Quebec H7V 1R2 Canada; barbara.
[email protected]
Marilyse Julien, MSc, is Statistician Consultant, Centre
Hospitalier Universitaire Sainte-Justine, Montreal, Quebec.
Annette Majnemer, PhD, OT, FCAHS, is Director and
Associate Dean, School of Physical and Occupational
Therapy, McGill University, and Montreal Childrens
HospitalMcGill University Health Centre, Montreal,
Quebec.

he prevalence of handwriting difficulties has been estimated at 10%30% in

typically developing school-age children (Karlsdottir & Stefansson, 2002).
Children with delays or impairments in the development of these skills are
frequently referred to occupational therapy for evaluation and intervention
(Chandler, 1994; Oliver, 1990). Although computers are now commonly used
in classrooms, traditional handwriting is still the primary medium of written
expression required by teachers in the school environment (Latio, 2009).
Handwriting is a challenging construct to measure. Although many
standardized evaluations exist (see Rosenblum, Weiss, & Parush, 2003, for a
review), pediatric occupational therapists do not tend to use standardized assessments of handwriting and prefer to use standardized tests to evaluate the
underlying components of the task rather than the handwriting performance
itself (Crowe, 1989; Feder, Majnemer, & Synnes, 2000; Rodger, 1994). One
reason for this practice may be that handwriting is a complex task involving
many sensorimotor, cognitive, and psychosocial components that must be
comprehensively evaluated before designing a treatment plan (Schneck &
Amundson, 2010). In addition, most standardized handwriting evaluations
involve only a single task, usually copying or sentence composition, which
might not be sufficient to capture the many aspects of a childs handwriting
performance (Feder & Majnemer, 2003).
The Evaluation Tool of Childrens Handwriting (ETCH; Amundson,
1995) is the only Latin alphabetbased standardized measure that assesses

414
Downloaded From: http://ajot.aota.org/ on 02/10/2015 Terms of Use: http://AOTA.org/terms

July/August 2012, Volume 66, Number 4

a range of handwriting tasks similar to those experienced

in the classroom. It takes approximately 30 min to administer and is designed to identify and characterize
handwriting difficulties in young school-age children.
The ETCH is available in both manuscript and cursive
versions. Letters, numerals, and words are judged for
legibility using a list of specific criteria such as omission,
closing, misplacing, reversion, and poor erasure. The
percentage of legibility is determined for each task by
counting the legible letters, numerals, or words and dividing by the total number of letters, numerals, or words
required. The percentages from each task are then averaged to provide a total legibility score for letters, numerals, and words. The manuscript version of the ETCH
(ETCHM) targets children in Grades 1 to 3 and examines legibility through seven different tasks: alphabet
writing from memory (upper- and lowercase), numeral
writing from memory, near-point copying, far-point
copying, dictation of nonwords (nonsense words) and
numbers, and composition of a short sentence. Performance time or writing speed is measured in seconds for
the alphabet and numeral writing and in letters per
minute for the copying and composition tasks. The variety of tasks makes this criterion-referenced measure the
most ecological evaluation of handwriting (Feder &
Majnemer, 2003).
Content validity was established in three pilot editions
of the ETCH (Amundson, 1995). The concurrent validity of the ETCHM, however, has been questioned.
Sudsawad, Trombly, Henderson, and Tickle-Degnen
(2001) obtained a low and insignificant level of correlation between ETCHM scores and teacher judgments
and found that the ETCHM was not sensitive enough
to identify the level of difficulty experienced by handwriters whom teachers perceived to be below average.
Nevertheless, an unpublished study (Grace-Frederick,
1998) provided evidence that the ETCHM was able
to discriminate between second-grade children who had
handwriting performance that was judged by teachers to
be either below or above average. Unfortunately, no cutoff
value has been published to identify children with belowaverage performance on the ETCHM. Moreover, the
concurrent validity of the ETCHM with the perceptions
of rehabilitation clinicians working with children who
experience handwriting difficulties has not been reported.
Clinicians often use the ETCHM as a measure of
change before and after an intervention. Testretest reliability indicators for the ETCHM are r 5 .63 for total
numeral legibility, r 5 .77 for total letter legibility, and
r 5 .71 for total word legibility. Total scores for letter
and word legibility were found to be more consistent

than the individual task scores (Diekema, Deitz, &

Amundson, 1998). The interrater reliability with intraclass correlation coefficients (ICCs) ranged from .42 to
.84 for the individual items on the ETCHM for children referred to occupational therapy (Amundson, 1995).
Specifically, the ICC was .48 for total word legibility
and .84 for total letter legibility. To our knowledge, responsiveness of the ETCHM has never been reported.
In general, the ability of a tool to discriminate among
clinical groups can be considered preliminary evidence
that it may be responsive to change (Revicki, Hays, Cella,
& Sloan, 2008). In health-related quality-of-life research,
in which self-report questionnaires are used extensively,
the construct of a minimal clinically important difference
(MCID) was documented. MCID corresponds to the
smallest change in scores that would likely be important
from the patients or clinicians perspective (Revicki et al.,
2008, p. 103). This construct can also apply to outcome
measures, because they are known to have their share of
variance caused by an individuals personal factors or
by the measurement procedure itself (Leemrijse, Meijer,
Vermeer, Lambregts, & Ade`r, 1999). It is thus relevant
and important that clinicians and researchers pay more
attention to MCID and integrate this construct when
measuring change in scores over time. This psychometric
construct has not yet been studied for the ETCH.
The purposes of this study were (1) to identify the
ETCHM cutoff values for word and letter legibility that
discriminate between children with handwriting difficulties who should be seen in rehabilitation for a comprehensive evaluation or treatment and those who do not
require services and (2) to determine the MCID on the
ETCHM letter and word total legibility by determining
the percentage of change that corresponds to the change
that experienced pediatric occupational therapists consistently visually detect.

Method
This cross-sectional study was approved by the Montreal
Childrens HospitalMcGill University Health Centre
Research Ethics Board. Written consent was obtained
before each evaluation.
Participant Selection
Participants included a convenience sample of pediatric
occupational therapists working in the greater Montreal,
Quebec, area. To be eligible for the study, the occupational therapists had to be working with children and had
to have at least 1 yr of experience in evaluating and
treating handwriting difficulties.

The American Journal of Occupational Therapy

Downloaded From: http://ajot.aota.org/ on 02/10/2015 Terms of Use: http://AOTA.org/terms

415

The handwriting samples were obtained from a group

of children involved in a cohort study of school-age
children with attention deficit hyperactivity disorder
(ADHD). This study evaluated handwriting performance
at diagnosis and after initiation of stimulant medication
(Brossard-Racine, Majnemer, Shevell, Snider, & Belanger,
2011; Brossard-Racine, Shevell, Snider, Belanger, &
Majnemer, 2012). Children ages 79 yr in Grades 2 and 3
who routinely used manuscript handwriting and did not
learn cursive writing for a period greater than 2 mo were
included in the cohort. Each child was evaluated once
before initiation of medication and also completed one
or two evaluations following initiation of medication. In
total, 26 children produced a total of 35 pairs of handwriting samples (referred to as Sample A [before initiation of medication] and Sample B [after initiation of
medication]).

much better than A, 2 5 B is somewhat better than A,

3 5 B is about the same as A, 4 5 B is somewhat worse
than A, 5 5 B is much worse than A). Change was defined
to the therapists as at least a small decrease or increase
between two samples, including a large amount of
change. Change was independent of the initial performance (e.g., it did not mean that a child no longer had
difficulties). The therapist was unaware of the childs
identity or condition, time between the two evaluations,
whether any intervention had occurred between the two
samples, and legibility scores obtained for each sample.
However, they were informed that the samples were
all produced by children in Grade 2 or 3, age 79 yr,
who were attending a regular school. In addition to these
two questions, each participant completed a short sociodemographic questionnaire to collect information on their
work environment and level of experience.

Instruments

Procedures

ETCHM. The ETCHM (Amundson, 1995) was the

standardized measure of handwriting used to evaluate
the children. An experienced occupational therapist who
was unaware of the study hypotheses administered the
ETCHM according to instructions in the examiners
manual. All samples had been previously scored by an
independent corrector who was blind to the childrens
identity and to the timing of testing (i.e., whether the
sample was obtained before or after medication initiation). The occupational therapist scorer was trained for
scoring consistency, as required by the ETCH examiners
manual. Only the total scores for word legibility and
letter legibility were included in the analysis.
Occupational Therapists Ratings of Handwriting. The
participating occupational therapists completed a questionnaire that asked two questions for each pair of
handwriting samples. The first questions referred only to
Sample A and aimed to establish consistent cutoff values
that would distinguish children with handwriting difficulties who should be seen in rehabilitation for comprehensive evaluation or treatment from those who did not
require services. Occupational therapists were asked to
answer the following question: Looking only at Sample
A, does this child need handwriting rehabilitation services? The therapists were directed to view the handwriting
sample as though it had been submitted by a teacher or
a parent and to respond only on the basis of their visual
inspection and clinical experience. The therapists replied
with one of the following responses: yes, no, or not sure.
The second question asked occupational therapists to
compare the two samples, A and B, using a 5-point Likert
scale to indicate the degree of observable change (1 5 B is

As part of the broader study described earlier (BrossardRacine et al., 2011, 2012), the ETCHM was administered by an occupational therapist to the children after
obtaining written informed parental consent and each
childs assent. The primary investigator (Marie BrossardRacine) selected independent samples of handwriting for
use in this study. An e-mail with a brief description of the
project was sent to the occupational therapy program
coordinators in the two major pediatric hospitals and
their affiliated rehabilitation center and to the largest
pediatric occupational therapy private practice clinic in
the province of Quebec. Once an occupational therapist
expressed an interest in participating, he or she was
contacted by the investigator to further clarify the study
objectives and procedures. A time and place of mutual
convenience to meet and conduct the study was then
arranged. The total time to complete the procedures and
the questionnaire was approximately 45 min.
Data Collection
Every occupational therapist met individually with Marie
Brossard-Racine to complete the questionnaire. The same
sequence of samples was presented to all occupational
therapists, who were subsequently directed to complete the
two questions for each pair of samples. A research assistant
entered all data into a database.
Data Analysis
Descriptive statistics were used to characterize participants demographic and clinical attributes. To determine
the agreement (interrater reliability) between the 34 participants on each of the two questions, ShroutFleiss

416
Downloaded From: http://ajot.aota.org/ on 02/10/2015 Terms of Use: http://AOTA.org/terms

July/August 2012, Volume 66, Number 4

reliability ICCs with random set were calculated (Shrout

& Fleiss, 1979). Receiver operating characteristic (ROC)
analyses were conducted to evaluate the ability of the
ETCHM total word and letter legibility scores to
identify children who required intervention, as determined by the occupational therapists perceptions. First,
a crude ROC curve was derived by setting cutoff scores
on the handwriting scores. The crude area under the
curve and its standard error were calculated using the
trapezoidal rule described by Hanley and McNeil (1982).
In addition, to account for the repeated-measures design
of the study, an adjusted ROC curve was derived by
setting cutoff probabilities on the predicted probabilities
of the generalized linear mixed model with random intercepts for children and occupational therapists. The
area under the adjusted curve was estimated using the
trapezoidal rule, and the standard error was estimated
using a bootstrap procedure (Liu & Li, 2005). For the
total word legibility and letter legibility scores, the optimal value was defined as the shortest distance to sensitivity 5 1 and 1 2 specificity 5 0 on the crude ROC
curve. The optimal value also corresponded to the value
that best discriminated between children with handwriting difficulties who should be seen in rehabilitation and
those who did not need services.
Although no consensus exists on the best statistical
way to determine MCID, a combination of an anchorbased approach, in which change scores are compared with
clinicians or clients perceptions, and a distribution-based
approach, which focuses on the variation of the sample and
on the measurements statistical properties, is recommended (Guyatt, Osoba, Wu, Wyrich, & Norman, 2002;
Hays & Woolley, 2000; Revicki et al., 2008). It is also
recommended to determine a range of values for MCID
because they are sensitive to variation across participants
(Yost & Eton, 2005). In light of these recommendations,
we first evaluated the ability of the ETCHM word and
letter legibility scores measured at two time points to
identify children whose samples were perceived by experienced occupational therapists as having changed (either
negatively or positively) in their handwriting skills using
the same methods used for ROC analysis. Of all the possible distribution-based approaches, it has been demonstrated that half a standard deviation at baseline (0.5
standard deviation) is the most reliable statistic to determine MCID (Norman, Sloan, & Wyrwich, 2003, 2004;
Revicki et al., 2008); therefore, the results for the anchorbased approach were compared with 0.5 standard deviation
for both word and letter total percentage of change. For all
analyses, a confidence interval (CI) of 95% was used, and
all analyses were performed using SAS (Version 9.0; SAS

Institute, Cary, NC) and R (Version 12.0; R Foundation

for Statistical Computing, Vienna, Austria).

Results
Participants
The mean age of the 26 children who produced the samples
was 8.1 0.8 yr. Mean total word legibility was 70.6%
24.9%; the range of values was 20.0%100.0%; mean
total letter legibility was 73.3% 13.9%, and the range of
values was 32.0%96.0%. For the 35 pairs of samples that
were compared, the range of percentage of change (using
the legibility scores of the ETCH) was 222.0% to 45.0%
for word legibility and 28.0% to 32.0% for letter legibility; mean change scores were 6.8% 14.7% for word
legibility and 3.8% 9.4% for letter legibility.
Thirty-four occupational therapists (33 women,
1 man) participated in this study. Twenty-one percent
of the sample had 12 yr of experience working as an
occupational therapist with clients who had handwriting
difficulties, 32% had 35 yr of such experience, and
47% had >6 yr of such experience. Nineteen therapists
were primarily based in a private clinic, 7 were in a rehabilitation center, 5 were part of a school board, and
3 were in a hospital setting. For most participants (30
of 34), 76%100% of their caseload consisted of schoolage children. The most common diagnoses or associated
conditions in the children treated with handwriting
difficulties were learning difficulties (including ADHD),
coordination difficulties (including developmental coordination disorder), and autism spectrum disorders.
Discriminative Validity
The ICC obtained for interrater reliability for the question, Does this child need handwriting rehabilitation
services? was .53. The ROC curves for word legibility
and letter legibility are presented in Figures 1 and 2. The
crude area under the ROC curve for word legibility was
.86 (95% CI 5 .84, .88), and the adjusted area was .96
(95% CI 5 .95, .97). We obtained similar results for
letter legibility, with a crude area under the curve of .82
(95% CI 5 .80, .84) and an adjusted area of .96 (95%
CI 5 .95, .97). We ignored the fact that the repeatedmeasures design of the study produced estimates that
were biased toward zero for the area under the ROC
curve, as suggested by Liu and Li (2005). The optimal
cutoff score in word legibility scores obtained from inspection of the crude ROC curve was 75%, which corresponded to a crude sensitivity of 77% and specificity of
82%. For letter legibility results, an optimal cutoff score

The American Journal of Occupational Therapy

Downloaded From: http://ajot.aota.org/ on 02/10/2015 Terms of Use: http://AOTA.org/terms

417

Figure 1. Receiver operating characteristics for total word legibility

and occupational therapists perception.

of 76% simultaneously maximized sensitivity and specificity, with a sensitivity of 78% and a specificity of 66%.

The optimal cutoff value for the percentage change in

word legibility score was 10.0%, for a sensitivity of 58.0%
and a specificity of 58.0%. For the absolute change in
letter legibility, we identified 6.0% as the optimal cutoff
score, which corresponded to a sensitivity of 62.0% and
a specificity of 57.0% to detect the presence of change in
handwriting. The consistency of responses across occupational therapists (interrater reliability) was weak (ICC 5
.19). We conducted subgroup analyses to identify clinical
and demographic variables that may have affected the homogeneity of responses; however, we found similar ICCs
among subgroups of occupational therapists working in
the same environment or with the same number of years of
experience.
Distribution-Based Approach. We used the simple estimate of half a standard deviation for both criteria and
obtained a value of 12.5% as the MCID for total word
legibility and 7.0% for total letter legibility. Thus,
combining the two approaches, the MCID for total word
legibility ranges from 10.0% to 12.5% and for total letter
legibility from 6.0% to 7.0%.

Minimal Clinically Important Difference

Anchor-Based Approach. Because the direction of

change (worse or better) was not of concern, the absolute

change in word and letter scores was used to construct the
ROC curves. The crude and adjusted areas under the ROC
curves were .60 (95% CI 5 .58, .62) and .77 (95% CI 5
.75, .80), respectively, for the absolute change in word
legibility and .61 (95% CI 5 .58, .62) and .77 (95%
CI 5 .75, .80), respectively, for the absolute change in
letter legibility. The ROC curves for word legibility and
letter legibility are presented in Figures 3 and 4.

Figure 2. Receiver operating characteristics for total letter legibility

and occupational therapists perception.

Discussion
Discriminative Validity
The primary objective of this study was to identify cutoff
values for total word legibility and letter legibility percentages on the ETCHM that would discriminate between children whom experienced therapists perceived as
requiring rehabilitation services (evaluation or treatment)
for handwriting difficulties and those who did not require
services. Because of the repetitive design of this study (34

Figure 3. Receiver operating characteristics for change scores in

ETCHM total word legibility and occupational therapists perception.

418
Downloaded From: http://ajot.aota.org/ on 02/10/2015 Terms of Use: http://AOTA.org/terms

July/August 2012, Volume 66, Number 4

Figure 4. Receiver operating characteristics for change scores

in ETCHM total letter legibility and occupational therapists
perception.

occupational therapists evaluating the same 26 samples),

the ROC curve was adjusted to minimize bias (Liu & Li,
2005). The bias appeared to be moderate, however,
because the occupational therapists obtained relatively
good interrater reliability (ICC 5 .53). Good agreement
between the crude and adjusted areas under the curve
was obtained. The high adjusted area under the curve
for both total word legibility and total letter legibility
implied that these two criteria had excellent accuracy
in distinguishing children. According to the pediatric
occupational therapists, 75.0% total word legibility and
76.0% total letter legibility discriminated between
children with and without handwriting difficulties.
These discriminative values are lower than the 85.0%
cutoff scores suggested by Admunson (1995) in the
ETCH manual. As Admunson explained, however, the
ETCH was still a work in progress at the time of publication of the manual, and the suggested values were only
a starting point that required validation. Because no other
cutoff scores for the ETCHM have been published, we
are unable to compare our results with other findings.
Nevertheless, our cutoff values are similar to those reported
for the cursive version of the ETCH (75.0% for total word
legibility and 82.0% for total letter legibility; Koziatek &
Powell, 2002).
Cutoff values are limited; they consider only the
handwriting product and do not take into account different factors that can affect performance (e.g., level
of experience, time of year, instructional method, environmental factors). Nevertheless, cutoff values provide
quantitative evidence to support therapists when performing comprehensive evaluations and justifying the

need for intervention (Majnemer & Limperopoulos,

2002). The cutoff values we determined in this study
relate specifically to children in Grades 2 and 3. Childrens handwriting usually reaches a plateau by the end of
Grade 2, and no significant improvement is expected in
the following years (Graham, Weintraub, & Berninger,
2001). Also, none of the children in Grade 2 were tested
in the early months of the school year, and we are
therefore confident that these cutoff scores can guide
clinicians in interpreting legibility for children within
this age group. In this study, several occupational therapists verbalized their frustration at being limited to a
handwriting sample and not being able to observe the
childrens performance. Nevertheless, matching visual
perception to scores obtained on a standardized evaluation is an unbiased way to ascertain the discriminative
validity of an instrument.
Minimal Clinically Important Difference in Legibility
Identifying the percentage of change in total word legibility and total letter legibility that clinicians consistently
detected resulted in lower levels of accuracy. According to
the adjusted area under the curve, only fair accuracy to
distinguish change was obtained. Considerable differences
between the crude and adjusted estimates of the area under
the curves were also found. Those differences could be
explained by the poor level of interrater reliability found
among the occupational therapists (ICC 5 .19), indicating that the therapists often rated the same sample
differently. Subgroup analyses according to workplace
and level of experience did not further clarify the inconsistency of ratings among occupational therapists. The
therapists who participated in the current study did not
score or administer the ETCH themselves; they were
asked to judge samples only by looking at written product. We felt this to be the best way to evaluate perceptions of legibility change because the examiners were not
potentially biased by any external factors that would have
interfered with their perspectives on the childs handwriting legibility. It was therefore surprising to see the
low level of agreement for this question, which contrasted
considerably with the level of agreement reached for the
question on the childrens handwriting difficulties requiring rehabilitation.
Several explanations are possible for the low level of
agreement among occupational therapists in rating clinically meaningful change. First, they were not previously
trained for scoring consistency; the likelihood of error was
therefore higher. Also, the group of occupational therapists who participated was not homogeneous; it included
clinicians with different degrees of experience, from

The American Journal of Occupational Therapy

Downloaded From: http://ajot.aota.org/ on 02/10/2015 Terms of Use: http://AOTA.org/terms

419

different workplaces, who served different clientele even

when from the same institution. Another possible explanation is that the question posed to the occupational
therapists might have been too broad and nonspecific.
It addressed overall change between the two samples and
was not specifically focused on change in word or letter
legibility.
Several factors are known to interfere with handwriting legibility. Difficulties in letter formation, spacing,
and alignment, to name just a few, can affect word and
letter legibility differently (Schneck & Amundson, 2010).
Change in word and letter legibility between two samples
was not always of the same magnitude. For instance, in
Sample B, spacing could have been improved, thereby
dramatically increasing word legibility; however, letter
formation could have been as poor as that of Sample A,
and thus letter legibility did not change sufficiently from
a clinical perspective. In this type of situation, each occupational therapist had to decide on the overall change,
with no directive regarding specific aspects of word or
letter legibility. Some therapists could have prioritized
change (or lack thereof) in letter legibility over change in
word legibility, whereas others may have done the opposite. This lack of guidance may have specifically affected perception of change. The first question on the
need for handwriting intervention might have been more
stable because it did not take into account the relative
severity of difficulties or the childs level of difficulty with
letter or word legibility, because both might appear in the
presence of handwriting difficulties.
This weak level of agreement on what constituted
a level of change also supported the importance of using
a combination of approaches when determining MCID.
In our case, we obtained fairly similar MCID values using
two methods. It has been proposed that anchor-based
values have more weight than distribution-based values
(Revicki et al., 2008), and we thus suggest that our
findings may be used as a starting point to appreciate
change in spite of the fair level of accuracy we reached
using the anchor-based method. Nevertheless, more
studies on the MCID of the ETCHM would help
confirm the most appropriate range of values.
Finally, the discriminative validity and MCID values
found for word legibility must be interpreted with caution
because the ETCHM interrater reliability for total word
legibility was somewhat low (ICC 5 .48). Low interrater
reliability may have influenced our ability to find accurate
cutoff and MCID values for this specific subscale. Scores
found for letter legibility are probably more accurate
because the interrater reliability for this domain was
higher (ICC 5 .84).

Implications for Occupational

Therapy Practice
For children in Grades 2 and 3, 75.0% total word legibility
and 76.0% total letter legibility on the ETCHM are suggested as the cutoff values to discriminate between children
with handwriting legibility difficulties who should be seen in
rehabilitation for evaluation and treatment from those who
have no such difficulties. With the ETCHM, clinicians
should not consider a change in scores over time as clinically
significant if that change corresponds to less than 6.0% for
total letter legibility and less than 10.0% for total word
legibility. To summarize, the findings of this study have the
following implications for occupational therapy practice:
Preliminary cutoff values of 75.0% for total word
legibility and of 76.0% for total letter legibility on
the ETCHM are suggested to discriminate between
children in Grades 2 and 3 who present with handwriting difficulties and those who do not.
These quantitative cutoff values can support clinicians
in justifying need for intervention.
Change in handwriting scores <6.0% for total letter
legibility and <10.0% for total word legibility should
not be considered clinically important.

Conclusion
Our results contribute to the documentation of the psychometric properties of the ETCHM and provide new
information on the concurrent (discriminative) validity
when comparing total legibility scores with the perceptions
of pediatric occupational therapists. Although the cutoff
scores demonstrated excellent discriminative abilities, discriminative values do not replace clinical judgment, and
clinicians should always perform a comprehensive evaluation of the childs performance. The MCID levels are essential to measure the real impact of intervention. Further
study is needed to determine cutoff values and clinically
meaningful change in handwriting legibility scores. s

Acknowledgments
Marie Brossard-Racine was supported by a doctoral fellowship from the Fond de Recherche en Sante du Quebec
(FRSQ) during preparation of this article. She was also
supported by an Edith and Richard Strauss Fellowship and
a studentship from the Stars/Montreal Childrens Hospital
Studentships in early phases of this work. The research
laboratory benefited from infrastructure provided by the
Montreal Childrens Hospital Research Institute and the
Centre de Recherche Interdisciplinaire en Readaptation,
both supported by the FRSQ. Special thanks to all the

420
Downloaded From: http://ajot.aota.org/ on 02/10/2015 Terms of Use: http://AOTA.org/terms

July/August 2012, Volume 66, Number 4

occupational therapists who took the time to participate in

this project, to Joey Waknin for research assistance, and to
Victoria Surtees for manuscript correction.

References
Amundson, S. J. (1995). Evaluation Tool of Childrens Handwriting: ETCH examiners manual. Homer, AK: O.T. Kids.
Brossard-Racine, M., Majnemer, A., Shevell, M., Snider, L., &
Belanger, S. A. (2011). Handwriting capacity in children
newly diagnosed with attention deficit hyperactivity disorder. Research in Developmental Disabilities, 32, 29272934.
Brossard-Racine, M., Shevell, M., Snider, L., Belanger, S. A.,
& Majnemer, M. (2012). Persistent handwriting difficulties
in children with attention deficit hyperactivity disorder
(ADHD) following treatment with stimulant medication.
Manuscript submitted for publication.
Chandler, B. (1994, December 15). The power of information:
School-based practice survey results. OT Week, 18, 24.
Crowe, T. K. (1989). Pediatric assessments: A survey of their
use by occupational therapists in northwestern school systems.
OTJR: Occupation, Participation and Health, 9, 273286.
Diekema, S. M., Deitz, J., & Amundson, S. J. (1998). Testretest
reliability of the Evaluation Tool of Childrens Handwriting
Manuscript. American Journal of Occupational Therapy, 52,
248255. http://dx.doi.org/10.5014/ajot.52.4.248
Feder, K. P., & Majnemer, A. (2003). Childrens handwriting
evaluation tools and their psychometric properties. Physical and Occupational Therapy in Pediatrics, 23, 6584.
http://dx.doi.org/10.1080/J006v23n03_05
Feder, K. P., Majnemer, A., & Synnes, A. (2000). Handwriting: Current trends in occupational therapy practice. Canadian Journal of Occupational Therapy, 67, 197204.
Grace-Frederick, L. (1998). Printing legibility, pencil grasp, and
the use of the ETCHM. Unpublished masters thesis, Boston University.
Graham, S., Weintraub, N., & Berninger, V. (2001). Which
manuscript letters do primary grade children write legibly?
Journal of Educational Psychology, 93, 488497 http://dx.
doi.org/10.1037/0022-0663.93.3.488
Guyatt, G. H., Osoba, D., Wu, A. W., Wyrwich, K. W., &
Norman, G. R.; Clinical Significance Consensus Meeting
Group. (2002). Methods to explain the clinical significance of health status measures. Mayo Clinic Proceedings,
77, 371383. http://dx.doi.org/10.4065/77.4.371
Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of
the area under a receiver operating characteristic (ROC)
curve. Radiology, 143, 2936.
Hays, R. D., & Woolley, J. M. (2000). The concept of clinically
meaningful difference in health-related quality-of-life research:
How meaningful is it? PharmacoEconomics, 18, 419423 http:
//dx.doi.org/10.2165/00019053-200018050-00001
Karlsdottir, R., & Stefansson, T. (2002). Problems in developing functional handwriting. Perceptual and Motor Skills,
94, 623662.
Koziatek, S. M., & Powell, N. J. (2002). A validity study of the
Evaluation Tool of Childrens HandwritingCursive.
American Journal of Occupational Therapy, 56, 446453.
http://dx.doi.org/10.5014/ajot.56.4.446

Latio, G. W. (2009). Examination of factors that influence computer technology use for classroom instruction by teachers in
Ohio public high schools. Unpublished doctoral dissertation, Ohio University College of Education, Athens.
Leemrijse, C., Meijer, O. G., Vermeer, A., Lambregts, B., & Ade`r,
H. J. (1999). Detecting individual change in children with mild
to moderate motor impairment: The standard error of measurement of the Movement ABC. Clinical Rehabilitation, 13,
420429. http://dx.doi.org/10.1191/026921599675491988
Liu, H., & Li, G. (2005). Testing statistical significance of the
area under a receiver operating characteristics curve for
repeated measures design with bootstrapping. Journal of
Data Science, 3, 257278.
Majnemer, A., & Limperopoulos, C. (2002). Importance of
outcome determination in pediatric rehabilitation. Developmental Medicine and Child Neurology, 44, 773777.
http://dx.doi.org/10.1111/j.1469-8749.2002.tb00285.x
Norman, G. R., Sloan, J. A., & Wyrwich, K. W. (2003). Interpretation of changes in health-related quality of life:
The remarkable universality of half a standard deviation.
Medical Care, 41, 582592.
Norman, G. R., Sloan, J. A., & Wyrwich, K. W. (2004). The
truly remarkable universality of half a standard deviation:
Confirmation through another look. Expert Review of
Pharmacoeconomics and Outcomes Research, 4, 581585.
Oliver, C. E. (1990). A sensorimotor program for improving
writing readiness skills in elementary-age children. American Journal of Occupational Therapy, 44, 111116. doi:
http://dx.doi.org/10.5014/ajot.44.2.111
Revicki, D., Hays, R. D., Cella, D., & Sloan, J. (2008). Recommended methods for determining responsiveness and
minimally important differences for patient-reported outcomes. Journal of Clinical Epidemiology, 61, 102109.
http://dx.doi.org/10.1016/j.jclinepi.2007.03.012
Rodger, S. (1994). A survey of assessments used by paediatric
occupational therapists. Australian Occupational Therapy
Journal, 41, 137142. http://dx.doi.org/10.1111/j.14401630.1994.tb01300.x
Rosenblum, S., Weiss, P. L., & Parush, S. (2003). Product and
process evaluation of handwriting difficulties: A review.
Educational Psychology Review, 15, 4181. http://dx.doi.
org/10.1023/A:1021371425220
Schneck, C. M., & Amundson, S. J. (2010). Prewriting and
handwriting skills. In J. Case-Smith (Ed.), Occupational
therapy for children (6th ed., pp. 555582). St. Louis,
MO: Mosby/Elsevier.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations:
Uses in assessing rater reliability. Psychological Bulletin, 86,
420428. http://dx.doi.org/10.1037/0033-2909.86.2.420
Sudsawad, P., Trombly, C. A., Henderson, A., & Tickle-Degnen,
L. (2001). The relationship between the Evaluation Tool of
Childrens Handwriting and teachers perceptions of handwriting legibility. American Journal of Occupational Therapy, 55,
518523. http://dx.doi.org/10.5014/ajot.55.5.518
Yost, K. J., & Eton, D. T. (2005). Combining distributionand anchor-based approaches to determine minimally important differences: The FACIT experience. Evaluation
and the Health Professions, 28, 172191. http://dx.doi.
org/10.1177/0163278705275340

The American Journal of Occupational Therapy

Downloaded From: http://ajot.aota.org/ on 02/10/2015 Terms of Use: http://AOTA.org/terms

421