An American National Standard

Designation: E 177 04e1

Standard Practice for

Use of the Terms Precision and Bias in ASTM Test Methods1
This standard is issued under the fixed designation E 177; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (e) indicates an editorial change since the last revision or reapproval.

e1 NOTEEditorial corrections were made in 27.3.3 and 28.2 in April 2005.

1. Scope (3) a consensus or certified value, based on collaborative

1.1 The purpose of this practice is to present concepts experimental work under the auspices of a scientific or
necessary to the understanding of the terms precision and engineering group.
bias as used in quantitative test methods. This practice also 3.1.2 accuracy, nthe closeness of agreement between a
describes methods of expressing precision and bias and, in a test result and an accepted reference value.
final section, gives examples of how statements on precision 3.1.2.1 DiscussionThe term accuracy, when applied to a
and bias may be written for ASTM test methods. set of test results, involves a combination of a random
1.2 This standard does not purport to address all of the component and of a common systematic error or bias compo-
safety concerns, if any, associated with its use. It is the nent.
responsibility of the user of this standard to establish appro- 3.1.3 bias, nthe difference between the expectation of the
priate safety and health practices and determine the applica- test results and an accepted reference value.
bility of regulatory requirements prior to use. 3.1.3.1 DiscussionBias is the total systematic error as
contrasted to random error. There may be one or more
2. Referenced Documents systematic error components contributing to the bias. A larger
2.1 ASTM Standards: 2 systematic difference from the accepted reference value is
E 178 Practice for Dealing with Outlying Observations reflected by a larger bias value.
E 456 Terminology Relating to Quality and Statistics 3.1.4 intermediate precision, nthe closeness of agreement
E 691 Practice for Conducting an Interlaboratory Study to between test results obtained under specified intermediate
Determine the Precision of a Test Method precision conditions.
E 1169 Guide for Conducting Ruggedness Tests 3.1.4.1 DiscussionThe specific measure and the specific
2.2 Other Document: conditions must be specified for each intermediate measure of
ISO 3534 StatisticsVocabulary and Symbols3 precision; thus, standard deviation of test results among
operators in a laboratory, or day-to-day standard deviation
3. Terminology within a laboratory for the same operator.
3.1 Definitions: 3.1.4.2 DiscussionBecause the training of operators, the
3.1.1 accepted reference value, na value that serves as an agreement of different pieces of equipment in the same
agreed-upon reference for comparison, and which is derived laboratory and the variation of environmental conditions with
as: (1) a theoretical or established value, based on scientific longer time intervals all depend on the degree of within-
principles, (2) an assigned or certified value, based on experi- laboratory control, the intermediate measures of precision are
mental work of some national or international organization, or likely to vary appreciably from laboratory to laboratory. Thus,
intermediate precisions may be more characteristic of indi-
vidual laboratories than of the test method.
1
This practice is under the jurisdiction of ASTM Committee E11 on Quality and 3.1.5 intermediate precision conditions, nconditions un-
Statistics and is the direct responsibility of Subcommittee E11.20 on Test Method
Evaluation and Quality Control.
der which test results are obtained with the same test method
Current edition approved Nov. 1, 2004. Published December 2004. Originally using test units or test specimens (10.3) taken at random from
approved in 1961. Last previous edition approved in 2002 as E 177 90a (2002). a single quantity of material that is as nearly homogeneous as
2
For referenced ASTM standards, visit the ASTM website, www.astm.org, or possible, and with changing conditions such as operator,
contact ASTM Customer Service at [email protected]. For Annual Book of ASTM
Standards volume information, refer to the standards Document Summary page on
measuring equipment, location within the laboratory, and time.
the ASTM website. 3.1.6 precision, nthe closeness of agreement between
3
Available from American National Standards Institute (ANSI), 25 W. 43rd St., independent test results obtained under stipulated conditions.
4th Floor, New York, NY 10036.

Copyright ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959, United States.

1
 E 177 04e1
3.1.6.1 DiscussionPrecision depends on random errors 3.1.10.3 DiscussionThe repeatability standard deviation,
and does not relate to the accepted reference value. usually considered a property of the test method, will generally
3.1.6.2 DiscussionThe measure of precision usually is be smaller than the within-laboratory standard deviation. (See
expressed in terms of imprecision and computed as a standard within-laboratory standard deviation.)
deviation of the test results. Less precision is reflected by a 3.1.11 reproducibility, nprecision under reproducibility
larger standard deviation. conditions.
3.1.6.3 DiscussionIndependent test results means re- 3.1.12 reproducibility conditions, nconditions where test
sults obtained in a manner not influenced by any previous results are obtained with the same method on identical test
result on the same or similar test object. Quantitative measures items in different laboratories with different operators using
of precision depend critically on the stipulated conditions. different equipment.
Repeatability and reproducibility conditions are particular sets
3.1.12.1 DiscussionIdentical material means either the
of extreme stipulated conditions.
same test units or test specimens are tested by all the
3.1.7 repeatability, nprecision under repeatability condi-
laboratories as for a nondestructive test or test units or test
tions.
specimens are taken at random from a single quantity of
3.1.7.1 DiscussionRepeatability is one of the concepts or material that is as nearly homogeneous as possible.
categories of the precision of a test method.
3.1.7.2 DiscussionMeasures of repeatability defined in A different laboratory of necessity means a different operator,
this compilation are repeatability standard deviation and re- different equipment, and different location and under different super-
peatability limit. visory control.
3.1.8 repeatability conditions, nconditions where inde- 3.1.13 reproducibility limit (R), nthe value below which
pendent test results are obtained with the same method on the absolute difference between two test results obtained under
identical test items in the same laboratory by the same operator reproducibility conditions may be expected to occur with a
using the same equipment within short intervals of time. probability of approximately 0.95 (95 %).
3.1.8.1 DiscussionSee precision, 3.1.6.3. The same op- 3.1.13.1 DiscussionThe reproducibility limit is 2.8 ('
erator, same equipment requirement means that for a particu- 1.96 =2 ) times the reproducibility standard deviation. The
lar step in the measurement process, the same combination of multiplier is independent of the size of the interlaboratory
operator and equipment is used for every test result. Thus, one study (that is, of the number of laboratories participating).
operator may prepare the test specimens, a second measure the
3.1.13.2 DiscussionThe approximation to 0.95 is reason-
dimensions and a third measure the mass in a test method for
ably good (say 0.90 to 0.98) when many laboratories (30 or
determining density.
more) are involved but is likely to be poor when fewer than
3.1.8.2 DiscussionBy in the shortest practical period of eight laboratories are studied.
time is meant that the test results, at least for one material, are
obtained in a time period not less than in normal testing and not 3.1.14 reproducibility standard deviation (SR), nthe stan-
so long as to permit significant change in test material, dard deviation of test results obtained under reproducibility
equipment or environment. conditions.
3.1.9 repeatability limit (r), nthe value below which the 3.1.14.1 DiscussionOther measures of the dispersion of
absolute difference between two individual test results obtained test results obtained under reproducibility conditions are the
under repeatability conditions may be expected to occur with a reproducibility variance and the reproducibility coefficient
probability of approximately 0.95 (95 %). of variation.
3.1.9.1 DiscussionThe repeatability limit is 2.8 (' 1.96 3.1.14.2 DiscussionThe reproducibility standard devia-
=2 ) times the repeatability standard deviation. This multi- tion includes, in addition to between-laboratory variability, the
plier is independent of the size of the interlaboratory study. repeatability standard deviation and a contribution from the
3.1.9.2 DiscussionThe approximation to 0.95 is reason- interaction of laboratory factors (that is, differences between
ably good (say 0.90 to 0.98) when many laboratories (30 or operators, equipment and environments) with material factors
more) are involved, but is likely to be poor when fewer than (that is, the differences between properties of the materials
eight laboratories are studied. other than that property of interest).
3.1.10 repeatability standard deviation, nthe standard 3.1.15 within-laboratory standard deviation, nthe stan-
deviation of test results obtained under repeatability condi- dard deviation of test results obtained within a laboratory for a
tions. single material under conditions that may include such ele-
3.1.10.1 DiscussionIt is a measure of the dispersion of the ments as different operators, equipment, and longer time
distribution of test results under repeatability conditions. intervals.
3.1.10.2 DiscussionSimilarly, repeatability variance 3.1.15.1 DiscussionBecause the training of operators, the
and repeatability coefficient of variation could be defined and agreement of different pieces of equipment in the same
used as measures of the dispersion of test results under laboratory and the variation of environmental conditions with
repeatability conditions.In an interlaboratory study, this is longer time intervals depend on the degree of within-laboratory
the pooled standard deviation of test results obtained under control, the within-laboratory standard deviation is likely to
repeatability conditions. vary appreciably from laboratory to laboratory.

2
 E 177 04e1
4. Significance and Use GENERAL CONCEPTS
4.1 Part A of the Blue Book, Form and Style for ASTM
5. Test Method
Standards, requires that all test methods include statements of
precision and bias. This practice discusses these two concepts 5.1 Section 2 of the ASTM Regulations describes a test
and provides guidance for their use in statements about test method as a definitive procedure for the identification, mea-
methods. surement, and evaluation of one or more qualities, character-
4.2 PrecisionA statement of precision allows potential istics, or properties of a material, product, system or service
users of a test method to assess in general terms the test that produces a test result.
methods usefulness with respect to variability in proposed 5.2 In this practice only quantitative test methods that
applications. A statement of precision is not intended to exhibit produce numerical results are considered. Also, the word
values that can be exactly duplicated in every users laboratory. material is used to mean material, product, system or service;
Instead, the statement provides guidelines as to the magnitude the word property is used herein to mean that a quantitative
of variability that can be expected between test results when test result can be obtained that describes a characteristic or a
the method is used in one, or in two or more, reasonably quality, or some other aspect of the material; and test method
competent laboratories. For a discussion of precision, see refers to both the document and the procedure described
Section 18. therein for obtaining a quantitative test result for one property.
4.3 BiasA statement of bias furnishes guidelines on the For a discussion of test result, see Section 9.
relationship between a set of typical test results produced by 5.3 During its development, a test method should be sub-
the test method under specific test conditions and a related set jected to a screening procedure and ruggedness test in order to
of accepted reference values (see Section 19). establish the proper degree of control over factors that may
4.4 AccuracyThe term accuracy, used in earlier editions affect the test results (see Guide E 1169).
of Practice E 177, embraces both precision and bias (see NOTE 1A screening procedure or ruggedness test is a procedure for
Section 20 and Note 4). investigation of the effects of variations in environmental and other
4.5 A Table of Contents is shown below, listing the concepts pertinent factors on the test results obtained from a test in order to
in this standard. determine how control of such factors should be specified in the written
TABLE OF CONTENTS Section
description of the method. For example, temperature of the laboratory or
of a heating device used in the test may have a significant effect in some
GENERAL CONCEPTS cases and less in others. In a screening procedure, deliberate variations in
Test Method 5 temperature would be introduced to establish the limits of significant
Measurement Terminology 6 effect, (1, 2, 3).4
Observation 7
Test Determination 8 5.4 A well-written test method specifies control over such
Test Result 9 factors as the test equipment, the test environment, the quali-
SOURCES OF VARIABILITY fications of the operator (explicitly or implicitly), the prepara-
Experimental Realization of a Test Method 10 tion of test specimens, and the operating procedure for using
Operator 11 the equipment in the test environment to measure some
Apparatus 12
Environment 13 property of the test specimens. The test method will also
Sample 14 specify the number of test specimens required and how
Time 15 measurements on them are to be combined to provide a test
STATISTICAL CONCEPTS result (Section 9), and might also reference a sampling proce-
Accepted Reference Value 16 dure appropriate for the intended use of the method.
Statistical Control 17 5.5 It is necessary that the writers of the test method provide
Precision 18
Bias 19 instructions or requirements for every known outside influence.
Accuracy 20
Variation of Precision and Bias with Material 21 6. Measurement Terminology
Variation of Precision and Bias with Sources of Variability 22
6.1 The following terms have been used to describe both the
COMBINATIONS OF SOURCES OF VARIABILITY measurement process and the partial or complete result of the
Repeatability and Laboratory Bias 23
Other Within-a-Single Laboratory Precisions 24
process: measurement, observation, observed value, test, test
Reproducibility and Bias of the Test Method 25 determination, test result, and others. These terms have often
Range of Materials 26 been used loosely and interchangeably.
METHODS OF EXPRESSING PRECISION AND BIAS
6.2 For clarity, it is necessary to select certain of these terms
Indexes of Precision 27 for specific use. However, the word measurement will be
Preferred Indexes of Precision for ASTM Test Methods 28 used in a generic sense to cover observation (or observed
Preferred Statements of Bias for ASTM Test Methods 29
Elements of a Statement of Precision and Bias 30

STATEMENTS OF PRECISION AND BIAS 4

The boldface numbers in parentheses refer to the list of references at the end of
Examples of Statements of Precision and Bias 31 this standard.

3
 E 177 04e1
value), test determination and test result. The use of the word each of ten specimens and that the ten tensile test determina-
test by itself is discouraged. tions be averaged to get the test result.
6.3 A quantitative test method may have three distinct 9.3.4 In chemical analyses a variety of situations may occur.
stages: (1) the direct measurement or observation of dimen- Thus, in some cases, the method may call for the preparation of
sions or properties; (2) the arithmetical combination of the a single solution from a test unit, and measurement on three
observed values to obtain a single determination; and (3) the aliquots (specimens) of the solution made up to a specified
arithmetical combination of a number of determinations to volume. The average of the three analytical determinations
obtain the test result of the test method. These three stages are would then be called the test result. In other cases of chemical
explained and illustrated in Sections 7-9. analysis, the method may call for two individual test determi-
nations, each one made on a different specimen with recalibra-
7. Observation tion of the measuring instrument for each of the two determi-
7.1 For the purposes of this practice, observation or ob- nations. The average of the two determinations would then be
served value should be interpreted as the most elemental single the test result.
reading or corrected reading obtained in the process of making 9.3.5 In rubber testing, the method may describe not only
a measurement. This statement is a narrower interpretation the shape of the test specimen to be taken from a sheet of
than is given in Terminology E 456 in that the latter applies to rubber, but also the preparation of the sheet, including com-
nonquantitative as well as quantitative test methods. pounding and curing. For example, one rubber test method
7.2 An observation may involve a direct reading (for ex- specifies that four sheets be individually compounded and
ample, a zero-adjusted micrometer reading of the thickness of cured and three specimens tested from each sheet. The test
a test strip at one position along the strip) or it may require the result is then defined as the average of the four medians, each
interpolation of the reading from a calibration curve. median being the middle determination, in the order of
magnitude, of the three values obtained from a sheet.
8. Test Determination 9.3.6 Some test methods, such as those for analytical
8.1 For a quantitative test method, a test determination may chemistry, involve calibration with known standard substances.
be described as (1) the process of calculating from one or more The originally collected test determinations may be subjected
observations a property of a single test specimen, or as (2) the to complex computational and statistical treatment prior to
value obtained from the process. Thus, the test determination being converted into a test result. Such treatment might include
may summarize or combine one or more observations. separation of the analytical response for the substance of
8.2 Examples: interest from the chromatographic absorption data, elimination
8.2.1 The measurement of the density of a test specimen or other treatment of outliers (see Practice E 178) in the data
may involve the separate observation of the mass and the for the known standard substances, and preparation of a
volume of the specimen and the calculation of the ratio calibration curve to determine the test result.
mass/volume. The density calculated from the ratio of one pair 9.4 Precision statements for ASTM test methods are appli-
of mass and volume observations made on one specimen is a cable to comparisons between test results, not test determina-
test determination. tions nor observations, unless specifically and clearly indicated
8.2.2 The determination of the thickness of a test specimen otherwise (see Section 18).
strip may involve averaging micrometer caliper observations
taken at several points along the strip. SOURCES OF VARIABILITY

9. Test Result 10. Experimental Realization of a Test Method

9.1 A test result is the value obtained by carrying out the 10.1 A realization of a test method refers to an actual
complete protocol of the test method once, being either a single application of the test method to produce a test result as
test determination or a specified combination of a number of specified by the test method. The realization involves an
test determinations. interpretation of the written document by a specific test
9.2 In general, a test method describes not only the manner operator, who uses a specific unit and version of the specified
in which each test determination is to be made, but also the test apparatus, in the particular environment of his testing
number of test determinations to be made and how these are to laboratory, to evaluate a specified number of test specimens of
be combined to provide the test result. the material to be tested. Another realization of the test method
9.3 Examples: may involve a change in one or more of the above emphasized
9.3.1 The test method on density might require that the mass experimental factors. The test result obtained by another
and volume observations of a specimen be combined to give a realization of the test method will usually differ from the test
test determination of density (8.2.1) and the test determination result obtained from the first realization. Even when none of
of each of five specimens be averaged to give a test result. the experimental factors is intentionally changed, small
9.3.2 The test method for paper thickness may require that changes usually occur. The outcome of these changes may be
the determination of strip thickness in 8.2.2 be made on ten seen as variability among the test results.
strips and that the ten test determinations be averaged to give 10.2 Each of the above experimental factors and all others,
the test result. known and unknown, that can change the realization of a test
9.3.3 The test method for a tensile strength test of paper may method, are potential sources of variability in test results. Some
specify that a tensile strength determination be performed on of the more common factors are discussed in Sections 11-15.

4
 E 177 04e1
11. Operator 14. Sample (Test Specimens)
11.1 Clarity of Test MethodEvery effort must be made in 14.1 A lot (or shipment) of material must be sampled. Since
preparing an ASTM standard test method to eliminate the it is unlikely that the material is perfectly uniform, sampling
possibility of serious differences in interpretation. One way to variability is another source of variability among test results. In
check clarity is to observe, without comment, a competent some applications, useful interpretation of test results may
laboratory technician, not previously familiar with the method, require the measurement of the sampling error. In interlabora-
apply the draft test method. If the technician has any difficulty, tory evaluation of test methods to determine testing variability,
the draft most likely needs revision. special attention is required in the selection of the material
11.2 Completeness of Test MethodIt is necessary that sample (see 18.4 and Practice E 691) in order to obtain test
technicians, who are generally familiar with the test method or specimens that are as similar as possible. A small residual
similar methods, not read anything into the instructions that is amount of material variability is almost always an inseparable
not explicitly stated therein. Therefore, to ensure minimum component of any estimate of testing variability.
variability due to interpretation, procedural requirements must
be complete. 15. Time
11.2.1 If requirements are not explicitly stated in the test 15.1 Each of the above sources of variability (operator
method (see 5.5), they must be included in the instructions for performance, equipment, environment, test specimens) may
the interlaboratory study (see Practice E 691). change with time; for example, during a period when two or
11.3 Differences in Operator TechniqueEven when opera- more test results are obtained. The longer the period, the less
tors have been trained by the same teacher or supervisor to give likely changes in these sources will remain random (that is, the
practically identical interpretations to the various steps of the more likely systematic effects will enter), thereby increasing
test method, different operators (or even the same operator at the net change and the observed differences in test results.
different times) may still differ in such things as dexterity, These differences will also depend on the degree of control
reaction time, color sensitivity, interpolation in scale reading, exercised within the laboratory over the sources of variability.
and so forth. Unavoidable operator differences are thus one In conducting an interlaboratory evaluation of a test method,
source of variability between test results. The test method the time span over which the measurements are made should be
should be designed and described to minimize the effects of kept as short as reasonably possible (see Sections 23 and 24).
these operator sources of variability.
STATISTICAL CONCEPTS
12. Apparatus
16. Accepted Reference Value
12.1 TolerancesIn order to avoid prohibitive costs, only
16.1 A measurement process is generated by the application
necessary and reasonable manufacturing and maintenance
of a test method. Variability can be introduced unintentionally
tolerances can be specified. The variations allowed by these
into the measurement process through the impact of many
reasonable specification tolerances can be one source of
sources, such as heterogeneity of the material, state of main-
variability between test results from different sets of test
tenance and calibration of equipment, and environmental
equipment.
fluctuations (Sections 10-15). The variability may include
12.2 CalibrationOne of the variables associated with the systematic as well as random components. The systematic
equipment is its state of calibration, including traceability to components may be evaluated (Section 19) if an accepted
national standards. The test method must provide guidance on reference value is available. An accepted reference value is a
the frequency of verification and of partial or complete value that serves as an agreed-upon reference for comparison.
recalibration; that is, for each test determination, each test It may be:
result, once a day, week, etc, or as required in specified (1) a theoretical or established value based on scientific
situations. principles;
(2) an assigned value based on experimental work of some
13. Environment national or international organization such as the U.S. National
13.1 The properties of many materials are sensitive to Institute of Standards and Technology;
temperature, humidity, atmospheric pressure, atmospheric con- (3) a consensus value based on collaborative experimental
taminants, and other environmental factors. The test method work under the auspices of a scientific or engineering group; or
usually specifies the standard environmental conditions for (4) for an isolated application, when no value for (1), (2),
testing. However, since these factors cannot be controlled or (3) exists, an agreed upon value obtained using an accepted
perfectly within and between laboratories, a test method must reference method.
be able to cope with a reasonable amount of variability that NOTE 2When the accepted reference value is a theoretical value, it is
inevitably occurs even though measurement and adjustment for sometimes referred to as the true value, but this usage is not recom-
the environmental variation have been used to obtain control mended.
(see 17.2). Thus, the method must be both robust to the
differences between laboratories and require a sufficient num- 17. Statistical Control
ber of test determinations to minimize the effect of within- 17.1 A process is in a state of statistical control if the
laboratory variability. variations between the observed test results from it can be

5
 E 177 04e1
attributed to a constant system of chance causes. By chance applied. The test results of two different processes expressed in
causes is meant unknown factors, generally numerous and the same units may be statistically compared as to precision, so
individually of small magnitude, that contribute to variation, that one process may be described as more (or less) precise
but that are not readily detectable or identifiable. than the other.
17.2 The measurement process is in a state of statistical 18.3 The precision of the measurement process will depend
control when the test results obtained vary in a predictable on what sources (Sections 10-15) of variability are purposely
manner, showing no unassignable trends, cycles, abrupt included and may also depend on the test level (see Section
changes, excess scatter, or other unpredictable variations as 21). An estimate of precision can be made and interpreted only
determined by application of appropriate statistical methods. if the experimental situation (prescribed like conditions) under
The ensurance of a state of statistical control is not a simple which the test results are obtained is carefully described. There
matter (4), but may be helped by the use of control charts (see is no such thing as the precision of a test method; a separate
Part 3, STP 15D) (5, 6). precision statement will apply to each combination of sources
17.2.1 If the set of test results to be considered in terms of of variability. The precision of a particular individual test result
statistical control is obtained in different laboratories, it may be depends on the prescribed conditions for which it is considered
possible to view the laboratories as a sample of all qualified a random selection. For example, will it be compared with
laboratories that are likely to use the given test method, or as other results obtained within the laboratory or with results
a set comprising a special category of such laboratories, and obtained in other laboratories? No valid inferences on the
that the differences between the laboratories represent random precision of a test method or a test result can be drawn from an
variability. Qualified may mean, for example, laboratories individual test result.
that have used this test method for a year or more. 18.4 In order to minimize the effect of material variability in
17.3 The presence of outliers (Practice E 178) may be evaluating the precision of a test method, it is desirable to
evidence of a lack of statistical control in the production select a relatively uniform material for each of several test
process or in the measurement process. It is quite proper to levels (magnitudes) chosen for the property being tested (see
discard outliers for which a physical explanation is known. Practice E 691 for further information).
Discarding outliers in the measurement process on the basis of
statistical evidence alone may yield biased results since one 19. Bias
can truly measure the value of the property of interest only if
19.1 The bias of a measurement process is a generic concept
the measurement process is in control. The presence of one or
related to a consistent or systematic difference between a set of
more outliers may indicate a weakness in the test method or its
test results from the process and an accepted reference value of
documentation.
the property being measured. The measuring process must be
17.4 The discussion in succeeding sections assumes that the
in a state of statistical control; otherwise the bias of the process
measurement process is in a state of statistical control for some
has no meaning. In determining the bias, the effect of the
specified set of conditions. If measurements are all to be made
imprecision is averaged out by taking the average of a very
in a given laboratory, for example, any systematic deviation
large set of test results. This average minus the accepted
from the expected value pertinent to that laboratory will show
reference value is an estimate of the bias of the process (test
up as a bias for measurements made under the prescribed
method). Therefore, when an accepted reference value is not
conditions (see Section 19).
available, the bias cannot be established.
18. Precision 19.2 The magnitude of the bias may depend on what sources
18.1 The precision of a measurement process, and hence the of variability are included, and may also vary with the test level
stated precision of the test method from which the process is and the nature of the material (see Section 21).
generated, is a generic concept related to the closeness of 19.3 When evaluating the bias of a test method, it is usually
agreement between test results obtained under prescribed like advisable to minimize the effect of the random component of
conditions from the measurement process being evaluated. The the measurement error by using at each test level the average
measurement process must be in a state of statistical control; of many (30 or more) test results, measured independently, for
else the precision of the process has no meaning. The greater each of several relatively uniform materials, the reference
the dispersion or scatter of the test results, the poorer the values for which have been established by one of the alterna-
precision. (It is assumed that the least count of the scale of the tives in 16.1 (see 23.3 and 25.3).
test apparatus is not so poor as to result in absolute agreement 19.4 If the bias of a test method is known, an adjustment for
among observations and hence among test results.) Measures the bias may be incorporated in the test method in the section
of dispersion, usually used in statements about precision, are, on calculation or in a calibration curve and then the method
in fact, direct measures of imprecision. Although it may be would be without bias.
stated quantitatively as the reciprocal of the standard deviation, 19.5 The concept of bias may also be used to describe the
precision is usually expressed as the standard deviation or systematic difference between two operators, two test sites (see
some multiple of the standard deviation (see Section 27). 23.3), two seasons of the year, two test methods, and so forth.
18.2 A measurement process may be described as precise Such bias is not a direct property of the test method, unless one
when its test results are in a state of statistical control and their of the test sites or test methods provides the accepted reference
dispersion is small enough to meet the requirements of the value. The effect of such bias may be reflected in the measured
testing situations in which the measurement process will be reproducibility of the test method.

6
 E 177 04e1
20. Accuracy COMBINATIONS OF SOURCES OF VARIABILITY
(TYPES OF PRECISION AND BIAS)
20.1 Accuracy is a generic concept of exactness related to
the closeness of agreement between the average of one or more
23. Repeatability and Laboratory Bias
test results and an accepted reference value. Unless otherwise
qualified, the use of the word accuracy by itself is to be 23.1 Within-Laboratory PrecisionInformation about a
interpreted as the accuracy of a test result. The accuracy of a frequently used within-laboratory precision, sometimes called
test result is the closeness of agreement between the test result single-operator-day-apparatus precision, can be obtained from
and the accepted reference value. It depends on both the at least the three experimental situations described in 23.1.1-
imprecision and the bias of the test method. 23.1.3, the last situation being most reliable; that is, the
estimate of this precision is improved progressively by pooling
20.2 There are two schools of thought on defining the additional information.
accuracy of a measuring process (5, 7). In either case, the
measurement process must be in a state of statistical control, NOTE 3If the test method requires a series of steps, the single-
otherwise the accuracy of the process has no meaning: operator-equipment requirement means that for a particular step the same
combination of operator and equipment is used for every test result and on
20.2.1 The closeness of agreement between the accepted every material. Thus one operator may prepare the test specimens, a
reference value and the average of a large set of test results second measure the dimensions and a third measure the breaking force.
obtained by repeated applications of the test method, prefer- The single-day requirement means that the test results, at least for a
particular material are obtained in the shortest practical period of time,
ably in many laboratories.
whether this be a fraction of a day or several days.
20.2.2 The closeness of agreement between the accepted
reference value and the individual test results (8, 9). 23.1.1 Precision From an Experiment Involving One Op-
erator, Day and ApparatusA single, well-trained operator
20.3 In 20.2.1 the imprecision is largely eliminated by the
using one set of equipment obtains two or more test results in
use of a large number of measurements and the accuracy of the a short period of time during which neither the equipment nor
measuring process depends only on bias. In 20.2.2 the impre- the environment is likely to change appreciably. The variability
cision is not eliminated and the accuracy depends on both bias is due primarily to small changes in equipment, calibration,
and imprecision. In order to avoid confusion resulting from use reagents, environment, and operators procedure, and possibly
of the word accuracy, only the terms precision and bias to some heterogeneity in the material tested. The last is kept
should be used as descriptors of ASTM test methods. small by use of test specimens from a reasonably uniform lot
of material. The precision estimate for this operator, day, and
21. Variation of Precision and Bias with Material equipment is determined from the variability of the test results.
21.1 A test method is intended to cover a class of materials. In this situation and the other experiments listed below, all
potential sources of variability must be carefully controlled
Any one material within the class differs from any other in the
within the tolerances specified in the test method.
following two basic ways: the level of the property that is being
23.1.2 Precision from Repeated Experiments Within a
measured; and the matrix of the material. The matrix is the
LaboratoryIn order to get an expression of precision that
totality of all properties, other than the level of the property to
applies to any operator and day with a specific set of equipment
be measured, that can have an effect on the measured value.
at a given laboratory, the experiment of 23.1.1 must be
Thus the precision and the bias of the test method may be repeated on different days by the same and different operators.
functions of the property level and of the material matrix. Then the precision estimates, obtained as in 23.1.1, for each
21.2 In some cases, a test method may be intended to be operator-day combination must be suitably combined or pooled
applied to more than one class of materials. If so, it may be to obtain an estimate of single-operator-day precision that
advisable to provide separate statements of precision for each applies to this laboratory and equipment. If the laboratory has
class (see 31.3). several sets of equipment for this test method, the experiment
may be enlarged to include tests on each set of equipment and
22. Variation of Precision and Bias with Sources of the test results pooled in order to obtain an overall single-
Variability operator-day-equipment precision for that laboratory.
23.1.3 Precision from Within-Laboratory Experiments in
22.1 The precision and bias of test results obtained by
Several LaboratoriesIn order to obtain an estimate of
repeated applications of a test method depend upon what within-laboratory precision that is characteristic of the test
combinations of the sources of variability (Sections 10-15) method and may reasonably be applied to any laboratory, the
affect the variability of the test results. For example, test results whole within-laboratory experiment of 23.1.2 could be re-
obtained by all possible operators within one laboratory using peated in a number of laboratories. Alternatively, this desired
one set of test apparatus would have a bias based in part on that broadly-applicable estimate may be obtained by pooling
laboratorys apparatus and environment and a precision that within-laboratory information from only one operator-day-
would depend in part on the quality of training and supervision equipment combination carried out in each of a number of
of operators in that laboratory. Many combinations of sources laboratories. Although only one operator, one day, and one set
of variability are possible. Some of the combinations used by of equipment are combined in each laboratory, the use of many
ASTM committees are described in Sections 23-25. laboratories, as in an interlaboratory study such as described in

7
 E 177 04e1
Practice E 691, provides an evaluation based on many opera- information on multi-operator precision is needed by a labo-
tors, many days and many units of equipment. This abbreviated ratory, it should be studied by that laboratory.
approach, only one operator-day-equipment combination in
each laboratory, is based on the assumption that this estimate of 25. Reproducibility and Bias of the Test Method
within-laboratory precision does not change, or should not be 25.1 Between-Laboratory PrecisionEach of several labo-
expected to change, significantly from laboratory to laboratory. ratories, each with its own operator, apparatus, and environ-
Consequently, this measure of precision can be treated as a mental conditions, obtains a test result on randomly-selected
characteristic of the test method. This pooled within-laboratory specimens from the same reasonably-uniform sample of mate-
precision is called the repeatability of the test method. rial. The variability of the test results may be used to calculate
23.2 Repeatability ConditionsWhile other conditions the between-laboratory precision, which, when based on a
(Section 24) have sometimes been used for obtaining repeated single test result from each laboratory, is also called the
test results in the determination of repeatability, the preferred reproducibility of the test method. The laboratories being
conditions (illustrated above in 23.1-23.1.3) are those under compared in order to obtain the between-laboratory reproduc-
which test results are obtained with the same test method in the ibility of the test method should be independent of each other.
same laboratory, by the same operator with the same equip- Independent means that the laboratories should not be under
ment, in the shortest practical period of time, using test units or the same supervisory control, nor should they have worked
test specimens (see Practice E 691, 10.3) taken at random, from together to resolve differences. The value found for the
a single quantity of material that is as nearly homogeneous as between-laboratory precision will depend on the choice of
possible. For meaning of same operator, same equipment laboratories and the selection of operators and apparatus within
and shortest practical period of time, see Note 3 above. each laboratory.
23.3 RepeatabilityThe closeness of agreement between 25.1.1 The precision within a single laboratory or facility
test results obtained under repeatability conditions. having multiple test stations will depend largely on the degree
23.4 Bias of a Particular Laboratory, relative to the other of supervision provided. If information on this precision is
laboratories may be calculated by averaging test values ob- required, the laboratory should run its own internal study,
tained as described in 23.1.2 for that laboratory and comparing possibly using Practice E 691, with each station treated as a
the result with the average of all test values obtained as laboratory. The precision determined (that is, between station
described in 23.1.3. The bias of the test method may be reproducibility), can be expected to be somewhat better than
calculated by comparing the latter average with the accepted the reproducibility of the test method, depending on the degree
reference value (Section 16), or it may be determined as of common supervision of the test stations.
described in 25.3. Once the bias is known, the method should 25.2 Reproducibility, as used in 25.1 and 25.1.1, is a general
be modified to correct for it (see 19.5). term for a measure of precision applicable to the variability
between single test results obtained in different laboratories
24. Other Within-A-Single Laboratory Precisions using test specimens taken at random from a single sample of
24.1 Single-Operator-Apparatus, Multi-Day PrecisionA material. This use of the word reproducibility is narrower
single operator using one set of equipment obtains replicate test then that defined in Terminology E 456 because it assumes the
results as in Section 23, but one on each of two or more days. simpler interlaboratory study of 23.1.3 and Practice E 691
Since the time interval is greater than in Section 23, there is a where only one operator-day-apparatus combination is in-
greater chance that the equipment (including its calibration) volved in each laboratory.
and the environment may change, and that the change will 25.3 Bias of Test MethodThe bias of the test method, for
depend on the degree of control or supervision maintained by a specific material, may be calculated by comparing the
the laboratory over these factors. Therefore, the precision average of all the test results obtained in 25.1 for that material
calculated in this between-day within-laboratory situation, may with the accepted reference value (see Section 16) for that
vary appreciably from laboratory to laboratory and often material. If no accepted reference value is available, bias
cannot be regarded as a universal parameter of the test method. cannot be calculated (however, see 29.2). For a valid determi-
While this multi-day precision has been called repeatability nation of bias, the results of the test method must indicate a
by some ASTM committees, it is better to reserve the term for state of statistical control (see Section 17).
the precision estimate described in 23.1.3, which is more likely
to be an estimate of a universal characteristic of the test 26. Range of Materials
method. If information on multi-day precision is needed by a 26.1 The estimates of precision and bias described in
laboratory, it should be studied in that laboratory, since the Sections 23-25 are based on test results from a material at one
estimate may vary widely from laboratory to laboratory. level of the property of interest. The experiments should be
24.2 Multi-Operator, Single-Day-Apparatus Precision extended to other related materials yielding test results at other
Each of several operators in one laboratory using the same set test levels. Related materials are materials that may have
of equipment obtains a test result. Since the operator effect may similar matrixes of other properties (see Section 21) and are
depend on the degree of training and supervision exercised in likely to be compared by means of the test method.
the laboratory, the precision among test results (between 26.2 Precision and bias may be constants or simple func-
operators within laboratory) may vary widely from laboratory tions of the test level or they may depend so appreciably on the
to laboratory, and therefore may not be regarded as a universal matrix of other properties of the materials that the test method
parameter of the test method (see Note in example in 31.7). If will have to be modified to take into account these other,

8
 E 177 04e1
possibly-interfering, properties before reasonable and consis- 27.3.4.2 It may be thought that the use of the multiplier
tent values for precision and bias can be obtained. 1.960 (or approximately 2.0) in 27.3.2 and 27.3.3 requires that
the sample standard deviation (s) be assumed to be equal to the
METHODS OF EXPRESSING PRECISION AND population (or true) standard deviation (s). No within or
BIAS between-laboratory study will yield a standard deviation (s)
exactly equal to the true standard deviation (s), and few will
27. Indexes of Precision come close unless at least 30 laboratories are included in the
27.1 GeneralPrecision may be stated in terms of an index study. No multiplier for s will ensure an actual limit of exactly
consisting of some positive value, a. The index is expressed in 95 %. The use of the multiplier t, (Students t), instead of the
the same units as those of the test result, or as a percent of the multiplier, 1.960 does not remedy the situation. In order to
test result. The numerical value of a will be smaller when the resolve this problem, a range of probabilities around 95 % must
individual test results from repeated applications of the test be accepted as defining the 95 % limit. For appropriate
method are more closely grouped. The larger the index, the less choices of the defining range, the multiplier 1.960 (or 2.0) may
precise the measurement process. A test method has a separate still be used. It has been shown that 1.960 is the best choice for
index of precision for each type of precision (see Sections achieving the desired (but approximate) 95 % coverage (10).
22-25) and this index may vary in a systematic way with the The multiplier is independent of the number of test results in
test level or it may vary from material to material even at the the within-laboratory study or the number of laboratories in the
same test level. study for between-laboratory precision. However, a within- or
between-laboratory study must be of reasonably large size in
27.2 BasisThe usual source of the index of precision is
order to provide reliable information on which to base a
the sample estimate of the standard deviation, (denoted by the
precision statement.
symbol s), of a random set of test results for that type of
27.3.5 Indexes in PercentIn some instances (see 28.5)
precision (for example, from an interlaboratory study such as
there may be some advantage in expressing the precision index
Practice E 691), where standard deviation has its usual mean-
as a percentage of the average test result; that is, percent
ing (for example, see Terminology E 456). The number of test
coefficient of variation (CV %). The notation may then be
results in the set should be sufficiently large (at least 30) so that
(CV %), (2CV %), (d2CV %), etc.
the sample standard deviation(s) computed from the randomly-
27.3.6 Other IndexesFor some applications, limits based
selected set be a good approximation to the standard deviation
on 95 % probability are not adequate. Basic multipliers other
of the population of all test results (denoted by the symbol s)
than 1.960 (or about 2.0) may be used, yielding probabilities
that could be obtained for that type of precision. See Practice
other than approximately 0.95. As discussed below, however,
E 691 for an example of the design of an interlaboratory study
the (d2s) = (2.8 s) and (d2CV %) = (2.8 CV %) indexes are
to determine within-laboratory and between-laboratory stan-
recommended, unless there is a special need.
dard deviations, also called repeatability and reproducibility
standard deviations.
28. Preferred Indexes of Precision for ASTM Test
27.3 Possible Indexes of Precision:
Methods
27.3.1 Standard Deviation(s)See 27.2.
27.3.2 Two Standard Deviation Limits (2s) 28.1 Preferred Types of Precision and Preferred Indexes
Approximately 95 % of individual test results from laborato- The types of precision described in 23.1.3 and 25.1, namely,
ries similar to those in an interlaboratory study can be expected repeatability and reproducibility, are the preferred types of
to differ in absolute value from their average value by less than precision statements for ASTM test methods. The preferred
1.960 s (about 2.0 s). index for each of these types is the 95 % limit on the difference
27.3.3 Difference Two Standard Deviation Limit (d2s) between two test results (see 27.3.3), namely, 2.8 s or 2.8
Approximately 95 % of all pairs of test results from laborato- CV %. Also the corresponding standard deviation (s) or percent
ries similar to those in the study can be expected to differ in coefficient of variation (CV %) shall be indicated.
absolute value by less than 1.960 =2 s (about =2 s ) = 2.77 28.2 Recommended Terminology for Preferred Indexes:
s (or about 2.8 s). This index is also known as the 95 % limit r 5 95 % repeatability limit (1)
on the difference between two test results. For the two cases R 5 95 % reproducibility limit (1)
described in Sections 23 and 25, these limits are known as the or, to help prevent confusion between r and R, use:
repeatability and reproducibility limits.
r 5 95 % repeatability limit ~within a laboratory! (2)
27.3.4 Multiplier for 95 % Limit:
27.3.4.1 The multiplier 1.960 or 2.0 used in 27.3.2 and R 5 95 % reproducibility limit ~between laboratories! (2)
27.3.3 assumes an underlying normal distribution for the test Similarly, the recommended terminology for the correspond-
results being compared. For methods in which the average of ing standard deviations is:
several test determinations is reported as a single test result, the sr 5 repeatability standard deviation ~within a laboratory! (3)
assumption of normality is usually reasonable, even for skewed
sR 5 reproducibility standard deviation ~between laboratories! (3)
or bimodal distributions. When normality cannot be assumed,
it is usually satisfactory to continue to use the multiplier 2.0 but and for the coefficients of variation:
recognize that the actual probability limit will differ somewhat CV %r 5 repeatability coefficient of variation in percent ~within a
from the nominal 95 % limit. laboratory! (4)

9
 E 177 04e1
CV %R 5 reproducibility coefficient of variation in percent ~between The statement on bias should then state how this correction is
laboratories! (4) provided for in the test method.
where: 29.2 If the bias of a test method, or the uncorrected balance
r = 1.960 =2 sr = 2.8 sr or r = 1.960 =2 CV %r = 2.8 of the bias, is not known because there is no accepted reference
CV %r value (see 25.3), but upper and lower bounds can be estimated
R = 1.960 =2 sR = 2.8 sR or R = 1.960 =2 CV %R = 2.8 by a theoretical analysis of potential systematic errors, credible
CV %R bounds for this uncorrectable balance of the bias should be
depending on how the indexes vary with the test level (see given in the bias statement (see example Ex 9 of 31.9). (9)
28.5). For other than the preferred types, the more general
NOTE 4No formula for combining the precision and the bias of a test
terminology 95 % limit may be used with a description of the method into a single numerical value of accuracy is likely to be useful.
sources of variability; for example: Instead separate statements of precision and bias should be presented. The
95 % limit (operator-to-operator, within-laboratory) and similarly value may then be used jointly in any specific application of the test
for the corresponding standard deviation: method.
operator-to-operator within-laboratory standard deviation.
30. Elements of a Statement of Precision and Bias
28.3 Whenever the general terms repeatability and re-
producibility or the more specific terminology repeatability 30.1 The precision and bias section of a test method should
limit and reproducibility limit are stated with numerical include, as a minimum, the elements specified in 30.2-30.5 and
values, users will have to assume that the 95 % limits are in 30.7:
intended, unless otherwise specified. 30.2 A brief description of the interlaboratory test program
28.4 Quantitative estimates of repeatability and reproduc- on which the statement is based, including (1) what materials
ibility may be obtained from an interlaboratory study con- were tested, (2) number of laboratories, (3) number of test
ducted as directed in Practice E 691. results per laboratory per material, and the (4) interlaboratory
28.5 Variation of Index With Test LevelThe choice be- practice (usually Practice E 691) followed in the design of the
tween 2.8 s and 2.8 CV % and the form for the statement of the study and analysis of the data. This section should give the
precision indexes depends upon how the indexes vary with the ASTM Research Report number for the interlaboratory data
test level. and analysis.
28.5.1 If a 2.8 s index is approximately constant throughout 30.3 A description of any deviation from complete adher-
the test range, then the 2.8 s index is recommended. Express ence to the test method for each test result, such as preparation
the index in the units of the measured property. in one laboratory of the cured test sheets and distribution
28.5.2 If a 2.8 s index is approximately proportional to the thereof to the participating laboratories, when curing is a
test level, then use the 2.8 CV % index. Express the index in specified part of the test method.
percentage of the test level. 30.4 The number of test determinations and their combina-
28.5.3 In either case, express the index as a single average tion to form a test result, if not clearly defined in the body of
(or pooled) number followed parenthetically by the actual the test method.
range of the index values (highest and lowest) encountered in
30.5 A statement of the precision between test results
the interlaboratory study.
expressed in terms of the 95 % repeatability limit and the 95 %
28.5.4 If a 2.8 s index is neither approximately constant nor
reproducibility limit (see 28.2), including any variation of these
approximately proportional to the test level, plot the index
statistics with test level or material (see 28.5 and section 28.6).
versus the test level to determine how they are related. If the
Report the repeatability and reproducibility standard deviations
index varies systematically with the test level, express the
(or percent coefficients of variation) among test results as
index by a combination of 2.8 s and 2.8 CV % (see example
indicated in 28.2. Finally, state that repeatability and reproduc-
31.3), by a simple formula, or by a plot. If the index varies in
ibility are used as directed in ASTM Practice E 177.
no systematic way with the level, but jumps from material to
material (perhaps because some materials are inherently more 30.6 If precision under additional conditions (for example,
variable than others), express the index by a table (see 31.6) or operator-to-operator or day-to-day) has been determined, re-
by a single compromise value selected by judgment. Carefully port the number of operators or days per laboratory. Include a
describe each material in the table. The jumping may be due to careful description of the additional conditions, and the preci-
interfering properties in the material matrixes (Section 21) and sion values obtained, using such terminology as 95 % limit
the description may eventually allow identification of the (operator-to-operator within laboratory).
cause. 30.7 A statement concerning what is known about bias,
including how the method has been modified to adjust for what
29. Preferred Statements of Bias for ASTM Test Methods is known about bias and that it is now without known bias. If
29.1 Some information may be available concerning the the value of the property being measured can be defined only
bias or part of the bias of a test method as determined from an in terms of the test method, state this and whether the method
interlaboratory study (25.3 and 23.3) or from known effects of is generally accepted as a reference method. If an estimate of
environmental or other deviations as determined in ruggedness the maximum bias of the method can be made on theoretical
tests (see 5.3). An adjustment for what is known about the bias grounds (for example, by examining the maximum probable
can be incorporated in the calculations or calibration curves. contributions of various steps in the procedure to the total

10
 E 177 04e1
bias), then describe these grounds in this section. Give the 31.3 If a sufficient number of different materials to cover the
ASTM Research Report number on the theoretical or experi- test range are included in the interlaboratory study (6 or more
mental study of bias. in accordance with Practice E 691), then the approximate
variation in precision with test level may be determined. Since
STATEMENTS OF PRECISION AND BIAS
two distinctly separate classes of material are tested by the
31. Statements of Precision and Bias method shown in illustrative example Ex.3, two separate
31.1 Example Statements of Precision and BiasIn the interlaboratory studies were made. In the first study, the
simplest case, the statement will appear essentially as shown in repeatability was found to be essentially proportional to the test
illustrative example Ex.1. Ex.1 is a simplified example. Nor- value (with minor variation from material to material as
mally, at least six laboratories and at least three materials shown), whereas the reproducibility had a more complex linear
should be included in the study in accordance with Practice relationship (that is, a constant as well as a proportional term).
E 691. (No general conclusions about the test method can be In the second study, the repeatability and the reproducibility
considered valid from so few materials and laboratories.) were each found to be proportional to the test value.
Ex.1 Precision and Bias Ex.3 Precision
Ex.1.1 Interlaboratory Test ProgramAn interlaboratory study Coarse-fiber materials
of the permanent deformation of elastomeric yarns was run in 1969.
Each of two laboratories tested five randomly drawn test specimens Test range 30 to 150 g
from each of three materials. The design of the experiment, similar 95 % repeatability limit (within laboratory) 7 % (6 to 8.5 %) of the test result
to that of Practice E 691, and a within-between analysis of the data 95 % reproducibility limit (between labora- 2 g + 10 % (8 to 12 %) of the test
are given in ASTM Research Report No. XXXX. tories) result
Ex.1.2 Test ResultThe precision information given below for Well beaten (fine-fiber) materials
average permanent deformation in percentage points at 100-min
relaxation time is for the comparison of Test range 20 to 75 g
two test results, each of which is the average of five test determinations. 95 % repeatability limit (within laboratory) 4 % (3.5 to 5 %) of the test result
Ex.1.3 Precision: 95 % reproducibility limit (between labora- 7 % (5 to 8 %) of the test result
95 % repeatability limit (within laboratory) 0.8 % tories)
95 % reproducibility limit (between labora- 2.9 %
tories) Ex.3.1 The values shown above for the limits are the average
The above terms (repeatability limit and reproducibility limit) are used as speci- and range) in each case as found in separate interlaboratory studies
fied in Practice E 177. The respective standard deviations among test results, for the coarse and fine-fiber materials. The terms repeatability limit
related to the above numbers by the factor 2.8, are: and reproducibility limit are used as specified in Practice E 177. The
repeatability standard deviation = 0.3 % respective standard deviations among test results may be obtained
reproducibility standard deviation = 1.0 %. by dividing the above limit values by 2.8.
Ex.1.4 BiasThis method has no bias because permanent deformation of elas-
tomeric yarns is defined in terms of this method. 31.4 Precision information can often be obtained from
31.2 The illustrative example Ex.2 is another simplified studies made for other purposes. Example below illustrates this
example in which only two materials have been used but with approach and also illustrates another way of showing variation
the required minimum number (six) of participating laborato- from material to material.
ries: Ex.4 Precision
Ex.4.1 Interlaboratory Test ProgramThe information given
Ex.2 Precision and Bias below is based on data obtained in the TAPPI Collaborative (11)
Ex.2.1 Interlaboratory Test ProgramAn interlaboratory study Reference Program for self-evaluation of laboratories, Reports 25
was run in which randomly drawn test specimens of two materials through 51 (Aug. 1973 through Jan. 1978). Each report covers two
(kraft envelope paper and wove envelope paper) were tested for materials with each of approximately 16 laboratories testing 5
tearing strength in each of six laboratories, with each laboratory specimens of each material.
testing two sets of five specimens of each material. Except for the use Ex.4.2 Test ResultThe precision information given below has
of only two materials, Practice E 691 was followed for the design been calculated for the comparison of two test results, each of which
and analysis of the data, the details are given in ASTM Research is the average of 10 test determinations.
Report No. XXXY. Ex.4.3 95 % Repeatability Limit (within laboratory)The repe-
Ex.2.2 Test ResultThe precision information given below in the atability is 5.4 % of the test result. For the different materials the
units of measurement (grams) is for the comparison of two test repeatability ranged from 3.7 to 9.6 %. The range of the central 90
results, each of which is the average of five test determinations: percent of the repeatability values was 3.9 to 8.7 %.
Ex.2.3 Precision: Ex.4.4 95 % Reproducibility Limit (between laboratories)The
Material A Material B reproducibility is 19.2 % of the test result. For the different
materials the range of all of the calculations of reproducibility was
Average Test Value 45 gf 100 gf 6.4 to 45.4 %. The range of the central 90 percent of the calculations
95 % repeatability limit (within laboratory) 3 gf 7 gf was 12.2 to 25.5 %.
95 % reproducibility limit (between laboratories) 6 gf 12 gf Ex.4.5 Definitions and Standard DeviationsThe above terms
The above terms (repeatability limit and reproducibility limit) are repeatability limit and reproducibility limit are used as specified in
used as specified in Practice E 177. The respective standard devia- Practice E 177. The respective percent coefficients of variation
ions among test results may be obtained by dividing the above limit among test results may be obtained by dividing the above numbers
values by 2.8. by 2.8.

Ex.2.4 BiasThe original draft of this abbreviated method was 31.5 Precision is often constant for low test values and
experimentally compared in one laboratory with the appropriate proportional for higher test values, as shown in the following
reference method (ASTM DXXXX) and was found to give results
approximately 10 % high, as theoretical considerations would sug- example:
est (See ASTM Research Report No. XXXW). An adjustment for
this bias is now made in Section XX on calculations, so that the
final result is now without known bias.

11
 E 177 04e1
Ex.5 Precision Ex.7 Precision
Test range 0.010 to 1200 mm Average test value 100 g
95 % repeatability limit (within labora- 0.002 mm or 2.5 % of the average, 95 % repeatability limit (within a labo- 7 % (6 to 8 %) of the test result
tory) whichever is larger ratory)
95 % reproducibility limit (between lab- 0.005 mm or 4.2 % of the average, 95 % reproducibility limit (between lab- 15 % (13 to 16 %) of the test result
oratories) whichever is larger oratories)
95 % limit (operator-to-operator, within 6 % to 15 % of the test result
The above terms repeatability limit and reproducibility limit laboratory)
are used as specified in Practice E 177. The respective standard Ex.7.1 The values shown above for the limits are, in each case,
the average (and range) found in the interlaboratory study. The
deviations and percent coefficients of variation among test terms, repeatability, reproducibility and operator-to-operator limit,
results may be obtained by dividing the above limit values by are used as specified in this practice. The respective standard devia-
ions may be obtained by dividing the above limit values by 2.8.
2.8.
31.6 A table may be used especially if the precision indexes NOTE 5Since the lower value for the operator-to-operator effect was
vary irregularly from material to material. Note in the follow- obtained in a laboratory that has a continuing training program for its
ing example that the materials have been arranged in increasing operators, it appears that the operator-to-operator effect may be reduced by
order of test value: training. Furthermore, since the upper value for the operator-to-operator
effect in some laboratories is as high as the reproducibility between
Ex.6A Precision
laboratories, it is possible that reproducibility also may be improved by
Repro-
Repeata- better operator training.
duci- Repro-
Glucose in bility Repeat-
bility duci- 31.8 An example of a bias statement when bias has been
Material Serum, Stand- ability
Stand- bility
Average ard Devi-
ard De-
Limit
Limit removed through comparison with a reference method is given
ation
viation in 31.2 and Ex.2.4. A similar statement would apply for any
A 41.518 1.063 1.063 2.98 2.98 accepted reference value, for example, from an accepted
B 79.680 1.495 1.580 4.19 4.42 reference material. If bias depends on other properties of the
C 134.726 1.543 2.148 4.33 6.02
D 194.717 2.625 3.366 7.35 9.42
material, a statement such as the following might be used:
E 294.492 3.935 4.192 11.02 11.74 Ex.8 Bias
Ex.8.1. BiasA ruggedness study (ASTM Research Report
Ex.6A.1 Interlaboratory Test ProgramAn interlaboratory
No. XXXZ) showed that test results are temperature dependent,
study of glucose in serum was conducted in accordance with
with the dependence varying with the type of material. Therefore, if
Practice E 691 in eight laboratories with five materials, with each
the test temperature cannot be maintained within the specified
laboratory obtaining three test results for each material. See ASTM
limits, determine the temperature dependence for the specific
Research Report No. XXXX.
material being tested and correct test results accordingly.
Ex.6A.2 The terms repeatability limit and reproducibility limit
in Ex.6A are used as specified in Practice E 177. 31.9 A maximum value for the bias of a test method may be
Ex.6B Precision estimated by an analysis of the effect of apparatus and
Repeat- Reproduc-
Pentosans ability ibility Repeat- Repro-
procedural tolerances on the test results, as illustrated below:
Material in Pulp, Stand- Stand- ability ducibility Ex.9 Bias
Average ard De- ard De- Limit Limit Ex.9.1. BiasError analysis shows that the absolute value of the
viation viation maximum systematic error that could result from instrument and
other tolerances specified in the test method is 3.2 % of the test
A 0.405 0.015 0.114 0.04 0.32
result.
B 0.884 0.032 0.052 0.09 0.14 31.10 Even when a quantitative statement on bias is not
C 1.128 0.143 0.196 0.40 0.55
D 1.269 0.038 0.074 0.11 0.21 possible, it is helpful to the user of the method to know that the
E 1.981 0.040 0.063 0.11 0.18 developers of the method have considered the possibility of
F 4.181 0.032 0.209 0.09 0.58 bias. In such cases, a statement on bias based on one of the
G 5.184 0.133 0.243 0.37 0.68
H 10.401 0.194 0.585 0.54 1.64 following examples may be used:
I 16.361 0.216 1.104 0.60 3.09 Ex.10.1 BiasThis method has no bias because (insert the name
Ex.6B.1 Interlaboratory Test ProgramAn interlaboratory of the property) is defined only in terms of this test method.
study of pentosans in pulp was conducted in accordance with Ex.10.2 BiasSince there is no accepted reference material,
Practice E 691 with seven participating laboratories each obtaining method, or laboratory suitable for determining the bias for the
three test results of each of nine materials. See ASTM Research procedure in this test method for measuring (insert the name of the
Report No. YYYY. property), no statement on bias is being made.
Ex.6B.2 The terms repeatability limit and reproducibility limit in Ex.10.3 BiasNo justifiable statement can be made on the bias
Ex.6B are used as specified in Practice E 177. of the procedure in this test method for measuring (insert the name
of the property) because (insert the reason).
31.7 If multi-operator precision (23.1) as well as repeatabil- 32. Keywords
ity and reproducibility has been evaluated, its variation among
32.1 accepted reference value; accuracy; bias; interlabora-
laboratories may be shown as in illustrative example Ex.7.
tory study; precision; precision conditions; repeatability; repro-
ducibility; standard deviation

12
 E 177 04e1
REFERENCES

(1) Youden, W. J., Experimental Design and ASTM Committee, Mate- (7) Murphy, R. B., On the Meaning of Precision and Accuracy,
rials Research and Standards, ASTM, November 1961, pp. 862867. Materials Research and Standards, ASTM, April 1961, pp. 264267.
(2) Wernimont, Grant, Ruggedness Evaluation of Test Procedures, (8) Eisenhart, Churchill, The Reliability of Measured ValuesPart I:
Standardization News, Vol. 5, No. 3, March 1977, pp. 1316. Fundamental Concepts, Photogrammetric Engineering, June 1952,
(3) Youden, W. J., Statistical Techniques for Collaborative Tests, Associa- pp. 542561.
tion of Official Analytical Chemists, Washington, DC, 1967, pp.
2932. (9) Eisenhart, Churchill, Realistic Evaluation of the Precision and Accu-
(4) Shewhart, Walter A., Statistical Method from the Viewpoint of Quality racy of Instrument Calibration Systems, Journal of Research of the
Control, The Graduate School of the Department of Agricultural, National Bureau of Standards, 67C, 1963, pp. 161187.
Washington, DC, 1939. (10) Mandel, John and Lashof, T. W., The Nature of Repeatability and
(5) Mandel, John, The Statistical Analysis of Experimental Data, Reproducibility, Journal of Quality Technology, Vol 19, No. 1,
Interscience-Wiley Publishers, New York, NY, 1964 (out of print); January 1987, pp. 2936.
corrected and reprinted by Dover Publishers, New York, NY, 1984, p. (11) TAPPI Collaborative Reference Program, Reports 25 through 51,
105. Aug. 1973 through Jan. 1978, Technical Association of the Pulp and
(6) Manual on Presentation of Data and Control Chart Analysis, MNL7A,
Paper Industry.
ASTM 2002.

ASTM International takes no position respecting the validity of any patent rights asserted in connection with any item mentioned
in this standard. Users of this standard are expressly advised that determination of the validity of any such patent rights, and the risk
of infringement of such rights, are entirely their own responsibility.

This standard is subject to revision at any time by the responsible technical committee and must be reviewed every five years and
if not revised, either reapproved or withdrawn. Your comments are invited either for revision of this standard or for additional standards
and should be addressed to ASTM International Headquarters. Your comments will receive careful consideration at a meeting of the
responsible technical committee, which you may attend. If you feel that your comments have not received a fair hearing you should
make your views known to the ASTM Committee on Standards, at the address shown below.

This standard is copyrighted by ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959,
United States. Individual reprints (single or multiple copies) of this standard may be obtained by contacting ASTM at the above
address or at 610-832-9585 (phone), 610-832-9555 (fax), or [email protected] (e-mail); or through the ASTM website
(www.astm.org).

13